trove-specs

Cassandra Configuration Groups

Sun, 19 Apr 2020 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/trove/+spec/cassandra-configuration-groups

Problem Description

The Cassandra datastore currently does not support configuration groups.

Proposed Change

The patch set will implement configuration groups for Cassandra 2.1.

Configuration

The configuration template will be updated to the default template for the target platform.

templates/cassandra/config.template

Database

None

Public API

None

Public API Security

None

Python API

None (empty section added after merging)

CLI (python-troveclient)

None (empty section added after merging)

Internal API

None

Guest Agent

Cassandra stores its configuration in ‘cassandra.yaml’ file (commonly in ‘/etc/cassandra’). The node (datastore service) has to be restarted for any changes to the configuration file to take effect. All configuration changes will therefore be requiring database restart and ‘apply_overrides’ will be implemented as no-op.

Overrides will be implemented by replacing the current file with an updated one. The old file will be backed up in the same directory (as *.old) and restored on configuration reset.

Most configuration properties will be available via configuration groups. Some, however, do not make sense in the Trove context.

These include:

irrelevant options (like automatic snapshots, since the Trove user cannot retrieve them)

guestagent specific (e.g. file paths, passwords)

items that Trove needs to control (replication/clustering properties)

See Available Configuration Properties for the full list of supported options.

The user should be able to specify configurations properties as standard Python YAML objects - key-value pairs and dicts.

Available Configuration Properties

Properties not included in the lists are kept at their default values and are not configurable via Trove.

The properties configurable by the user via the Trove API:

cluster_name

listen_address

commit_failure_policy

disk_failure_policy

endpoint_snitch

seed_provider

compaction_throughput_mb_per_sec

memtable_total_space_in_mb

concurrent_reads

concurrent_writes

phi_convict_threshold

commitlog_sync

commitlog_segment_size_in_mb

commitlog_total_space_in_mb

compaction_preheat_key_cache

concurrent_compactors

in_memory_compaction_limit_in_mb

preheat_kernel_page_cache

sstable_preemptive_open_interval_in_mb

memtable_allocation_type

memtable_cleanup_threshold

file_cache_size_in_mb

memtable_flush_queue_size

memtable_flush_writers

memtable_heap_space_in_mb

memtable_offheap_space_in_mb

column_index_size_in_kb

index_summary_capacity_in_mb

index_summary_resize_interval_in_minutes

reduce_cache_capacity_to

reduce_cache_sizes_at

stream_throughput_outbound_megabits_per_sec

inter_dc_stream_throughput_outbound_megabits_per_sec

trickle_fsync

trickle_fsync_interval_in_kb

auto_bootstrap

batch_size_warn_threshold_in_kb

broadcast_address

initial_token

initial_token

num_tokens

partitioner

key_cache_keys_to_save

key_cache_save_period

key_cache_size_in_mb

row_cache_keys_to_save

row_cache_size_in_mb

row_cache_save_period

memory_allocator

counter_cache_size_in_mb

counter_cache_save_period

counter_cache_keys_to_save

counter_cache_keys_to_save

tombstone_warn_threshold

tombstone_failure_threshold

range_request_timeout_in_ms

read_request_timeout_in_ms

counter_write_request_timeout_in_ms

cas_contention_timeout_in_ms

truncate_request_timeout_in_ms

write_request_timeout_in_ms

request_timeout_in_ms

cross_node_timeout

internode_send_buff_size_in_bytes

internode_recv_buff_size_in_bytes

internode_compression

inter_dc_tcp_nodelay

streaming_socket_timeout_in_ms

native_transport_max_threads

native_transport_max_frame_size_in_mb

broadcast_rpc_address

rpc_keepalive

rpc_max_threads

rpc_min_threads

rpc_recv_buff_size_in_bytes

rpc_send_buff_size_in_bytes

rpc_server_type

dynamic_snitch_badness_threshold

dynamic_snitch_reset_interval_in_ms

dynamic_snitch_update_interval_in_ms

hinted_handoff_enabled

hinted_handoff_throttle_in_kb

max_hint_window_in_ms

max_hints_delivery_threads

batchlog_replay_throttle_in_kb

request_scheduler

request_scheduler_id

request_scheduler_options

thrift_framed_transport_size_in_mb

thrift_max_message_length_in_mb

permissions_validity_in_ms

permissions_update_interval_in_ms

Non-configurable properties with updated default values:

authenticator: org.apache.cassandra.auth.PasswordAuthenticator

authorizer: org.apache.cassandra.auth.CassandraAuthorizer

snapshot_before_compaction: false

auto_snapshot: false

rpc_address: (controlled-by guestagent)

broadcast_rpc_address: (controlled-by guestagent)

listen_address: (controlled-by guestagent)

seed_provider.parameters.seeds: (controlled-by guestagent)

Alternatives

None

Implementation

Assignee(s)

Petr Malik <pmalik@tesora.com>

Milestones

Liberty

Work Items

Implement functionality to handle (read/write/update) YAML files.
Implement configuration-related manager API calls.

Upgrade Implications

None

Dependencies

The patch set will be building on functionality implemented in blueprint: cassandra-database-user-functions

Testing

Unittests will be added to validate implemented functions and non-trivial codepaths.

Documentation Impact

The datastore documentation should be updated to reflect the enabled features.

References

Documentation on Cassandra 2.1: http://docs.datastax.com/en/cassandra/2.1/cassandra/gettingStartedCassandraIntro.html Documentation on Cassandra 2.1 configuration properties: http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configTOC.html

Cassandra Database and User Functions

Sun, 19 Apr 2020 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/trove/+spec/cassandra-database-user-functions

Problem Description

The Cassandra datastore currently does not support keyspace * and user management features.

*: A keyspace is the outermost container for data in Cassandra, corresponding closely to a relational database.

Proposed Change

The patch set will implement the following keyspace and user related functionality for Cassandra 2.1 2 datastore:

User Functions:

create/delete/get user
list users
change password
grant/revoke/list access
update attributes

Keyspace Functions:

create/delete database
list databases

Configuration

Cassandra stores its configuration in ‘cassandra.yaml’ file (commonly in ‘/etc/cassandra’). The node (datastore service) has to be restarted for any changes to the configuration file to take effect. The configuration template will have to be updated to enable authentication and authorization in order to support datastore users and related functions. Client-specific settings (authentication defaults) are stored in ‘~/.cassandra/cqlshrc’ where ‘~’ is the home directory of the Trove user.

Database

None

Public API

None

Public API Security

The current implementation allows original anonymous connections therefore making the datastore wide open for anybody with connection URL. This has to be changed first as the user functions are not even enabled in this setting.

In Cassandra only SUPERUSERS can create other users and grant permissions to database resources. Trove uses the ‘cassandra’ superuser to perform its administrative tasks. The users it creates are all ‘normal’ (NOSUPERUSER) accounts. The permissions it can grant are also limited to non-superuser operations. This is to prevent anybody from creating a new superuser via the Trove API. Similarly, all list operations include only non-superuser accounts. Updatable attributes include username and password. We are not going to implement enabling superuser account in this patch set.

The datastore configuration template had to be updated to enable authentication and authorization support (original configuration allowed anonymous connections). Default implementations used are:

authenticator: org.apache.cassandra.auth.PasswordAuthenticator
authorizer: org.apache.cassandra.auth.CassandraAuthorizer

The superuser password needs to be changed from the default ‘cassandra’ to a random Trove password which is then stored in a Trove-read-only file in ‘~/.cassandra/cqlshrc’ which is also the default location for client settings.

Python API

None (empty section added after merging)

CLI (python-troveclient)

None (empty section added after merging)

Internal API

None

Guest Agent

The current implementation uses the CQLSH 3 command line client to interface with the underlying database. Trove talks to the CQLSH client via the available shell and relies on parsing the output of the client to determine the current state of the datastore and status of the last operation. It also requires Trove to provide the input as a formatted (and sanitized) string accepted by the installed version of the client. This is not very portable and poses numerous potential compatibility issues in the future when the guest agent gets ported to other platforms. It also completely bypasses the native exception handling implemented by the Python client. In order to make the guestagent communicate via the native Python API and avoid future portability issues we reimplement the communication interface using the the official open-source Python driver for Cassandra. The native interface will be implemented in CassandraConnection and will also be used to obtain the database status leveraging Python exceptions framework.

User-related functions require having the authorization and authentication support enabled on the server. The configuration template will be updated to reflect this requirement (see the section on ‘Public API Security’).

The following section elaborates on keyspace (“database”) functions. Cassandra natively supports row replication. It can store a configurable number of copies (replicas) of each row on multiple nodes to ensure reliability and fault tolerance. All such replicas are equally important; there is no primary or master. It supports several replication strategies which determine how the replicated rows get distributed across the entire Cassandra cluster. The total number of row replicas in a cluster is referred to as the replication factor. The ‘create database’ implementation will be using ‘SimpleStrategy’ which is the only strategy that makes sense for a single node setup. It is a very simplistic configuration with just a single copy (replica) on the guest machine only good for the most basic applications and demonstration purposes. The following system keyspaces will be by-default excluded from database operations and listing by having them included in the ‘ignore_dbs’ list: ‘system’, ‘system_auth’, ‘system_traces’

Alternatives

None

Implementation

Assignee(s)

Petr Malik <pmalik@tesora.com>

Milestones

Liberty-1

Work Items

Implement native-driver-based connection for Cassandra datastore.
Enable authentication and authorization on server.
Implement user-function API calls.

Upgrade Implications

The required native Python driver 2.0.x supports Cassandra 1.2 and higher.

Dependencies

Trove uses the official open-source Python driver 1 for Cassandra to connect to the database and execute queries. The driver already exists in OpenStack global requirements. It does not have to be included in the ‘requirements.txt’ file, but it will need to be added to ‘test-requirements.txt’ file to enable unit tests. The image provider will be required to install it (‘cassandra-driver’) in the Cassandra images. The current Trove integration images will be updated to do that.

Testing

Unittests will be added to validate implemented functions and non-trivial codepaths.

Documentation Impact

The datastore documentation should be updated to reflect the enabled features.

References

1: Native Python Driver for Cassandra: http://docs.datastax.com/en/developer/python-driver/2.5/common/drivers/introduction/introArchOverview_c.html
2: Documentation on Cassandra 2.1: http://docs.datastax.com/en/cassandra/2.1/cassandra/gettingStartedCassandraIntro.html
3: CQL Reference: http://docs.datastax.com/en/cql/3.1/cql/cql_reference/cqlReferenceTOC.html

Implement Percona cluster provisioning

Sun, 19 Apr 2020 00:00:00

To continue the Trove HA story, high availability of database instances is required in order to give users a peace of mind when running their production workloads in Trove. Apart from replication, clustering makes the deployment to be highly available by allowing data to be accessed from any node in the cluster. It enhances system uptime, it prevents from data loss and makes the architecture more scalable for future growth. Trove Clustering for the Percona datastore will be based on the Percona XtraDB Cluster, which is a true Multimaster Cluster based on synchronous replication.

This change will focus on the Create, List, Show, and Delete of a Peronca cluster. Grow and shrink of the cluster will be handled in a separate blueprint.

https://blueprints.launchpad.net/trove/+spec/support-pxc-56

Problem Description

The Percona datastore does not have support for clustering. Adding a cluster implementation will allow users to create read and write scalable and highly available deployments.

Proposed Change

Percona cluster will leverage the base cluster implementation that currently exists in Trove by creating plugabble and configurable strategies.

Requirements:

The cluster should not have less than three nodes.
Initial cluster creation will start with 3 or more.
The cluster will return all IPs of its nodes. This is done so that the cluster can be used for multi write and read.
The name of each instance will be automatically generated; it will be the <cluster_name>-<instance_num>.
The clusters shall have nodes of same volumes but not for flavors.
Clusters shall allow the instances/cluster members to be resizable for volume and flavors. No validation will occur if a user resizes volumes or flavors of an instance.

The following will be changed:

A percona-cluster implementation will be added to common/strategies/cluster/experimental/mysql/xtradbcluster
Task manager strategy to handle the cluster provisioning for percona.
Guestagent strategy to handle the clustering provisioning, configuration and execution.
Unit and integration tests.

Configuration

A new datastore configuration section will be added. This initial implementation will add no Percona-Cluster specific configuration options.

Trove Integration

The Redstack script will create a datastore configuration for percona-cluster to open ports 3306, 4444, 4567, and 4568.

Elements will be added to support “redstack kick-start percona-cluster-5.6”.

Database

This implementation will leverage the existing database schema for cluster.

Public API

The API request for creating/deleting cluster will remain unchanged. However, there will be subtle differences in the response. Percona clustering will return all IPs of its nodes. This will be used for the user who wants to scale read/write.

Create Cluster

Request:

POST /v1.0/<tenant_id>/clusters
{
  "cluster": {
    "name": "products",
    "datastore": {
      "type": "percona",
      "version": "5.5"
    },
    "instances": [
      {
        "flavorRef": "2",
        "volume": {
          "size": 100
        },
      },
      {
        "flavorRef": "2",
        "volume": {
          "size": 100
        },
      },
      {
        "flavorRef": "2",
        "volume": {
          "size": 100
        },
      }
    ],
  }
}

Response:

{
  "cluster": {
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "task": {
      "id": 2,
      "name": "BUILDING",
      "description": "Building the initial cluster."
    },
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "percona",
      "version": "5.5"
    },
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "status": "BUILD",
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
        "status": "BUILD",
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
        "status": "BUILD",
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
      }
    ],
  }
}

Show Cluster

Request:

GET /v1.0/<tenant_id>/clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "cluster": {
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "task": {
      "id": 1,
      "name": "NONE",
      "description": "No tasks for the cluster."
    },
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "percona",
      "version": "5.5"
    },
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "status": "ACTIVE",
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
        "ip": "10.0.0.1"
      }
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
        "status": "ACTIVE",
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
        "ip": "10.0.0.2"
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
        "status": "BUILD",
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
        "ip": "10.0.0.3"
      }
    ],
  }
}

Show Instance

Request:

GET /v1.0/<tenant_id>/clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1

Response:

{
  "instance": {
    "status": "ACTIVE",
    "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
    "cluster_id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products-1",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "percona",
      "version": "5.5"
    },
    "ip": ["10.0.0.1"],
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 100,
      "used": 0.17
    },
  }
}

List Clusters

Request:

GET /v1.0/<tenant_id>/clusters

Response:

{
  "clusters": [
    {
      "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
      "task": {
        "id": 1,
        "name": "NONE",
        "description": "No tasks for the cluster."
      },
      "name": "products",
      "created": "2014-04-25T20:19:23",
      "updated": "2014-04-25T20:19:23",
      "links": [{...}],
      "ip": ["10.0.0.1", "10.0.0.2", "10.0.0.3"],
      "datastore": {
        "type": "percona",
        "version": "5.5"
      },
      "instances": [
        {
          "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
          "status": "ACTIVE",
          "flavor": {
            "id": "7",
            "links": [{...}]
          },
          "volume": {
            "size": 100
          },
          "ip": "10.0.0.1",
        }
        {
          "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
          "status": "ACTIVE",
          "flavor": {
            "id": "7",
            "links": [{...}]
          },
          "volume": {
            "size": 100
          },
          "ip": "10.0.0.2",
        },
        {
          "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
          "status": "ACTIVE",
          "flavor": {
            "id": "7",
            "links": [{...}]
          },
          "volume": {
            "size": 100
          },
          "ip": "10.0.0.3",
        }
      ]
    },
    ...
  ]
}

Delete Cluster

Request:

DELETE /v1.0/<tenant_id>/clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

HTTP 202 (Empty Body)

Public API Security

None

Internal API

No changes in the internal API yet.

Guest Agent

No API changes in the guest agent. This feature will just be adding a new strategy to support a different type of clustering.

CLI (python-troveclient)

The following illustrates the clustering CLI

Create Cluster

$ trove help cluster-create

usage: trove cluster-create <name> <datastore> <datastore_version>
                            [--instance <instance>]

Creates a new cluster.

Positional arguments:
  <name>                Name of the cluster.
  <datastore>           A datastore name or UUID.
  <datastore_version>   A datastore version name or UUID.

Optional arguments:

  --instance <flavor_id=flavor_id,volume=volume,parameters=<key=value>>
                        Create an instance for the cluster. Specify
                        multiple times to create multiple instances.

Request:

$ trove cluster-create products percona "5.5" \
  --instance flavor_id=7,volume=2 \
  --instance flavor_id=7,volume=2 \
  --instance flavor_id=7,volume=2

Response:

+-------------------+--------------------------------------+
| Property          | Value                                |
+-------------------+--------------------------------------+
| created           | 2014-08-16T01:46:51                  |
| datastore         | percona                              |
| datastore_version | 5.5                                  |
| id                | aa6ef0f5-dbef-48cd-8952-573ad881e717 |
| name              | products                             |
| task_description  | Building the initial cluster.        |
| task_name         | BUILDING                             |
| updated           | 2014-08-16T01:46:51                  |
+-------------------+--------------------------------------+

The cluster-create command will create a cluster of functionally equivalent trove instances. Trove will choose a instance to start with the –wsrep-new-cluster option, then the remaining instances will be started sequentially. The cluster will be marked ACTIVE when all instances have started and the database is available for use.

Show Cluster

$ trove help cluster-show

usage: trove cluster-show <cluster>

Shows details of a cluster.

Positional arguments:
  <cluster>  ID or name of the cluster.

Request:

$ trove cluster-show aa6ef0f5-dbef-48cd-8952-573ad881e717

Response:

+-------------------+--------------------------------------+
| Property          | Value                                |
+-------------------+--------------------------------------+
| created           | 2014-08-16T01:46:51                  |
| datastore         | percona                              |
| datastore_version | 5.5                                  |
| id                | aa6ef0f5-dbef-48cd-8952-573ad881e717 |
| ip                | 10.0.0.2, 10.0.0.1, 10.0.0.3         |
| name              | products                             |
| task_description  | No tasks for the cluster.            |
| task_name         | NONE                                 |
| updated           | 2014-08-16T01:59:33                  |
+-------------------+--------------------------------------+

Show Cluster Instance

$ trove help cluster-instances

usage: trove cluster-instances <cluster>

Lists all instances of a cluster.

Positional arguments:
  <cluster>  ID or name of the cluster.

Request:

$ trove cluster-instances aa6ef0f5-dbef-48cd-8952-573ad881e717

Response:

+-------------------------------------+----------------+-----------+------+
| ID                                  | Name           | Flavor ID | Size |
+-------------------------------------+----------------+-----------+------+
| 45532fc4-661c-4030-8ca4-18f02a2b337 | products-1     | 7         |    2 |
| 7458a98d-6f89-4dfd-bb61-5cf1d65c121 | products-2     | 8         |    2 |
| 1557208f-5c23-4537-a9f2-52a9db38d3a | products-3     | 7         |    2 |
+-------------------------------------+----------------+-----------+------+

List Clusters

$ trove help cluster-list

usage: trove cluster-list [--limit <limit>] [--marker <ID>]

Lists all the clusters.

Optional arguments:
  --limit <limit>  Limit the number of results displayed.
  --marker <ID>    Begin displaying the results for IDs greater than the
                   specified marker. When used with --limit, set this to
                   the last ID displayed in the previous run.

Request:

$ trove cluster-list

Response:

+--------+----------+-----------+-----------+----------+-----------+
| ID     | Name     | Datastore | DsVersion | IP       | Task Name |
+--------+----------+-----------+-----------+----------+-----------+
| uuid-1 | products | percona   | 5.5       | ip1      | NONE      |
| uuid-2 | items    | percona   | 5.5       | ip2, ip3 | BUILDING  |
+--------+----------+-----------+-----------+----------+-----------+

Delete Cluster

$ trove help cluster-delete

usage: trove cluster-delete <cluster>

Deletes a cluster.

Positional arguments:
  <cluster>  ID of the cluster.

Request:

$ trove cluster-delete aa6ef0f5-dbef-48cd-8952-573ad881e717

Response:

(None)

Replication Impact

Due to the scope of adding a replica to a cluster this will be disabled as part of this blueprint and added as a separate blueprint. 4

Python API

???

Alternatives

None defined yet.

Upgrade Implications

To enable MySQL cluster support, a different package will be needed to install on the guest instance rather than the default. There are 2 ways to mitigate this in an upgrade path.

Update the packages installed on the datastore version.
Build a new image with the new package installed into the guest image.

Implementation

Assignee(s)

Name	Launchpad Id	IRC	Email
Name	Launchpad Id	IRC	Email
Steve Leon	steve-leon	esmute	kokhang@gmail.com
Steve Leon	steve-leon	esmute	kokhang@gmail.com
Morgan Jones	6-morgan	vgnbkr	morgan@tesora.com
Morgan Jones	6-morgan	vgnbkr	morgan@tesora.com
Craig Vyvial	cp16net	cp16net	cp16net@gmail.com
Craig Vyvial	cp16net	cp16net	cp16net@gmail.com

Milestones

Liberty-3

Work Items

trove-integration change to add percona-xtradb-cluster-server package
API strategy implementation
Taskmanager strategy implementation
Guestagent strategy implementation
Unit and integration tests

Dependencies

None

Testing

There will be unit tests that test every components in the strategies. There will be integration tests which will test end-to-end cluster features

Documentation Impact

The response API will look different due to the nature of XtraDB clusters. For example, the view will contain all the IPs of the instances.

References

API strategy to handle clustering request for Percona. Besides the existing API, there will be new API to grow/shrink cluster handled in another blueprint. (bp/cluster-scaling 1 )

Trove does not monitor the status of cluster instances. Should an instance in the cluster become detached from the primary instance it will be up to the user to detect the situation and take corrective action. If the instance does not re-attach to the cluster in a state where an Incremental State Transfer (IST) is possible, it is suggested to delete the instance and add a new node to the cluster. (In the future, a user will be able to delete the instance from a cluster via the shrink action and add a new instance to the cluster with the grow action.)

In the event of a nova compute reboot, the nodes will automatically restart, and the cluster will automatically recover provided that the nodes can determine that the PRIMARY component can be properly recovered. In the case that the PRIMARY component cannot be recovered, operator intervention would be required to manually restart the cluster.

List of other cluster related blueprints. These blueprints will likely cause conflicts with changes.

1: bp/cluster-scaling: https://blueprints.launchpad.net/trove/+spec/cluster-scaling
4: percona-xtradb-with-replica: https://blueprints.launchpad.net/trove/+spec/percona-add-replica-of-cluster

bp/mysql-manager-refactor:: https://blueprints.launchpad.net/trove/+spec/mysql-manager-refactor
bp/cluster-user-management:: https://blueprints.launchpad.net/trove/+spec/cluster-user-management

Add MariaDB GTID Replication Strategy

Sun, 19 Apr 2020 00:00:00

Global Transaction ID (GTID) Replication support was added during the Kilo release for MySQL 5.6 and later. The equivalent for MySQL 5.6, MariaDB 10, has a different implementation of GTID. By creating a similar strategy implementation we could get support for GTID Replication for the MariaDB 10 datastore.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/mariadb-gtid-replication

Problem Description

In order to create a replica of a MariaDB instance you can only use binary logs based replication. GTID replication was added as an alternative for replication for MySQL 5.6, but it’s not available for the equivalent for it, MariaDB 10.

Proposed Change

We would need to create trove/trove/guestagent/strategies/replication/experimental/mariadb_gtid.py in order to support replication for MariaDB.

The SQL issued to set a new empty slave server and replicate all of the master’s binlog from the start is as follows 2:

CHANGE MASTER TO master_host="%(host)s", master_port=%(port)s, \
master_user="%(user)s", master_password="%(password)s", \
master_use_gtid=current_pos;
START SLAVE;

By default, the GTID position for a newly installed server is empty, which makes the slave replicate from the start of the master’s binlogs. 3 current_pos in the SQL query carries that position.

It doesn’t differ too much on what is being done for MySQL 1, but doing a refactoring might not be the best call right now since it would add unnecessary complexity to the existing strategy.

Configuration

None

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

None

Guest Agent

None

Alternatives

Don’t support GTID-based Replication for MariaDB.

Dashboard Impact (UX)

None

Implementation

Assignee(s)

Primary assignee:: vkmc vgnbkr

Milestones

Target Milestone for completion:: Mitaka-2

Work Items

Create MariaDB GTID Replication strategy
Create integration tests for replication in MariaDB 10

Upgrade Implications

None

Dependencies

None

Testing

Integration tests will be added to cover this functionality.

Documentation Impact

Docs explaining this new addition will be added.

References

1: https://github.com/openstack/trove/blob/master/trove/guestagent/strategies/replication/mysql_gtid.py
2: https://mariadb.com/kb/en/mariadb/gtid/
3: https://mariadb.com/kb/en/mariadb/gtid/

Appendix

None

PostgreSQL Replication

Sun, 19 Apr 2020 00:00:00

From version 9.0 onwards, PostgreSQL ships with streaming replication as part of the core product, reversing a long held policy of leaving replication to third-party add-ons such as Slony or pgpool.

This blueprint proposes to support streaming replication since it is shipped with the core product, although a discussion of some of the other PostgreSQL options can be found in the alternatives section.

https://blueprints.launchpad.net/trove/+spec/postgresql-replication

Problem Description

In order to achieve feature parity with MySQL, Trove should provide support for at least one of the replication solutions for PostgreSQL.

Proposed Change

Configuration

The standard parameters for enabling replication strategies, replication_strategy and replication_namespace, will be added to point to the strategy code for PostgreSQL guests.

Database

None.

Public API

As with other datastores that support replication, the create instance operation will support the replica_of and replica_count fields for PostgreSQL guests:

POST http://127.0.0.1:8779/v1.0/<tenant id>/instances
{
    "instance": {
        "volume": {"size": <size>},
        "flavorRef": "<flavor-id>",
        "name": "s",
        "replica_of": "<master id>",
        *"replica_count": "<n>"*
    }
}

For guests with pg_rewind 9 support, the following instance actions are supported:

POST http://127.0.0.1:8779/v1.0/<tenant id>/instance/<instance id>/action
{
    *"detach_replica": null*
}

POST http://127.0.0.1:8779/v1.0/<tenant id>/instance/<instance id>/action
{
    *"eject_replica_source": null*
}

POST http://127.0.0.1:8779/v1.0/<tenant id>/instance/<instance id>/action
{
    *"promote_to_replica_source": null*
}

Public API Security

None.

Python API

None.

CLI (python-troveclient)

As with other datastores that support replication, this will enable the commands:

trove create <inst> .. --replica_of <master_inst>

on PostgreSQL guests. With pg_rewind 9 support on the guest, failover and reconfiguration commands are supported:

trove detach-replica <inst>
trove eject-replica-source <inst>
trove promote-to-replica-source <inst>

Internal API

None.

Guest Agent

Write-Ahead Log (WAL) and Continuous Shipping

PostgreSQL provides both high performance and crash recovery capability through the use of a write-ahead log. The WAL files are appended to for every change to the database, with periodic checkpoints to purge the logs and integrate the changes into the database files 4.

PostgreSQL streaming replication is built on top of the recovery system: a master database operates in continuous archiving mode, sending the contents of its WAL files to one or more slave servers. These slave servers, in turn, operate in continuous recovery mode, “recovering” from the logs as they are received. A slave can be promoted in place of a failed master, and can be configured to support read-only transactions while in recovery mode.

Replication interface

Replication Snapshots

The PostgreSQL guestagent currently provides a backup and restore strategy which makes use of the pg_dump and pg_restore commands. These produce logical dumps which cannot be used as part of a replication system that depends on continuous archiving. As such, a backup/restore strategy using the pg_basebackup tool is a requirement for replication.

Enabling a master

In order to support hot standby slaves, a PostgreSQL master must have wal_level set to hot_standby, which is the most verbose mode. Replication is handled through the use of a user that has been given REPLICATION privilege 6 and has been explicitly allowed to access the special replication database in the pg_hba.conf file. After enabling these changes, a configuration reload is done.

Enabling a slave

Enabling a slave requires a recent backup to be restored. Since streaming replication bootstraps the recovery system, a recovery.conf file is written to the PGDATA directory containing the connection details for the master that should be replicated from. A restart is required to enable continuous recovery mode.

Detaching a slave

To detach a slave in PostgreSQL means to stop recovery mode. This is done by writing a special trigger file, configured with the trigger_file option in the recovery configuration.

Demote master

To demote a master requires no special action other than to revert configuration changes to their defaults.

Failover and Failback

The failover process in Trove is controlled by the task-manager, but the guest agent must implement functions that allow the task manager to determine the best slave to promote and when it can proceed.

Global Transactions

Standard PostgreSQL does not support an equivalent of the GTID in MySQL, so the combination of host + WAL location 8 will be used as a transaction identifier where necessary.

A simple polling mechanism will be implemented to determine when a slave has caught up to the point of a particular change.

Reattaching Slaves

Failback in postgresql is complicated by recovery timelines 2 . When a slave is triggered out of recovery mode, it jumps to a new timeline, generating new WAL data into a “fork” of the previous database state. This can be seen in the example of these 24-character WAL filenames:

000000010000000000000006
000000020000000000000007

These represent WAL files 6 and 7, but the 7th file is on a second timeline forked from the first.

When a master is demoted, however, it does not change timelines, and so in order to safely reattach this demoted master to a newly-promoted slave, a timeline resync is required.

This can only be done safely through the use of the tool pg_rewind 9. This tool is supported for PostgreSQL 9.4, but must be compiled separately. In PostgreSQL 9.5 it will be shipped with the core product.

For guests that have pg_rewind available, failback can be done, otherwise a manual recreate of another slave from the master is required.

Alternatives

A number of third-party replication options exist for PostgreSQL, including Slony, pgpool-II and a number of commercially-available solutions 1.

pgpool-II depends on middleware that inserts itself between the client and the underlying database instances. It provides the benefit of multi-master replication, however conflict-resolution may be required in some cases.

Slony provides master-slave replication using table-level triggers. It has greater overhead on the master database than standard streaming replication, but has the benefit of table-level granularity.

Dashboard Impact (UX)

TBD (section added after approval)

Implementation

Assignee(s)

Primary assignee: atomic77

Milestones

Target Milestone for completion:

mitaka-1

Work Items

implement basic streaming replication
implement failover-related APIs
add postgresql-specific hooks as necessary to enable generic int-tests for replication to run against PostgreSQL guests

Upgrade Implications

None.

Dependencies

pg_basebackup incremental backup and restore strategy for PostgreSQL. 3 10

Testing

Postgresql-specific hooks to the generic int-test framework will be added as necessary.

Documentation Impact

The documentation will need to be updated to indicate that the PostgreSQL guest supports replication.

Appendix

None.

References

1: http://www.postgresql.org/docs/9.4/static/different-replication-solutions.html
2: http://www.postgresql.org/docs/9.4/static/continuous-archiving.html
3: http://www.postgresql.org/docs/current/static/app-pgbasebackup.html
4: http://www.postgresql.org/docs/9.0/static/wal-configuration.html

[5] http://www.postgresql.org/docs/9.4/static/warm-standby-failover.html

6: http://www.postgresql.org/docs/current/static/sql-createrole.html

[7] http://www.postgresql.org/message-id/flat/CA+TgmobWQJ-GCa_tWUc4=80A: 1RJ2_+Rq3w_MqaVguk_q018dqw@mail.gmail.com#CA+TgmobWQJ-GCa_tWUc4=80A1RJ 2_+Rq3w_MqaVguk_q018dqw@mail.gmail.com

8: By “WAL location” we mean the position in the WAL file, as would be returned by the pg_current_xlog_location() system administration function
9(1,2,3): https://github.com/vmware/pg_rewind/tree/REL9_4_STABLE
10: https://blueprints.launchpad.net/trove/+spec/postgresql-incremental-backup

Percona XtraDB Cluster Grow and Shrink Support

Sun, 19 Apr 2020 00:00:00

For the Percona XtraDB Cluster (pxc) datastore we need the ability to grow and shrink a cluster. This will add these 2 core features to the pxc solution we currently have.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/pxc-grow-shrink-cluster

Problem Description

PXC is lacking support for grow and shrink clusters. This will allow a user to create a new instance in the cluster and delete an existing instance from the cluster.

An example use case would be if an instance in the cluster is having issues due to the underlying hardware, network, or other issues then a user can remove the instance and create a new one to keep the cluster healthy.

Proposed Change

PXC will use the existing grow and shrink API calls and implement them for the pxc manager. This change will allow the cluster to be heterogeneous when needing to grow or shrink the cluster. A heterogeneous cluster means that the flavor and volume may not all be the same for every instance in a cluster.

Requirements for grow:

A new instance in the cluster can be any valid flavor size.
A new instance in the cluster must have a volume of the smallest size of the instances in the cluster or larger. This is to make sure that new instances created in the cluster have enough volume space to operate with the rest of the cluster instance.
Quota validation that user has the capacity to grow the cluster by the number of instances and volume space in their account.

Requirements for shrink:

There must be at least 2 instances in the cluster in order to shrink a cluster. A cluster instance count must have at least 1 instance to operate.

A few new guestagent API calls will be needed in order for grow to work.

PXC handles joining a cluster automatically when given an ip address of one of the nodes in the existing cluster. When given all the ips of the existing cluster instances PXC will sequentially attempt to connect to the ips in the list and attempt to join the cluster. If one of the instances in the list is not up (crash or network or otherwise and issue) then PXC will try the next ip in the list. This will help allow a user to join a new instance to the cluster so that they can then shrink or remove the problematic instance from the cluster.

Configuration

No configuration changes are anticipated.

Database

None

Public API

The following public API calls will be made available to the PXC datastore.

Cluster Grow - The existing call payload will not be changed. Implementing the grow cluster feature will add the new instances to the existing cluster.
Cluster Shrink - The existing call payload will not be changed. Implementing the shrink cluster feature will allow a user to remove instances from their existing cluster.

Public API Security

None

Python API

None

CLI (python-troveclient)

Support for the following existing CLI calls.

cluster-grow
cluster-shrink

No changes should be nessesary to accomplish these actions.

Internal API

There are few new guestagent calls that will need to be made to get the data from the existing instances in the cluster as well as updating the cluster configuration on the existing instances. 1

The process of growing the cluster or adding a new instance(s) to the cluster:

First set the cluster state to GROWING
Boot a new pxc instance.
Once the instance is up then we get the configuration setting of the existing cluster from one of the instances in the cluster. These setting include the cluster name, replication settings, and ips of the existing cluster.
Once the configuration on the new instance(s) is set then we start mysql on each instance. The new instance(s) find the cluster from the ips set and the cluster name.
After mysql is started up on the new instance(s) the cluster recognizes the new instances in the cluster.
At this point the cluster is in a good state but in order to be able to help recover or restart an instance in the cluster we push down a new configuration that includes the new ip address of the new instance(s) to all the instances in the cluster.
After all the instances in the cluster have the new configuration there is no need to restart the mysql process again because the cluster already recognizes all the instances in the cluster.
Then we set the task state of the cluster to complete or NONE.

The process of shrinking a cluster is removing a given instance(s) from the cluster:

First we set the task state to SHRINKING.
Then stop mysql on the given instance(s)
Then delete these instance(s) as well.
Next the existing instance(s) in the cluster need to have their configurations updated to match the current state of the cluster. This will entail just updating the ips of the current instance(s) in the cluster.
No need for a restart after this since the cluster automatically knows about the active instance(s) in the cluster. The configuration update is for recovery and restart if needed later.
After the configuration update is done then we can reset the task state of the cluster to completed or NONE.

Guest Agent

Like mentioned above in the Internal API section there will be a few new guestagent calls to support grow and shrink.

Methods added to the Guest Agent

write_cluster_configuration_overrides - This method will write out the updated cluster configuration changes needed for the cluster to be grown or shrunk.
get_cluster_context - This method will return the context of the current cluster. The context is the information that is used to connect the new instances to the existing cluster setup. Context includes cluster settings and replication settings.

No calls will be deprecated in order to complete this.

Alternatives

Not to support grow and shrink of a pxc cluster.

Dashboard Impact (UX)

TBD (section added after approval)

Implementation

Assignee(s)

Name	Launchpad Id	IRC	Email
Name	Launchpad Id	IRC	Email
Craig Vyvial	cp16net	cp16net	cp16net@gmail.com
Craig Vyvial	cp16net	cp16net	cp16net@gmail.com

Milestones

Targeting this for milestone: Mitaka

Work Items

This work will be broken up into 2 separate items.

Grow a cluster (adding new instance(s) to cluster)
Shrink a cluster (removing instance(s) from cluster)

Upgrade Implications

The guestagents on the existing cluster instances must be updated prior to calling the grow or shrink API calls. Otherwise the cluster may get stuck in a GROWING_CLUSTER or SHRINKING_CLUSTER state, because the guestagent will not have the calls to support growing or shrinking a cluster.

Dependencies

None

Testing

There will be unit tests that test the new calls for the strategy.

There will be integration tests which will test grow and shrink cluster features.

Documentation Impact

We need to update the docs to show support for grow and shrink clusters for the pxc manager.

References

1: https://www.percona.com/doc/percona-xtradb-cluster/5.5/howtos/ubuntu_howto.html

https://www.percona.com/doc/percona-xtradb-cluster/5.5/manual/bootstrap.html#bootstrap

Appendix

None

Remove SQL Schema Downgrades

Sun, 19 Apr 2020 00:00:00

Trove still has the downgrade method on its migration files. Following the cross-project spec https://review.openstack.org/#/c/152337/ we are supposed to remove the downgrade to avoid data inconsistency, lack of integrity, etc. This approach was already done in many projects like Keystone, Magnum, Nova and others [2][3][4][5].

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/remove-sql-schema-downgrades

Problem Description

Downgrades are not the best idea when thinking about data integrity. A cross project spec was proposed to start removing downgrades and it was already done in many projects. We need to delete downgrade from migration files.

Proposed Change

Remove downgrades from migration files and remove the command from trove-manage.

Configuration

None

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

None

Guest Agent

None

Alternatives

None

Dashboard Impact (UX)

TBD (section added after approval)

Implementation

Assignee(s)

Milestones

Mitaka-1

Work Items

This work is basically removing downgrades, removing the command from trove-manage and update the tests.

Upgrade Implications

None

Dependencies

None

Testing

We need to update the tests so they will not fail when trying to downgrade.

Documentation Impact

None

References

Appendix

None

Add ‘volume_type’ parameter to instance create

Sun, 19 Apr 2020 00:00:00

Cinder allows for multiple storage backends. When creating a volume, the ‘volume_type’ parameter will be used in determining which type of backend to send to. 1 The user should be allowed to specify the backends for Trove volumes on instance/cluster creation.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/volume-type-on-create

Problem Description

Trove operators would like to expose different types of storage to their Trove users to provide more flexibility in the types of configurations they can use for their database instances and database clusters

Proposed Change

An optional ‘volume_type’ property will be added to the volume information accepted by the API for instance and cluster create API calls. This property will be passed to the Cinder client when a new Trove volume gets created. Default configuration value will be used if no volume_type is specified by the user.

Configuration

None

Database

None

Public API

An optional ‘volume_type’ property will be added to the volume information structure accepted by the instance and cluster create API calls.

API payload defining a volume of size ‘1’ and Cinder volume type ‘my-type-1’:

'volume': {'size': '1', 'type': 'my-type-1'}

API payload defining a volume of size ‘1’ and no Cinder volume type:

'volume': {'size': '1', 'type': None}
'volume': {'size': '1'}

Public API Security

None

Python API

None

CLI (python-troveclient)

The volume_type value is a string name of the volume type as returned by the ‘cinder type-list’ command. The value won’t be validated on the client site.

For instance creation an optional ‘–volume_type’ argument will be added. The volume type will be appended to the volume size in the volume information structure. If volume support is disabled or ‘–size’ argument is not specified on instance create the ‘volume_type’ argument will be ignored.

trove create ... --size 1 --volume_type my-type-1 ...

For cluster creation the ‘–instance’ argument will be extended with a ‘volume_type’ option. If volume support is not enabled or volume size is not specified the ‘volume_type’ option will be ignored.

trove cluster-create ... --instance volume=1,volume_type=my-type-1 ...

Internal API

A new ‘volume_type’ argument will be added where necessary.

Guest Agent

None

Alternatives

None

Dashboard Impact (UX)

TBD (section added after approval)

Implementation

Assignee(s)

Milestones

Mitaka-1

Work Items

This work will consist of updates to the Trove client and server code.

Upgrade Implications

None

Dependencies

None

Testing

The existing Trove tests will be extended to test the ‘volume_type’ argument. New client tests will be added to cover the added arguments.

Documentation Impact

The new ‘volume_type’ arguments on instance and cluster create commands need to be documented.

References

1: https://wiki.openstack.org/wiki/Cinder-multi-backend

Appendix

None

CouchDB Configuration Groups

Sun, 19 Apr 2020 00:00:00

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/couchdb-configuration-groups

Problem Description

The CouchDB datastore currently does not support configuration groups.

Proposed Change

The patch set will implement configuration groups for CouchDB.

CouchDB stores its configuration in ‘local.ini’ file.

Most configuration properties will be available via configuration groups. Some, however, do not make sense in the Trove context.

These include (will be documented in the configuration template):

guestagent specific (e.g. file paths, passwords, file access rules)

items that Trove needs to control (replication/clustering/log properties)

See Available Configuration Properties for the full list of supported options.

Configuration

None

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

CLI (python-troveclient)

This work will enable the following client commands:

configuration-attach

configuration-detach

Guest Agent

Implement update_overrides and apply_overrides in the manager and service modules.
A configuration template and validation rules with changes noted in Available Configuration Properties will be provided.

The existing ‘IniCodec’ implementation will be reused to handle text-file operations.

Alternatives

None

Dashboard Impact (UX)

None

Implementation

Assignee(s)

Sonali Goyal <sonaligoyal654321@gmail.com> Victoria Martinez de la Cruz <victoria@redhat.com>

Milestones

Netwon-1

Work Items

All changes will be done in the context of a single task.

Implement configuration-related manager API calls:

def update_overrides(self, context, overrides, remove=False)
def apply_overrides(self, context, overrides)

Upgrade Implications

None

Dependencies

The patch set will be building on functionality implemented in blueprint: couchdb-database-user-functions

Testing

The change is largely covered by the existing configuration tests. Unit tests will be added to validate any CouchDB-specific codepaths.

Documentation Impact

The datastore documentation should be updated to reflect the enabled features.

References

Documentation on CouchDB configuration: http://docs.couchdb.org/en/stable/config/index.html

Appendix

Available Configuration Properties

The properties configurable by the user via the Trove API:: [attachments] - compressible_types - compression_level [couchdb] - delayed_commits - max_attachment_chunk_size - attachment_stream_buffer_size - max_dbs_open - max_document_size - os_process_timeout [daemons] - auth_cache - db_update_notifier - external_manager - httpd - httpsd - query_servers - stats_aggregator - stats_collector - uuids - view_manager - vhosts [httpd] - allow_jsonp - authentication_handlers - changes_timeout - config_whitelist - default_handler - enable_cors - log_max_chunk_size - redirect_vhost_handler - secure_rewrites - server_options - socket_options - vhost_global_handlers - x_forwarded_host - x_forwarded_proto - x_forwarded_ssl [replicator] - checkpoint_interval - connection_timeout - db - http_connections - max_replication_retry_count - retries_per_request - socket_options - ssl_certificate_max_depth - ssl_trusted_certificates_file - use_checkpoints - verify_ssl_certificates - worker_batch_size - worker_processes [query_server_config] - commit_freq - os_process_limit - reduce_limit [os_daemon_settings] - max_retries - retry_time [couch_httpd_auth] - allow_persistent_cookies - auth_cache_size - authentication_db - authentication_redirect - iterations - max_iterations - min_iterations - proxy_use_secret - public_fields - require_valid_user - secret - timeout - users_db_public - x_auth_roles - x_auth_token - x_auth_username - use_users_db [compaction_daemon] - check_interval - min_file_size [database_compaction] - doc_buffer_size - checkpoint_after [log] - include_sasl - level [view_compaction] - keyvalue_buffer_size

Guestagent-controlled properties:

[httpd] - port - bind_address [log] - file [couchdb] - uri_file - util_driver_dir - view_index_dir - database_dir

Limit volume types for database instances

Sun, 19 Apr 2020 00:00:00

Different volume types have different cost and performance characteristics. Operators would like the flexibility to limit specific databases to specific volume types similar to the way in which databases can be limited to specific Nova flavors.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/associate-volume-type-datastore

Problem Description

Operators can create different volume types and a user can specify a volume type on a create instance request. Operators would like the ability to limit the volume types that can be used for a specific datastore-version in the same way that they can limit Nova flavors.

Proposed Change

The implentation that associated flavors with datastores 1 2 created a generic framework that provided for the specification of datastore-version-metadata. This metadata was stored into a table in the database called datastore_version_metadata.

 CREATE TABLE `datastore_version_metadata` (
              `id` varchar(36) NOT NULL,
              `datastore_version_id` varchar(36) DEFAULT NULL,
              `key` varchar(128) NOT NULL,
              `value` varchar(128) DEFAULT NULL,
              `created` datetime NOT NULL,
              `deleted` tinyint(1) NOT NULL,
              `deleted_at` datetime DEFAULT NULL,
              `updated_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UQ_datastore_version_metadata_datastore_version_id_key_value` (
              `datastore_version_id`,`key`,`value`),
CONSTRAINT `datastore_version_metadata_ibfk_1`
FOREIGN KEY (`datastore_version_id`)
REFERENCES `datastore_versions` (`id`)
ON DELETE CASCADE)

CREATE TABLE `datastore_versions` (
             `id` varchar(36) NOT NULL,
             `datastore_id` varchar(36) DEFAULT NULL,
             `name` varchar(255) DEFAULT NULL,
             `image_id` varchar(36) NOT NULL,
             `packages` varchar(511) DEFAULT NULL,
             `active` tinyint(1) NOT NULL,
             `manager` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `ds_versions` (`datastore_id`,`name`),
CONSTRAINT `datastore_versions_ibfk_1`
FOREIGN KEY (`datastore_id`)
REFERENCES `datastores` (`id`))

CREATE TABLE `datastores` (
             `id` varchar(36) NOT NULL,
             `name` varchar(255) DEFAULT NULL,
             `default_version_id` varchar(36) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `name` (`name`))

The code populates flavor to datastore-version mappings by using a key of ‘flavor’ in the table datastore_version_metadata. We propose to extend this capability by using a key of ‘volume-type’ in the same table.

We extend the trove-manage command and provide two new commands to add and delete a volume-type mapping. In addition it is proposed that two new commands be added to list existing associations. These list commands will exploit an existing method (list datastore version flavor associations).

Changes will be made to add a new mapping, see the section below on Public API for details. The endpoint will provide a list of all volume types allowed for a given tenant_id, datastore, and datastore version.

Suitable error messages will be provided by adding new exception classes. One error will be for a missing mapping between a given datastore-version and a volume-type, and the other will be for a situation where a mapping already exists.

The datastore version metadata model (trove/datastore/models.py) has code for handling the existing flavor mappings. This code will be refactored to extend the metadata mappings to also support volume type.

The service class that will be referenced in the WSGI endpoint will be added.

As we will be extending the existing metadata capability, no new database tables will be required.

During instance creation, a check will be added for the volme type being specified. If the volume type provided is not found to be a valid volume type for the specified datastore version, an error will be reported.

Currently volume type support does not extend to the cluster create API.

Since this change will return volume type information, service, model and view for VolumeType and VolumeTypes will be added.

If no datastore/version/volume-type has been configured then all volume-types known to cinder are allowed.

A python-troveclient command to list datastore/version/volume-type associations will be provided. It will call the new public API and show the results.

A trove-manage command to list datastore/version/volume-type associations will be provided. It will list only the associations in the database and not intersect with valid cinder volume-types.

The trove-manage command performs no validations, it does not check with cinder whether a volume-type being specified is actually a volume-type known to cinder or not. And the volume-type, once defined in trove could be deleted by cinder at a later time. It is therefore possible that the volume type association in the trove metadata table could become stale. Therefore, the implementation relies on the concept of defined mappings and allowed mappings.

A defined mapping is something specified by the user in the metadata table. An allowed mapping is the intersection of the defined mapping and valid cinder volume types.

If a datastore does not support volumes, all of the checking described in this specification is moot.

When the user has defined mappings for a particular datastore and version, (and volume support is enabled), we look to see whether any allowed mappings exist. If not, a distinct error message is generated indicating that no valid volume types could be found.

If there are defined volume types, we ensure that volume type is specified and is one of the allowed volume types.

New APIs will also be added to list all volume types available in cinder and also to show the details of a specified volume type.

Tests will be added to exercise the new code paths.

Issues

In the present flavor implementation (that is used as the template for this volume_type) implementation, there is a discrepancy between the values returned by the list_datastore_version_flavor_associations (datastore/models.py) method, and the flavors that will be accepted in the check found in the create() call.

This has been addressed in the code for volume type by instead calling the datastore/models method to list valid volume type associations.
The volume type was never stored with the instance and therefore when you show an instance, you can’t tell what volume_type was specified on create.
volume_type is not required in the create call. If an association is present for datastore/version/volume_type and no volume_type is specified for the create, then the cinder default is used. The option would be to make the volume_type required which would be a change to the API.

As implemented, if a volume_type is specified in create, it must be in the list of allowed volume types. Guessing the volume type is the alternative, and if more than one is allowed, a default would have to be provided. This has not been implemented.

Configuration

None

Database

No database changes will be required.

Public API

This change will add the following Public APIs:

List all volume types

This API will return all volume types as reported by cinder.

Request:

GET /{tenant_id}/volume-types

Response:

{
    "volume_types": [
        {
            "is_public": true,
            "description": "Example volume type",
            "id": <id>,
            "name": "example_volume_type"
        }
    ]
}

Show volume type details

This API will return the specified volume type’s details as reported by cinder.

Request:

GET /{tenant_id}/volume-types/{id}

Response:

{
    "volume_type": {
        "is_public": true,
        "description": null,
        "id": "53c71e3d-ef8f-4967-8b16-8a0ee6380d66",
        "name": "lvmdriver-1"
    }
}

List allowed datastore version volume types

This API will return all supported volume-types for the given combination of tenant_id, datastore, and datastore vesion. If no association has been configured then all volume types known to cinder are returned.

Request:

GET /{tenant_id}/datastores/{datastore}/versions/{version_id}/volume-types

Response:

{
    "volume_types": [
        {
            "is_public": true,
            "description": "Example volume type",
            "id": <id>,
            "name": "example_volume_type"
        }
    ]
}

Public API Security

There are no API security issues associated with this change.

Python API

No changes.

CLI (python-troveclient)

The trove-manage command will be the only affected as described above.

Internal API

None.

Guest Agent

None

Alternatives

One alternative that was considered was to associate datastores with supported volume types (not datastore-versions). This would have resulted in almost completely reimplementing the metadata linking scheme that was created for flavors.

Dashboard Impact (UX)

The create dialog volume type dropdown will be filtered to show only valid values based on the selected datastore version.

Implementation

Assignee(s)

Primary assignee:: mvandijk

Milestones

Target Milestone for completion:: N-Release-1

Work Items

As described above in detail in the “Proposed Change” section.

Upgrade Implications

There are no upgrade implications. We aren’t adding or changing the structure of any existing tables. The existing code can’t create mappings that could conflict with this proposed feature (all existing code creates mappings with a key of ‘flavor’).

Dependencies

None

Testing

New tests will be added.

Documentation Impact

Yes, documentation of the mechanism to add, delete and list mappings will be provided.

References

1: http://specs.openstack.org/openstack/trove-specs/specs/kilo/associate-flavors-datastores.html
2: https://review.openstack.org/#/c/109824/

Appendix

None

Configuration Group Management for DB2

Sun, 19 Apr 2020 00:00:00

This spec talks about how to enable configuration group management for DB2.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/db2-config-group

Problem Description

Trove supports configuration group management but this feature has not been implemented for DB2.

Proposed Change

DB2 supports both database manager configuration parameters and database configuration parameters. The database manager configuration parameters apply to an instance whereas the database configuration parameter applies to each database. In this spec, we talk about implementing configuration management for the DB2 database manager.

The database manager configuration parameters are stored in the db2systm file under the sqllib directory. To update the configuration parameters, DB2 recommends using the UPDATE DBM CONFIGURATION and RESET DBM CONFIGURATION commands instead of directly updating the config file. To see a complete list of configuration parameters, check the Appendix section.

Configuration

None

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

None

Guest Agent

Implement update_overrides and apply_overrides in the manager and service modules.

The following files will be updated:

/opt/stack/trove/trove/guestagent/datastore/experimental/db2/manager.py
/opt/stack/trove/trove/guestagent/datastore/experimental/db2/service.py
/opt/stack/trove/trove/guestagent/datastore/experimental/db2/system.py

The following file will be added:

/opt/stack/trove/trove/templates/db2/validation-rules.json

This will fit into the existing configuration manager framework. The existing ‘PropertiesCodec’ implementation will be reused to handle text-file operations. Configuration overrides will be implemented using the ‘ImportStrategy’ of the guestagent configuration manager.

Alternatives

None

Dashboard Impact (UX)

None

Implementation

Assignee(s)

mariamj@us.ibm.com

Milestones

Newton

Work Items

Implement configuration-related manager API calls

def update_overrides(self, context, overrides, remove=False)
def apply_overrides(self, context, overrides)

Write unit tests/integration tests were applicable

Upgrade Implications

None

Dependencies

None

Testing

Unit and integration tests will be added to test the new functionality

Documentation Impact

The documentation will be updated to reflect the new features supported by Trove for DB2.

References

1: https://www.ibm.com/support/knowledgecenter/SSEPGG_10.5.0/com.ibm.db2.luw.admin.config.doc/doc/c0060794.html

Appendix

The database manager configuration parameters have been listed below. For a detailed description of each parameter, please refer[1]_. The parameters listed below are the ones that are relevant for the DB2 Express-C version. The link referenced in 1 gives a complete list of parameters for the DB2 enterprise edition:

AGENTPRI
AGENT_STACK_SZ
ALTERNATE_AUTH_ENC
ASLHEAPSZ
AUDIT_BUF_SZ
CATALOG_NOAUTH
CLNT_KRB_PLUGIN
CLNT_PW_PLUGIN
COMM_EXIT_LIST
CUR_EFF_ARCH_LVL
CUR_EFF_CODE_LVL
DFT_ACCOUNT_STR
DFT_MON_BUFPOOL
DFT_MON_LOCK
DFT_MON_SORT
DFT_MON_STMT
DFT_MON_TABLE
DFT_MON_TIMESTAMP
DFT_MON_UOW
DIAGLEVEL
DIAGSIZE
DIR_CACHE
DISCOVER
DISCOVER_INST
FCM_NUM_BUFFERS
FCM_NUM_CHANNELS
FEDERATED
FED_NOAUTH
FENCED_POOL
GROUP_PLUGIN
HEALTH_MON
INDEXREC
INTRA_PARALLEL
JAVA_HEAP_SZ
KEEPFENCED
KEYSTORE_TYPE
LOCAL_GSSPLUGIN
MAX_CONNECTIONS
MAX_COORDAGENTS
MAX_QUERYDEGREE
MON_HEAP_SZ
NOTIFYLEVEL
NUMDB
NUM_INITAGENTS
NUM_INITFENCED
NUM_POOLAGENTS
RESYNC_INTERVAL
RQRIOBLK
SHEAPTHRES
SPM_LOG_FILE_SZ
SPM_MAX_RESYNC
SRVCON_AUTH
SRVCON_GSSPLUGIN_LIST
SRVCON_PW_PLUGIN
SRV_PLUGIN_MODE
SSL_CIPHERSPECS
SSL_CLNT_KEYDB
SSL_CLNT_STASH
SSL_SVCENAME
SSL_SVR_KEYDB
SSL_SVR_LABEL
SSL_SVR_STASH
SSL_VERSIONS
START_STOP_TIME
SYSADM_GROUP
SYSCTRL_GROUP
SYSMAINT_GROUP
SYSMON_GROUP
TM_DATABASE
TP_MON_NAME
TRUST_ALLCLNTS
TRUST_CLNTAUTH
UTIL_IMPACT_LIM
WLM_DISPATCHER
WLM_DISP_CONCUR
WLM_DISP_CPU_SHARES
WLM_DISP_MIN_UTIL

Guest agent controlled parameters:

ALT_DIAGPATH
AUTHENTICATION
DFTDBPATH
DIAGPATH
JDK_PATH
KEYSTORE_LOCATION

DB2 Full Online Backup and Restore

Sun, 19 Apr 2020 00:00:00

Currently in Trove, we support full offline backups for DB2 which is the default backup mechanism for DB2. This enables users to take full backups of DB2 databases when no applications are connected to or using these databases. While useful for cases where this downtime is okay, it is not very useful in a production type environment were applications are expected to be online all the time. DB2 provides a way to do online backups and this spec outlines how we can implement a full online backup mechanism in Trove.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/db2-online-backup

Problem Description

There are two types of backups we can perform using DB2 - offline and online. Offline backups require users and applications to be disconnected from the databases thereby requiring a certain down time. Online backups on the other hand, can be performed while users are still connected to databases. Trove currently supports full offline backups for DB2. In this spec, we explain how to enable full online backups for DB2 in Trove.

Proposed Change

As mentioned above, the difference between offline and online backup is that in the former case all applications must disconnect from the database while in the latter, applications can still be connected to databases while backup is being taken. In order to enable online backups, archive logging needs to be enabled. This allows applications to use the database even while backups are being taken. Any updates made to the database is recorded in the logs.

Logging is a mechanism where every updates to a database is recorded to transaction logs (also called active logs) as the data is being modified. There are 2 types of logging mechanism - circular and archive logging. By default, circular logging is enabled which means that after the last active log is filled, the first active log is overwritten. With circular logging as the default logging mechanism, only offline backups can be taken.

Using archive logging, the logs are stored in sequence and as logs get full, they are archived. Archive logging allows users to restore databases by roll forwarding to a particular point in time or a point before failure. Hence, users can restore a database from a backup image and roll forward to a particular point using logs thus getting it to a consistent state. For an introduction to archive logging, please refer 1 and for an overview of how transactional logging works in DB2, please refer 2.

To enable full online backup for Trove, the following steps need to be taken:

Create a directory to store the backups and archive logs - /home/db2inst1/db2inst1/backup/ArchiveLogs
When a database is created, if online backups is configured for DB2(through the backup_strategy config property), then configure the following database configuration parameter: ‘LOGARCHMETH1’ to point to a location where the archive logs can be stored using the command:

db2 update database configuration for <db> using LOGARCHMETH1 ‘DISK:/home/db2inst1/backup/ArchiveLogs’

Setting this database configuration parameter enables archive logging and stores the archive logs in a location different from the primary/active logs.Once this change is made, a full offline backup needs to be taken before any connections can be made to the database (this is because the database goes into a ‘BACKUP_PENDING’ state when this configuration change is made).

db2 backup database <db> to /home/db2inst1/db2inst1/backup

This backup can be deleted from the backup directory after this operation.
The online backup strategy for DB2 will comprise of the following actions:
- Check if archive logging is enabled for each database
- Check if there is enough space to store backups (which will include database and archived logs). The get_dbsize_info function that DB2 provides can be used to get the size of the database directory. We can query the SYSIBMADM.LOG_UTILIZATION table to get the log space used for each database.
- Do an online backup of the database and include the archive logs along with the backup image itself:
  
  db2 backup database <db> ONLINE to /home/db2inst1/db2inst1/backup INCLUDE LOGS
Once the backup is complete, compress/encrypt the backup inside the volume storage.
The compressed/encrypted file can then be stored in Swift under the database_backups container.
Once the backups are stored in Swift, we can go ahead and delete the backup files and archive logs from the backup directory.

Restoring a database includes restoring from an online backup image and rolling forward the logs to the end of the backup to make sure the database is in a consistent state.

Create a directory where the backup files and archive logs can be stored
Retrieve the backup from storage to the proper location
Use the following commands to restore the database and ensure it is in a consistent state: - db2 RESTORE DATABASE <db> FROM <backup_dir> LOGTARGET <log_dir>

The above command restores the database from the specified online backup directory and extracts the archive log files into the log_dir
- db2 ROLLFORWARD DATABASE <db> TO END OF BACKLOG AND COMPLETE
This command will roll forward the restored database to a consistent state by applying the archive logs. To see more details on the ROLLFORWARD command, please refer 3.

Configuration

The default configuration values of the following will need to change to correspond to the respective DB2 locations.

backup_strategy
backup_namespace
restore_namespace

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

None

Guest Agent

The DB2 guestagent will be modified to add support for full online backup and restore functionality. In addition, new strategies will also be added to support these new features.

The following guestagent files will be modified:

trove/guestagent/datastore/experimental/db2/service.py
trove/guestagent/datastore/experimental/db2/system.py

and the following new strategy files will be modified to add a new class for online backups:

trove/guestagent/strategies/backup/experimental/db2_impl.py

trove/guestagent/strategies/restore/experimental/db2_impl.py

New classes will be added for online backup strategy and the existing class needs to be renamed from DB2Backup -> DB2OfflineBackup

Alternatives

None

Dashboard Impact (UX)

None

Implementation

Assignee(s)

Mariam John (johnma)

Milestones

Newton

Work Items

There will only be one work item for this feature. This includes implementing the strategies for DB2 online backup and restore and make the necessary guestagent changes. This will also include test-cases necessary to test the new functionalities.

Upgrade Implications

None

Dependencies

None

Testing

Add new test cases to test online backup and restore functionality for DB2:
- Add new unit tests to test the newly implemented functionality
- Add a new DB2 helper class to the existing integration-test framework.

Documentation Impact

The datastore documentation should be updated to reflect the enabled features.

References

1: https://www.ibm.com/support/knowledgecenter/SSEPGG_10.5.0/com.ibm.db2.luw.admin.ha.doc/doc/c0051344.html
2: http://www.ibm.com/developerworks/data/library/techarticle/0301kline/0301kline.html
3: https://www.ibm.com/support/knowledgecenter/SSEPGG_10.5.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0001978.html

Appendix

None

Implement Couchbase cluster provisioning

Sun, 19 Apr 2020 00:00:00

Implement Couchbase cluster provisioning.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/couchbase-cluster-support

Problem Description

Couchbase is by nature a distributed database. Documents are distributed evenly across all nodes based on the hash of the document key. There are no query routers or configuration servers like in some other databases (MongoDB). Each document may have up to three replicas in addition to the active copy. What this means is that there will be three extra copies of that data within the cluster. Couchbase clients are provided with an updated ‘cluster map’ which they use to lookup data.

When a node gets added to the cluster the existing documents are incrementally redistributed within the cluster (process also known as rebalancing). Updated cluster maps are continuously provided to the clients. Any active client connections should not be interrupted throughout the process.

When a node gets removed from the cluster the replicas on the other nodes are promoted to the active and cluster maps updated so that the clients can keep serving the requests. The promoted documents are then re-replicated within the remaining nodes (rebalanced).

See References section for further details.

Proposed Change

This specification proposes the following cluster-related actions:

create
grow
shrink
delete
upgrade

Couchbase imposes cluster wide quota on the memory that get evenly distributed between node services. All nodes (including future nodes) need to be able to accommodate this quota. We therefore require the cluster to be homogeneous. The quota size is configurable by ‘cluster_ramsize_pc’ Trove configuration as the percentage of the total node RAM.

Create Cluster

Provision a requested number of instances while initializing them as Couchbase nodes (all nodes receive the same randomly generated Administrative credentials from the API).
Wait for the instances to become active.
Execute cluster initialization on the first node (coordinator).
Add the remaining nodes to the cluster via the coordinator node.
Wait for the cluster rebalance to finish.

Grow Cluster

Retrieve the current Administrative credentials from the first node (coordinator) of the existing cluster.
Provision and initialize a requested number of instances as Couchbase nodes with the current credentials (above).
Wait for the instances to become active.
Add the new nodes to the cluster via the coordinator node.
Wait for the cluster rebalance to finish.

Shrink Cluster

Load the removed instances.
Select one of the remaining nodes and use it as the coordinator.
Remove the nodes from the cluster via the coordinator.
Wait for the cluster rebalance to finish.
Delete decommissioned instances.

Configuration

The following configuration values will be implemented in the Couchbase configuration group:

cfg.BoolOpt('cluster_support', default=True,
            help='Enable clusters to be created and managed.'),
cfg.StrOpt('api_strategy',
           default='trove.common.strategies.cluster.experimental.'
           'couchbase.api.CouchbaseAPIStrategy',
           help='Class that implements datastore-specific API logic.'),
cfg.StrOpt('taskmanager_strategy',
           default='trove.common.strategies.cluster.experimental'
           '.couchbase.taskmanager.CouchbaseTaskManagerStrategy',
           help='Class that implements datastore-specific task manager '
                'logic.'),
cfg.StrOpt('guestagent_strategy',
           default='trove.common.strategies.cluster.experimental'
           '.couchbase.guestagent.CouchbaseGuestAgentStrategy',
           help='Class that implements datastore-specific Guest Agent API '
                'logic.'),
cfg.IntOpt('cluster_ramsize_pc', default=80, min=0, max=80,
           help='Per node RAM quota in for the Data service expressed as a'
           ' percentage of the available memory.'
           ' Minimum of 256MB will be used if the given percentage amounts'
           ' for less.'),

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

None

Guest Agent

The work will require some preliminary refactoring of the guest manager. This may also include tasks such that:

providing a common framework for executing Couchbase CLI calls The guest-agent currently executes only two CLI commands hardcoded in ‘system.py’ file. We will need to execute a greater variety of commands. This would include CouchbaseAdmin to execute CLI calls and handle results in a generic way. It will also facilitate reducing ‘system.py’ which we have been moving away from in other guests as well.

making the ramsize quota relative to the node’s total RAM Couchbase 3 requires a parameter to cluster-init is RAMSIZE quota. The valid value is between 256MB and 80% of available RAM. Make the default % a Trove variable.

using appropriate guestagent models for the Couchbase administrative user Couchbase should be using it’s own User model rather than relying on generic MySQL-object.

The following methods will be implemented in the CouchbaseGuestAgentAPI:

def initialize_cluster(self):
    LOG.debug("Configuring cluster parameters via node: %s" % self.id)
    self._call("initialize_cluster",
               guest_api.AGENT_HIGH_TIMEOUT, self.version_cap)

def get_cluster_password(self):
    LOG.debug("Retrieving cluster password from node: %s" % self.id)
    return self._call("get_cluster_password",
                      guest_api.AGENT_LOW_TIMEOUT, self.version_cap)

def get_cluster_rebalance_status(self):
    LOG.debug("Retrieving status of current cluster rebalance via node: %s"
              % self.id)
    return self._call("get_cluster_rebalance_status",
                      guest_api.AGENT_LOW_TIMEOUT, self.version_cap)

def add_nodes(self, nodes):
    LOG.debug("Adding nodes to the cluster: %s" % self.id)
    self._cast('add_nodes', self.version_cap, nodes=nodes)

def remove_nodes(self, nodes):
    LOG.debug("Removing nodes from the cluster: %s" % self.id)
    self._cast('remove_nodes', self.version_cap, nodes=nodes)

def cluster_complete(self):
    LOG.debug("Sending a setup completion notification for node: %s"
              % self.id)
    return self._call("cluster_complete", guest_api.AGENT_HIGH_TIMEOUT,
                      self.version_cap)

Alternatives

None

Dashboard Impact (UX)

Will need to enable Couchbase as a clustering datastore.

Implementation

Assignee(s)

Primary assignee:: Petr Malik <pmalik@tesora.com>

Milestones

Ocata

Work Items

The preliminary work in the guest manager and cluster strategy itself may be delivered in two separate commits if it facilitates smoother review process.

Upgrade Implications

None

Dependencies

None

Testing

Manager unittests will be added where appropriate.
The scenario tests already cover implemented functionality.

Documentation Impact

Datastore documentation for Couchbase will need to be updated to reflect clustering support.

References

Couchbase cluster management: http://docs.couchbase.com/admin/admin/Tasks/cluster-management.html
Couchbase CLI reference: http://developer.couchbase.com/documentation/server/4.0/cli/cli-intro.html
Couchbase cluster API: http://docs.couchbase.com/admin/admin/REST/rest-cluster-intro.html

Appendix

None

Trove Extended Filesystem Support

Sun, 19 Apr 2020 00:00:00

Trove currently only supports a single volume on a guest instance, and that volume must be a block device. This spec outlines a proposal for supporting multiple volumes, both block devices and filesystems, on the guest.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/extended-filesystems

Problem Description

In the current implementation, during instance creation the taskmanager allocates a single cinder volume and informs the guest during the prepare phase. The guest then formats the volume and mounts it as a volume to hold the data for the database.

Additional functionality is required:

Some datastores require more than a single volume
Guests should support both block devices and preformatted filesystems

Proposed Change

To support this new functionality, the taskmanager will pass to the guest a collection of volumes, each tagged as to their type and intended use. The taskmanager will be responsible for configuring the types of volumes to be attached to a guest and how those volumes will be provisioned with Cinder or Nova; the guest will be responsible for preparing the block device, if required, and connecting the device to the filesystem on the guest operating system.

Configuration

Three configuration values will be used to assist in attaching volumes to instances.

cg_volumes

The global cg_volumes configuration setting in the DEFAULT section of the taskmanager.conf file will contain a list of volume_types which support consistency groups. In future feature development, only volume types which support consistency groups will be available for snapshots and backups.

cg_volumes = NETAPP,SCALEIO

For Openstack allocated volumes, the values of the cg_volumes setting should correspond to the provider_type parameters in the sections that follow.

guest_device_type

The global guest_device_type configuration setting describes how the guest instance should deal with devices provided to it by the taskmanager.

cfg.DictOpt('guest_device_type',
            help="Dictionary of device types with help",
    default={
        'block_ext4': "Block storage to be formatted and managed by guest",
        'shared': "Shared storage, typically NFS, to be mounted as is"
    }
)

available_volume_kind

The per-datastore available_volume_kind configuration setting configures the volume_kinds available to a specific datastore:

cfg.DictOpt('available_volume_kind',
            help="Dictionary of available volume types with help",
    default={
        'data': "Storage volume for the database",
        'binlog': "Volume to hold binary logs for the database",
        'user_log': "Volume to hold user and error logs"
    }
)

In the initial implementation, this setting will be used to prevent spinning up a collection of instances in a cluster, only to find that an inappropriate volume configuration was selected.

/etc/trove/guest.conf

A new olso config file will be introduced on the guest. This file is to be installed on the guest image by the DIB elements and will contain configuration options specific to the operating system and database software installed on the guest.

The device_type configuration setting describes how the guest instance should prepare a volume before mounting, plus any other device specific configuration.

The volume_mapping configuration setting configures for each operating system how the guest should handle the volumes provided to it. The most common setting here would be where in the filesystem the device should be mounted. The mount_priority option, if supplied, is used to configure mount order in the event that devices need to be mounted beneath each other.

This description outlines the set of options defined specifically for extended filesystem support:

from oslo_config import cfg

device_type = [
    cfg.DictOpt('block_ext4', help="Block device to be formatted by guest",
        default={
            'raw_block_device': True,
            'format_options': 'ext4',
            'mount_type': 'ext4',
            'mount_options': '-o rw'
        },
    ),
    cfg.DictOpt('shared', help="Shared filesystem (NFS) to be used as is",
        default={
            'raw_block_device': False
            'mount_type': 'nfs',
            'mount_options': '-o rw'
        },
    )
]

volume_mapping = [
    cfg.DictOpt('data', help="Storage volume for the database",
        default={
            'mount_point': '/var/lib/mysql',
            'mount_priority': 1
        },
    ),
    cfg.DictOpt('binlog',
                help="Volume to hold binary logs for the database",
        default={
            'mount_point': '/var/lib/mysql/binlog',
            'mount_priority': 2
        },
    ),
    cfg.DictOpt('user_log', help="Volume to hold user and error logs",
        default={
            'mount_point': '/var/lib/mysql/userlog'
            'mount_priority': 2
        }
    )
]

Database

A new table volume_config will be created to describe the volumes to be configured. Each volume configuration will detail each volume to be attached to an instance. The volume_config table will have the following columns:

Column Name	Description
id	uuid identifier
name	Name of volume configuration
volume_kind	intended use - defined by datastore (data, binlog, userlog)
size	size of volume (unless overridden by user)
provider	Openstack component providing volume (“cinder”, “nova”)
provider_type	Volume type passed to Cinder/Nova
device_type	Indicates to guest how to prepare volume

For example, an instance which requires its data to be stored on a block device and its logs to be stored on a different volume would be configured as follows:

id	name	volume_kind	size	provider	provider_type	device_type
<uuid0>	vc1	root	3	cinder	vtype1	root
<uuid1>	vc1	data	10	cinder	vtype1	block_ext4
<uuid2>	vc1	binlog	2	cinder	vtype1	block_ext4

Instances which do not specify a volume_config to use in allocating the instance will use the volume_config named “default”. If no such volume_config exists, the instance will default to the current single Cinder volume.

Each volume defined in the volume_config will have a size specified for it. This size can be overridden by the user, either by the “–sysvol:<volume_kind>,size=<n>” option, or by the “–size” option for the data volume. This eliminates the current requirement that the “–size” option be specified for “trove create” or “trove cluster-create”; note that this means that should the user specify neither volume_config nor size when no “default” volume config is defined, the create call will fail in the taskmanager rather than the current behaviour of catching the discrepancy in the shell.

Two volume providers will initially be supported. If the “cinder” provider is selected, the volume will be allocated through the Cinder API. If the “nova” provider is specified, root volumes will be alloced on a root disk on the compute host and volumes with a volume_kind other than “root” will be allocated on Ephemeral storage. An error will be raised by Nova should the allocated volume sizes exceed the amount of storage configured in the instance flavor selected. Future volume provide support may include support for Manila volumes and/or multi-attach Cinder volumes.

A new table instance_volumes will be added to track the volumes associated with an instance. This will include both volumes created by the taskmanager based on the datastore volume configuration and volumes specified by the user.

Column Name	Description
instance_id	Id of instance to which volumes are associated
volume_kind	intended use - defined by datastore (data, binlog, userlog)
volume_id	Id of volume in Cinder or Manila (Null for ephemeral)
provider	Component providing volume (“cinder”, “nova”, “manila”)
device_type	Indicates to guest how to prepare volume

An instance configured as above would be represented by the following entries in the instance_volumes table:

volume_id	instance_id	volume_kind	provider	device_type
<volid0>	<inst1>	root	cinder	root
<volid1>	<inst1>	data	cinder	block_ext4
<volid2>	<inst1>	binlog	cinder	block_ext4

For backward compatibility, instances which have no entries in the instance_volumes table will be assumed to have their root volume allocated on Nova local storage.

Public API

volume_config and volume_list parameters will be added to the: payload of the following REST APIs:

create
cluster create
cluster grow

The volume_config parameter, if specified, will select the volume configuration to select from the volume_config table. If not specified, a root volume on Nova storage and a single Cinder data volume will be configured.

The volume_list will encapsulate the information from the –sysvol options. For each of these options, for backward compatibility, if the volume_list parameter is not included in the API payload, it will be computed from the volume parameter. It will be an error if both volume and volume_list (or neither) are specified.

The resize_volume API will be updated to include the volume_kind of the volume to be resized. For backwards compatibility, if no volume_kind is specified, the “data” volume will be resized. An error will be returned if the provider does not support the resize operation on the specified volume.

There will additionally be a new “GET volume_configs” API which will return a list of available volume configurations. These volume configurations will be the valid values to be specified for the above volume_config parameter.

Request:

GET v1/{tenant_id}/volume_configs
{
}

Response:

{
    [
        "default",
        "ora_prod"
    ]
}

Plus a new “GET volume_configs/<config>” API which will return details about a given volume_config.

Request:

GET v1/{tenant_id}/volume_configs/<config>
{
}

Response:

{
    "instance_id": <uuid>,
    "volume_kind": 'data',
    "volume_id": <uuid>,
    "provider": "cinder",
    "device_type": "block_ext4"
}

Plus a new “GET instance/<instance_id>/volumes” API which will return details about the volumes attached to a given instance.

Request:

GET v1/{tenant_id}/instance/<instance_uuid>/volumes
{
}

Response:

{
    [
        {
            'volume_id': <volid0>,
            'size': 8,
            'volume_kind': 'root',
            'provider': 'cinder',
            'device_type': 'root'
        },
        {
            'volume_id': <volid1>,
            'size': 50,
            'volume_kind': 'data',
            'provider': 'cinder',
            'device_type': 'block_ext4'
        },
        {
            'volume_id': <volid2>,
            'size': 2,
            'volume_kind': 'binlog',
            'provider': 'cinder',
            'device_type': 'block_ext4'
        }
    ]
}

Public API Security

No impact.

Python API

volume_config and volume_list parameters will be added to the: following python APIs:

Instance.create
Cluster.create
Cluster.grow

See above for descriptions of the volume_config and volume_list parameters.

The volume parameter will be deprecated in the above APIs.

The Instance.resize_volume python API will be updated to include the volume_kind of the volume to be resized.

A new volume_configs object will be added to the Trove python API. This client API will implement “list” and “show” methods.

CLI (python-troveclient)

Three new sub-commands will be added to the trove-manage command to support associating volumes with datastore versions:

$ trove-manage volume-config-add <name> <volume_kind> \
            <provider> <provider_type> <device_type> <required>

$ trove-manage volume-config-delete <name> <volume_kind>

$ trove-manage volume-config-list <name>

For example, to configure a volume configuration “ora_prod” to have a data volume and an optional binlog volume, both Cinder block devices, the following commands would be executed:

$ trove-manage volume-config-add ora_prod \
            data cinder ram_backed block_ext4 True
$ trove-manage volume-config-volume-add ora_prod \
            binlog cinder ram_backed block_ext4 False

A new –sysvol option will be added to several trove CLI commands to support the configuration of volumes created by the taskmanager. The –sysvol option may be specified multiple times to configure multiple volumes. Each specification will include the volume_kind for which a specification is being made followed by a colon separated list of configuration options - for the initial implementation, only size will be supported.

$ trove create mydb myflavor --datastore=foo --datastore_version=1.0 \
            --volume_config=ora_prod \
            --sysvol=data:size=5 --sysvol=binlog:size=2

For backwards compatibility, an alternative to this command would be one which specifies the size of the data volume with the “–size” option, though this may be deprecated in future releases:

$ trove create mydb myflavor –datastore=foo –datastore_version=1.0
–volume_config=ora_prod –size=5 –sysvol=binlog:size=2

A new option will be added to the resize-volume command to support specifying which of the volumes attached to the instance are to be resized:

$ trove resize-volume my_inst 3 --volume_kind binlog

For backwards compatibility, if the –volume_kind option is not specified, the volume of type “data” will be resized.

A new “trove volume-config-list” CLI command will be added to return a list of the names of the available volume_configs.

$ trove volume-config-list
+==============+
| VolumeConfig |
+==============+
| default      |
| ora_prod     |
+==============+

And a corresponding new “trove volume-config-show”.

$ trove volume-config-show ora_prod
===========  ====  ========  =============  ===========
volume_kind  size  provider  provider_type  device_type
===========  ====  ========  =============  ===========
root         3     cinder    vtype1         root
data         10    cinder    vtype1         block_ext4
binlog       2     cinder    vtype1         block_ext4
===========  ====  ========  =============  ===========

And of course a command to show which volumes are attached to a specific instance.

$ trove volumes-list <instance> =========== =========== ======== =========== ========= volume_kind size provider device_type volume_id =========== =========== ======== =========== ========= root 8 cinder root <volid0> data 50 cinder block_ext4 <volid1> binlog 2 cinder block_ext4 <volid2> =========== =========== ======== =========== =========

Internal API

The create_instance API will be updated to include the volume_info parameter.

The resize_volume API will be updated to include the id and volume_kind of the volume to be resized.

Guest Agent

A new prepare call will be implemented which replaces the device_path and mount_point parameters with a new device_config list. The device_config will supply information about the number and types of volumes to be configured on the guest:

[
    {
        'volume_kind': 'data',
        'device_type': 'block_ext4',
        'device_path': '/dev/vdb1'
    },
    {
        'volume_kind': 'binlog',
        'device_type': 'block_ext4',
        'device_path': '/dev/vdb2'
    },
]

To support backwards compatibility when the guest is upgraded before the guestagent API, the guest agent’s prepare method will take both old and new parameters. The correct behaviour will be deduced based on the values of the paramaters.

It is up to each datastore to define what volumes are supported, the meaning of each value of the volume_kind, and where each volume will be mounted on the local filesystem.

The guestagent resize_fs API will be updated to include the volume_kind parameter. The mount_point parameter will be ignored when the volume_kind parameter is specified.

As the parameter to the post_upgrade RPC endpoint is a dict, it will be updated to remain compatible with both old and new guests, providing both the mount point of the data volume plus the full device_config context.

Alternatives

Not available.

Dashboard Impact (UX)

The dashboard would be changed to allow the user to specify the volume_config on the instance.create, cluster.create, and cluster.grow functions. A volume.configs.list method will be added to the Python API to facilite this functionality.

A new panel/action will be added to the instance panel to show the volumes configured for a particular instance.

The volume_resize functionality will be updated to allow the user to specify the volume_kind that they wish to resize.

Implementation

Assignee(s)

Primary assignee:: 6-morgan
Dashboard assignee:: <launchpad-id or None>

Milestones

Target Milestone for completion:: post ocata

Work Items

implement taskmanager changes
implement base guestagent and guestagent.api changes
adapt datastores to new APIs
develop unit tests

Upgrade Implications

Backward compatibility for this feature will be dependent on the new RPC versioning feature.

Dependencies

Testing

An int-test should be developed to test adding multiple volumes for at least one datastore.

Documentation Impact

This feature will affect the following documents:

Installation
API
CLI
Building Guest Images for Openstack Trove

References

Appendix

Any additional technical information and data.

Adapt to file injection deprecation in nova

Thu, 29 Mar 2018 00:00:00

Trove currently uses –personality to inject files while booting an instance via Nova api. Unfortunately, this parameter is deprecated from Queens release, nova microversion 2.57. As bp Deprecate file injection 1 mentioned, the code to support it would remain, but having a microversion boundary would give Nova the ability to eventually remove the code in the future. In this spec, we try to adapt to file injection deprecation and implement another way to inject files into instance.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/adapt-to-file-injection-deprecation-in-nova

Problem Description

Nova deprecates file injection by popping out –personality parameter in request schema of microversion 2.57, then no value of –personality would be passed to Nova api service. It’s the current way using this parameter to inject files in Trove. We’re facing the risk that it would break Trove completely.

Proposed Change

Using –user-data is one of the alternatives to inject files while booting an instance.

Here is what we are going to do:

Reorganize function trove.instance.models.BaseInstance#get_injected_files to get the files contents, injected path, owner and permission, then build an InjectedFile object. In case of injecting multiple files into instance, we should build and return an InjectedFile object list.
Change trove.taskmanager.models.FreshInstanceTasks#_prepare_userdata. Build cloud-config scripts based on InjectedFile object list. If there is a <datastore_manager>.cloudinit script, e.g, mysql.cloudinit, we should detect the format of the script, and then convert it alongside with cloud-config scripts 2 into a mime multi part file 3 in case that we have to pass more than one type of data to cloud-init. So here should be two main helpers, one for organizing cloud-config scripts which looks like below:
```
#cloud-config
write_files:
-   encoding: b64
    content: CiMgVGhpcyBmaWxlIGNvbnRyb2xzIHRoZSBzdGF0ZSBvZiBTRUxpbnV4...
    owner: trove:trove
    path: /etc/trove/trove.conf
```
the other one for generating mime messages if there is any *.cloudinit scripts.

Note

Noted that currently we’re going to support ONLY two formats for *.cloudinit file, cloud-config and user-data script 4. It may be difficult for us to organize one user-data scripts based on one mime multi-part file or gzip compressed content, and other formats may not be used a lot. As for formats reference, please see 5 .
Use –user-data to pass the parameter we build above while booting an instance, then we can inject files into instance’s specified location, with proper owner and permission. One thing to clarify here is that according to configuration item:
```
cfg.BoolOpt('use_nova_server_config_drive', default=True,
            help='Use config drive for file injection when booting '
            'instance.'),
```
we enable –config-drive when calling novaclient to create a server by default.
Remove unused and deprecated codes.

Configuration

None

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

None

Guest Agent

None

Alternatives

None

Dashboard Impact (UX)

None

Implementation

Assignee(s)

Primary assignee:: Fan Zhang <zh.f@outlook.com>

Milestones

Target Milestone for completion:: Rocky-R2

Work Items

Add new InjectedFile object.
Add new helper functions.
Modify current code, including deprecating files parameter and changing to user-data.
Modify related code which uses –personality parameter to boot an instance.

Upgrade Implications

None

Dependencies

None

Testing

Current functional tests should cover the creating instance scenario.
Would need some new unit tests.

Documentation Impact

Docs needed for new configuration item usage.

References

1: https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/deprecate-file-injection.html
2: http://cloudinit.readthedocs.io/en/latest/topics/format.html#cloud-config-data
3: http://cloudinit.readthedocs.io/en/latest/topics/format.html#mime-multi-part-archive
4: http://cloudinit.readthedocs.io/en/latest/topics/format.html#user-data-script
5: http://cloudinit.readthedocs.io/en/latest/topics/format.html#formats

Appendix

None.

Implement root-enable/root-disable/root-show for Redis

Wed, 25 Oct 2017 00:00:00

Trove currently has support for enabling root user, disabling root user and showing root-enabled status for database instances, but that functionality is lacking for redis. This blueprint outlines a framework and API for implementing authentication management for redis.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/root-enable-in-redis

Problem Description

Trove does not currently support root-enable, root-disable, root-show for redis instances.

Proposed Change

Implement root-enable, root-disable and root-show API for redis. It will support two scenarios including redis single instance and redis replica instances which these operations only support master instance but can make effect on slave instances.

This implementation does not support a cluster of redis instances.

Here are the details:

Root-enable

Precondition checks to make sure that it’s not cluster or slave instance to execute this action.
Get slave instances of given instance.
Try to get original auth password for the sake of rolling back.
Do root enable for given instance. Considering it’s a single instance or a master of redis replica sets, roll back once using original auth password and raise exception if any error occurs.
If things go well, and there are some slave instances, get root password and use root password above to do root enable. Get failed slave instances’ id if any, store them into a list.
Return redis root created view alongside the failed slave instances’ id list.

Root-disable

Precondition checks to make sure that it’s not cluster or slave instance to execute this action.
Get slave instances of given instance.
Try to get original auth password for the sake of rolling back.
Do root disable for given instance. Considering it’s a single instance or a master of redis replica sets, roll back once using original auth password and raise exception if any error occurs.
If things go well, and there are some slave instances, do root disable. Get failed slave instances’ id if any, store them into a list.
If there are any failed slaves, return http code 200 with failed slaves list. If not, just return None with http code 204.

Root-show

Using root-show of mysql. It can satisfy what we ask for.

Configuration

Changing the follow configuration value:

cfg.StrOpt('root_controller',
           default='trove.extensions.redis.service.RedisRootController',
           help='Root controller implementation for redis.'),

Remove requirepass in trove/templates/redis/validation-rules.json to avoid modifying authentication by configuration group.

Database

None

Public API

Change the return contents of root-enable and root-disable. Since redis does not have a root user, just show ‘-‘ as user name. Return failed slaves if any. For example:

{
    "failed_slaves": [
        "67c2f6d6-7c01-4ce9-bb18-aa951ca5a39b"
    ],
    "user": {
        "password": "bdQhBXVpk7TE689aGgqdNmJmg4qHdpdBegae",
        "name": "-"
    }
}

Public API Security

None

Python API

None

CLI (python-troveclient)

Change the return content to meet API above.

Internal API

None

Guest Agent

The work will require some implementation of the guest manager:

Add requirepass and masterauth into redis.conf when executing root enable, and apply overrides to bring into effect without restarting redis service. Rebuild admin client to make sure guest agent can still talk to task manager after adding enabling/disabling root in redis, which has been done in this patch 1.
Remove requirepass and masterauth in redis.conf when executing root disable, and apply overrides to bring into effect without restarting redis service. Still, rebuild admin client.

Alternatives

We can achieve authentication management for redis by configuration group, but it looks like less convenient than using root-enable and root-disable.

Dashboard Impact (UX)

None

Implementation

Assignee(s)

Primary assignee:: fanzhang <zh.f@outlook.com>

Milestones

Target Milestone for completion:: Queens-Q2

Work Items

Implement redis root controller, service, models and views.
Implement redis guest agent, including manager, service and models.

Upgrade Implications

None

Dependencies

None

Testing

Inside the int_tests.py, root_actions_groups will be added to redis supported groups in order to reuse some scenario tests with proper modification to meet the root actions functionality of redis.

Unittests will be added to test the derived controller functionality, for example:

Test root-enable on single redis instance.
Test root-enable on master instance of redis replication.
Test root-enable on slave instance of redis replication.
Test root-enable with is_cluster=True.
Test root-delete in the cases of above.

Unittests will be also added to test the implemented functions inside guest- agent including enable_root and disable_root.

Documentation Impact

The documentation 2 should be updated to add the following features:

Successful response examples of enabling/disabling root in redis.
Failed response examples of enabling/disabling root in redis.

But it is not mandatory to add.

References

1: https://bugs.launchpad.net/trove/+bug/1708376
2: https://developer.openstack.org/api-ref/database/#users-users

Appendix

None.

Sample Pike Spec for Trove

Tue, 24 Oct 2017 00:00:00

Introduction paragraph – what is the motivation for the spec/blueprint? (Don’t forget to change the title above to something more relevant.)

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/name-of-blueprint

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

If your specification proposes any changes to the Trove REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.openstack.org/#/q/status:open+project:openstack/trove-specs+message:apiimpact,n,z

Code snippets, etc. should be placed in appropriately marked blocks:

# This is a bash command
ls -lf

# sample code
for count in range(1, 10):
    print count

Configuration

Does this impact any configuration files? If so, which ones?

Database

Does this impact any existing tables? If so, which ones? Are the changes forward and backward compatible? Be sure to include the expected migration process

Public API

Does this change any API that an end-user has access to? Are there any exceptions in terms of consistency with other APIs?

Public API Security

If this change proposes a new API, or if this change relates to security on an existing API, provide details here.

What are the expectations of, and implications to security on the Public API.

Python API

Does this change the Python API? If anything was removed, has it been properly marked as deprecated?

CLI (python-troveclient)

Will the Trove CLI need to be modified? If the CLI will just implement the changes mentioned in the Python API section, it may be enough to just mention it here.

Internal API

Does this change any internal messages between API and Task Manager or Task Manager to Guest?

Guest Agent

Does this change behavior on the Guest Agent? If so, is it backwards compatible with API and Task Manager?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Dashboard Impact (UX)

This section should detail how the dashboard (Horizon) should display the new changes, if relevant. For example, if adding cluster support for Redis, this section could say:

Enabling Redis clustering will simply reuse the existing Launch Cluster
dialog. The Redis datastore will be in the datastore pulldown. When the
user selects Redis the Launch Cluster dialog fields will dynamically
change to display the default Launch Cluster fields.

There will be a new detail overview panel with Redis cluster specific
information.

There are no additional actions to be added at this point for the Redis
cluster.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a spec where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Dashboard assignee:: <launchpad-id or None>

Can list additional ids if they intend on doing substantial implementation work on this spec.

Milestones

Target Milestone for completion:: eg. Liberty-1

Work Items

Work items or tasks – break the feature up into the things that need to be done to implement it. Those parts might end up being done by different people, but we’re mostly trying to understand the timeline for implementation.

Upgrade Implications

In this section, describe the upgrade implications (if any) of the proposed change. This could include such details as:

changes to location of files, or layout of the source tree if this impacts configuration files,
invalidates old backups,
changes the CLI in a manner that could impact existing scripting,
eliminates or adds new notifications (events),
any changes that an operator or user must perform as part of the upgrade.

If the change has upgrade implications, also remember to:

add the DocImpact keyword to the commit, and
provide sufficient information in the commit message or in the documentation bug that gets created.

For more information about the DocImpact keyword, refer to https://wiki.openstack.org/wiki/Documentation/DocImpact

Note: Documentation for the CLI commands are automatically generated from the help strings when a new version of the CLI is released, so a DocImpact keyword is not typically required for python-troveclient changes.

Dependencies

Include specific references to specs and/or blueprints in Trove, or in other projects, that this one either depends on or is related to.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Please discuss how the change will be tested. We especially want to know what int tests and tempest tests will be added. It is assumed that unit test coverage will be added so that doesn’t need to be mentioned explicitly, but discussion of why you think unit tests are sufficient and we don’t need to add more tempest tests would need to be included.

Documentation Impact

What is the impact on the docs team of this change? Some changes might require donating resources to the docs team to have the documentation updated. Don’t repeat details discussed above, but please reference them here.

References

Please add any useful references here. You are not required to have any references. Moreover, this specification should still make sense when your references are unavailable. Examples of what you could include are 1, 2, 3 and 4.

1: Links to mailing list or IRC discussions
2: Links to notes from a summit session
3: Links to relevant research, if appropriate
4: Anything else you feel it is worthwhile to refer to

Appendix

Any additional technical information and data.

Trove support in python-openstackclient

Mon, 31 Oct 2016 00:00:00

Implement a new set of trove commands as python-openstackclient plugins.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/trove-support-in-python-openstackclient

Problem Description

python-openstackclient is becoming the default command line client for many OpenStack projects. Trove would benefit from implementing all of its client commands as a single python-openstackclient plugin implemented in the python-troveclient repository.

Proposed Change

The intent of this spec is to identify the commands to be implemented and establish conventions for command and argument names. This spec is not intended to be a full and correct specification of command and argument names. The details can be left to the code reviews for the commands themselves.

The following conventions will be adopted for command names:

As per the OpenStackClient convention, the command name shall always take the following form:

openstack [<global-options>] <object-1> <action> [<object-2>] \
          [command-arguments]

The following conventions will be adopted for arguments and argument flags:

Single character flags will be avoided as per the openstack convention, except for very common arguments.
When the database or cluster name and ID are specified it will be the first and second positional arguments respectively after the full command names.
When an argument is required it will be a positional argument.

The following trove commands are already implemented for openstack:

trove secgroup-add-rule
openstack security group rule create

trove secgroup-delete-rule
openstack security group rule delete

trove secgroup-list
openstack security group list

trove secgroup-list-rules
openstack security group rule list

The following trove commands will be implemented for openstack initially suggesting these command names:

trove backup-create <instance> <name>
openstack database backup create <instance> <name>

trove backup-delete <backup>
openstack database backup delete <backup>

trove backup-list
openstack database backup list

trove backup-list-instance <instance>
openstack database backup list <instance>

trove backup-show <backup>
openstack database backup show <backup>

trove cluster-create <name> <datastore> <datastore_version>
openstack database cluster create <name> <datastore> <datastore_version>

trove cluster-delete <cluster>
openstack database cluster delete <cluster>

trove cluster-list
openstack database cluster list

trove cluster-modules <cluster>
openstack database module list cluster <cluster>

trove cluster-show <cluster>
openstack database cluster show <cluster>

trove configuration-attach <instance> <configuration>
openstack database configuration add <instance> <configuration>

trove configuration-create <name> <values>
openstack database configuration create <name> <values>

trove configuration-default <instance>
openstack database configuration show <instance>

trove configuration-delete <configuration_group>
openstack database configuration delete <configuration_group>

trove configuration-detach <instance>
openstack database configuration remove <instance>

trove configuration-instances <configuration_group>
openstack database configuration list --instance <configuration_group>

trove configuration-list
openstack database configuration list

trove configuration-parameter-list <datastore_version>
openstack database configuration parameter list <datastore_version>

trove configuration-parameter-show <datastore_version> <parameter>
openstack database configuration parameter show <datastore_version> <parameter>

trove configuration-patch <configuration_group> <values>
openstack database configuration set --patch <configuration_group> <values>

trove configuration-show <configuration_group>
openstack database configuration show <configuration_group>

trove configuration-update <configuration_group> <values>
openstack database configuration set <configuration_group> <values>

trove create <name> <flavor>
openstack database cluster create <name> <datastore> <datastore_version>

trove database-create <instance> <name>
openstack database add <instance> <name>

trove database-delete <instance> <database>
openstack database remove <instance> <database>

trove database-list <instance>
openstack database list <instance>

trove datastore-list
openstack datastore list

trove datastore-show <datastore>
openstack datastore show <datastore>

trove datastore-version-list <datastore>
openstack datastore version list <datastore>

trove datastore-version-show <datastore_version>
openstack datastore version show <datastore_version>

trove delete <instance>
openstack database cluster delete <cluster>

trove detach-replica <instance>
openstack database replica unset <instance>

trove flavor-list
openstack database flavor list

trove flavor-show <flavor>
openstack database flavor show <flavor>

trove limit-list
openstack database limits list

trove list
openstack database instance list
openstack database cluster list

trove module-apply <instance> <module>
openstack database instance set module <instance> <module>

trove module-create <name> <type> <filename>
openstack database module create <name> <type> <filename>

trove module-delete <module>
openstack database module delete <module>

trove module-instances <module>
openstack database instance list module <module>

trove module-list
openstack database module list

trove module-list-instance <instance>
openstack database module list instance <instance>

trove module-query <instance>
openstack database instance list module <instance> --status

trove module-remove <instance> <module>
openstack database instance remove module <instance> <module>

trove module-retrieve <instance>
openstack database instance show module <instance>

trove module-show <module>
openstack database module show <module>

trove module-update <module>
openstack database module set <module>

trove resize-instance <instance> <flavor>
openstack database instance resize <instance> <flavor>
openstack database cluster resize <cluster> <flavor>

trove resize-volume <instance> <size>
openstack database instance volume resize <instance> <size>
openstack database cluster volume resize <cluster> <size>

trove restart <instance>
openstack database instance restart <instance>
openstack database cluster restart <cluster>

trove root-enable <instance_or_cluster>
openstack database set --root <instance_or_cluster>

trove root-show <instance_or_cluster>
openstack database show --root <instance_or_cluster>

trove show <instance>
openstack database cluster show <cluster>

trove update <instance>
openstack database cluster set <cluster>

trove user-create <instance> <name> <password>
openstack database user create <instance> <name> <password>

trove user-delete <instance> <name>
openstack database user delete <instance> <name>

trove user-grant-access <instance> <name> <databases>
openstack database user add --access <instance> <name> <databases>

trove user-list <instance>
openstack database user list <instance>

trove user-revoke-access <instance> <name> <database>
openstack database user remove --access <instance> <name> <database>

trove user-show <instance> <name>
openstack database user show <instance> <name>

trove user-show-access <instance> <name>
openstack database user show --access <instance> <name>

trove user-update-attributes <instance> <name>
openstack database user set --attributes <instance> <name>

Configuration

None

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

A new directory named osc will be created under /troveclient/osc for the OpenStackClient plugin and the commands mentioned above.

Internal API

None

Guest Agent

None

Alternatives

None

Dashboard Impact (UX)

None

Implementation

Assignee(s)

Primary assignee:: twm2016
Dashboard assignee:: None

Milestones

Target Milestone for completion:: Ocata-3

Work Items

CLI commands as stated above. Integration tests Functional tests

Upgrade Implications

None

Dependencies

python-openstackclient osc-lib

Testing

Functional tests will be located in: /troveclient/tests/osc/ Functional testing will test the inputs and outputs of listed commands. Integration tests will verify the OpenStackClient plugin is working with other projects. These should be placed in /trove/tests/tempest/tests/api

Documentation Impact

OpenStack Client adoption list will be updated to include python-troveclient.

References

http://docs.openstack.org/developer/python-openstackclient/commands.html http://docs.openstack.org/cli-reference/trove.html

Appendix

None

1 Secure oslo.messaging.rpc message

Sun, 30 Oct 2016 00:00:00

Contents

Trove utilizes oslo_messaging.rpc to perform RPC calls and the transport underlying this is oslo_messaging. Messages sent on oslo.messaging are currently treated as genuine. There is a benefit to adding a layer of validation that will ensure that the RPC calls are in fact genuine. We propose that the RPC calls be encrypted with unique keys.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/secure-oslo-messaging-messages

1.1 Problem Description

Messages sent on oslo.messaging are currently treated as genuine by the recipient. Given that the names of the topics used are predictable, it is possible for a person with sufficient knowledge of Trove to, for example, compromise a guest instance or otherwise obtain credentials to connect to RabbitMQ (or the underlying transport to oslo-messaging) and then generate messages to, for example, the task manager by impersonating the API service. While there are already safeguards in place to contain the scope of this, such as by requiring that the message contain a valid keystone token with the appropriate access, this is still a point of vulnerability.

Currently, when a client wishes to make an asynchronous RPC (cast()), the method name and parameters are marshalled and sent down to oslo_messaging.rpc. It is the responsibility of oslo_messaging.rpc to transmit the information to the remote side, and then find and invoke the method specified. After the cast() is invoked on the client side, the next thing that is seen by the consumer of oslo_messaging.rpc is an invocation of the desired method on the server side.

The same thing happens for a synchronous RPC (call()) with the additional step of the client blocking, the server completing the operation and sending a response to the client, and the client receiving that and unblocking.

1.2 Proposed Change

After experimenting with several other alternative approaches, we propose to implement custom serializers (and deserializers) which can be provided to oslo_messaging.rpc.

All messages sent and return values in RPC call() will be serialized through these custom methods which will encrypt the content. Due to a bug Failure to use serializer in exception if an RPC function throws an exception, the exception is not encrypted.

1.2.1 How does TroveRPCDispatcher verify legitimacy of a message

The proposed implementation relies on cryptography and unique keys for the control plane and the guests. We propose to use symmetric keys for the purpose of encryption.

Trove has the following entities who are party to RPC invocations:

Trove API Service (client)
Trove Taskmanager Service (client and server)
Trove Conductor Service (server)
Trove Guestagent (client and server)

When an RPC call() or cast() is made, the client invokes the serializer which will encrypt all arguments. When received on the server side, oslo_messaging.rpc will invoke the deserializer which will decrypt the arguments.

It is assumed that the control plane is secure and the control plane symmetric key is secure. If it is compromised, then all bets are off.

In communication with the guest agent, each guest has a unique symmetric key that is generated by the control plane and passed to the guest at launch.

1.2.2 Securing the response

As described earlier, a response to a call() method will be secured in the same way as the request. As observed earlier, due to a bug, an exception thrown by an RPC function is not (currently) being serialized and will therefore be returned unencrypted. When (if) that bug is fixed in oslo_messaging.rpc, this exposure is minimized.

1.2.3 Contol plane key

The control plane key is constructed at system initialization time. The key is stored on the control plane (in the configuration file).

If the control plane consists of multiple machines, then the control plane services on all machines must have access to the control plane key.

1.2.4 Getting keys to the guest instance

On instance launch, the guest key is created and passed to the guest as an injected file. We assume that the mechanism for file injection is secure in that it cannot be intercepted and compromised by a bad actor.

A unique key is created for each instance.

1.2.5 Why is this secure?

We make two assumptions above; these are:

The control plane is secure, the control plane key is not compromised, and
The transmission of the guest key to the guest is secure and is not compromised.

These are, meaningful and reasonable assumptions to make given the architecture of an OpenStack system.

Should a guest be compromised, the bad actor can connect to the underlying transport (say Rabbit) but all they will be able to see are encrypted messages that they cannot decrypt.

1.2.6 Configuration

The control plane key is stored on the control plane in a secure way and there are configuration options to tell each service where to find it.

Each guest instance will have a key and that will be stored securely on the instance and a configuraiton setting will tell the guestagent where to find it.

cfg.StrOpt('tm_rpc_encr_key',
           default='bzH6y0SGmjuoY0FNSTptrhgieGXNDX6PIhvz',
           help='OpenSSL aes_cbc key for taskmanager RPC encryption.'),
cfg.StrOpt('inst_rpc_key_encr_key',
           default='emYjgHFqfXNB1NGehAFIUeoyw4V4XwWHEaKP',
           help='OpenSSL aes_cbc key to encrypt instance keys in DB.'),
cfg.StrOpt('instance_rpc_encr_key',
           help='OpenSSL aes_cbc key for instance RPC encryption.'),

1.2.7 Database

The guest key for each guest instance will be stored in the database. A table instance_keys is proposed for this.

Field	Type	Null	Key	Default	Extra
id instance_id encrypted_key created updated deleted deleted_at	varchar(64) varchar(64) varchar(255) datetime datetime tinyint(1) datetime	NO NO NO NO NO NO YES	PRI UNI	NULL NULL NULL NULL NULL NULL NULL

The guest instance keys are encrypted and stored in the encrypted_key column. A foreign key constraint links instance_id with instances.id. A unique constraint on instance_id is placed on this table.

1.2.8 Public API

No changes to the public API.

1.2.9 Public API Security

No changes.

1.2.10 Python API

No changes.

1.2.11 CLI (python-troveclient)

No changes.

1.2.12 Internal API

The internal API (from the perspective of developers, and invocations) will remain unaffected by this change as the implementation seeks to work below the Trove code entirely. As a result, the internal API will be radically different, and code must be in place to ensure that encrypting and non-encrypting clients and servers know how to interoperate.

1.2.13 Guest Agent

The guestagent will receive its key as a part of the configdrive/boot process and can use it to decrypt all messages.

1.2.14 Alternatives

Several alternatives were considered, prototyped, and abandoned. A short summary of each is provided below.

We proposed to the oslo_messaging.rpc team to implement a lightweight message signing and encryption mechanism in their code by providing a mechanism of callbacks which would allow the consumer (trove) to perform the signing and encryption. The oslo_messaging team did not want to go this route as they felt that the message included other private data structures which we (the consumer) could modify and cause unexpected behavior.
We proposed that that oslo_messaging.rpc allow consumers to provide a custom dispatcher for messages on the receiver side. With this implementation, a signature or message encryption could be performed on the client side and intercepted on the server side and reversed allowing us to have minimal changes on the server side. Again, the oslo_messaging.rpc team felt that the dispatcher was a private data structure and they did not feel that we should be encapsulating it.
We prototyped and experimented with a change where each RPC endpoint would be decorated and the decorator would provide a mechanism to construct the proper parameters and the invocation to the RPC method. The client side change would be identical to (b) but the server side change would involve a change to every RPC method to add the decorator. In addition, the call context would not be encrypted in this approach and it was abandoned.
We were advised that we should NOT be using oslo_messaging.rpc the way we are using it as it was only intended for use on the control plane. And that we should instead make the guest an RPC server. Unfortunately that’s not what we need? In Trove, the guest agent is an extension of the control plane and not well suited to a REST based communication strategy. What we need is an RPC mechanism, and it is sad that oslo_messaging.rpc can’t seem to provide a secure one.

1.3 Dashboard Impact (UX)

None.

1.4 Implementation

1.4.1 Assignee(s)

Primary assignee:: amrith
Dashboard assignee:: none

1.4.2 Milestones

Ocata-1

1.4.3 Work Items

Implementing code on control plane and guest
Implement changes to devstack plugin to create control plane key
Implement unit tests
Implement upgrade handling
Update documentation

1.5 Upgrade Implications

Minimal upgrade implications are anticipated, code is proposed that handles this transition.

The control plane key will be generated and persisted on all control plane nodes.
When guests are upgraded a key will be sent to them as part of the nova migrate process.

The API’s will be rev’ed one major version to account for this.

1.6 Dependencies

There is an assumed dependency on the RPC API versioning which has now merged.

1.7 Testing

Oh yeah, we’ll need some of this.

1.8 Documentation Impact

And some of this; details to follow.

1.9 References

Failure to use serializer in exception: https://bugs.launchpad.net/oslo.messaging/+bug/1648254

1.10 Appendix

Any additional technical information and data.

Trove Policy Support

Tue, 25 Oct 2016 00:00:00

Trove needs to provide users with more fine-grained control over which users/roles can access which APIs. The Oslo Policy library provides support for RBAC policy enforcement across all OpenStack services.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/trove-policy

Problem Description

Trove currently does not have a unified way of role-based access control. It needs to provide users with more fine-grained control over which users/roles can access which APIs.

Proposed Change

Add Oslo policy check calls on all user-facing APIs. 1 Also see Appendix for the list of proposed rules.

The checks will be implemented by means of Oslo policy ‘enforce’ call at the beginning of each Trove API.

The call will be given extra information, parent ‘tenant_id’ (AKA the owner), on the target object (e.g. deleted instance in trove-delete API, updated configuration group in configuration-patch API). This will allow users to use this information within their rules.

Actions that do not have a particular target (e.g. trove-create, trove-list) will get the tenant itself as the target.

Actions that involve multiple rules will check all of them simultaneously. One good example of this is trove-create. If the policy does not allow creating users or applying modules the end user should not be allowed to create a new instance with initial users and modules applied either.

The Policy engine used will be >= 1.9.0 which supports new registered policy rules. While being fully backwards-compatible the registered rules allow for more robust development.

Configuration

None

Database

None

Public API

All API calls may rise ‘PolicyNotAuthorized’ (HTTP 403) if the request is not authorized by the policy framework. The default access rules will be set to mimic the current behavior (i.e. users can freely execute operations on their own tenant).

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

None

Guest Agent

None

Alternatives

None

Dashboard Impact (UX)

None

Implementation

Assignee(s)

Petr Malik <pmalik@tesora.com>

Milestones

Ocata-1

Work Items

Work will be delivered in a single patch set.

Upgrade Implications

None

Dependencies

Python library ‘oslo.policy>=1.9.0’ will be required.

Testing

Unittests will be added to cover the policy framework. Scenario tests will be testing the default behavior (matching the existing behavior).

Documentation Impact

The exposed policy rules and policy.json file should be documented (see Appendix).

References

1: Information on the rule engine and policy.json file http://docs.openstack.org/mitaka/config-reference/policy-json-file.html

Appendix

Proposed contents of ‘policy.json’ (Note: datastore and flavor APIs are unrestricted by default):

{
    "admin_or_owner":  "role:admin or is_admin:True or tenant:%(tenant)s",
    "default": "rule: admin_or_owner",

    "instance:create": "rule:admin_or_owner",
    "instance:delete": "rule:admin_or_owner",
    "instance:index": "rule:admin_or_owner",
    "instance:show": "rule:admin_or_owner",
    "instance:update": "rule:admin_or_owner",
    "instance:edit": "rule:admin_or_owner",
    "instance:restart": "rule:admin_or_owner",
    "instance:resize_volume": "rule:admin_or_owner",
    "instance:resize_flavor": "rule:admin_or_owner",
    "instance:reset_password": "rule:admin_or_owner",
    "instance:promote_to_replica_source": "rule:admin_or_owner",
    "instance:eject_replica_source": "rule:admin_or_owner",
    "instance:configuration": "rule:admin_or_owner",
    "instance:guest_log_list": "rule:admin_or_owner",
    "instance:backups": "rule:admin_or_owner",
    "instance:module_list": "rule:admin_or_owner",
    "instance:module_apply": "rule:admin_or_owner",
    "instance:module_remove": "rule:admin_or_owner",

    "instance:extension:root:create": "rule:admin_or_owner",
    "instance:extension:root:delete": "rule:admin_or_owner",
    "instance:extension:root:index": "rule:admin_or_owner",

    "instance:extension:user:create": "rule:admin_or_owner",
    "instance:extension:user:delete": "rule:admin_or_owner",
    "instance:extension:user:index": "rule:admin_or_owner",
    "instance:extension:user:show": "rule:admin_or_owner",
    "instance:extension:user:update": "rule:admin_or_owner",
    "instance:extension:user:update_all": "rule:admin_or_owner",

    "instance:extension:user_access:update": "rule:admin_or_owner",
    "instance:extension:user_access:delete": "rule:admin_or_owner",
    "instance:extension:user_access:index": "rule:admin_or_owner",

    "instance:extension:database:create": "rule:admin_or_owner",
    "instance:extension:database:delete": "rule:admin_or_owner",
    "instance:extension:database:index": "rule:admin_or_owner",
    "instance:extension:database:show": "rule:admin_or_owner",

    "cluster:create": "rule:admin_or_owner",
    "cluster:delete": "rule:admin_or_owner",
    "cluster:index": "rule:admin_or_owner",
    "cluster:show": "rule:admin_or_owner",
    "cluster:show_instance": "rule:admin_or_owner",
    "cluster:action": "rule:admin_or_owner",

    "cluster:extension:root:create": "rule:admin_or_owner",
    "cluster:extension:root:delete": "rule:admin_or_owner",
    "cluster:extension:root:index": "rule:admin_or_owner",

    "backup:create": "rule:admin_or_owner",
    "backup:delete": "rule:admin_or_owner",
    "backup:index": "rule:admin_or_owner",
    "backup:show": "rule:admin_or_owner",

    "configuration:create": "rule:admin_or_owner",
    "configuration:delete": "rule:admin_or_owner",
    "configuration:index": "rule:admin_or_owner",
    "configuration:show": "rule:admin_or_owner",
    "configuration:instances": "rule:admin_or_owner",
    "configuration:update": "rule:admin_or_owner",
    "configuration:edit": "rule:admin_or_owner",

    "configuration-parameter:index": "rule:admin_or_owner",
    "configuration-parameter:show": "rule:admin_or_owner",
    "configuration-parameter:index_by_version": "rule:admin_or_owner",
    "configuration-parameter:show_by_version": "rule:admin_or_owner",

    "datastore:index": "",
    "datastore:show": "",
    "datastore:version_show": "",
    "datastore:version_show_by_uuid": "",
    "datastore:version_index": "",
    "datastore:list_associated_flavors": "",
    "datastore:list_associated_volume_types": "",

    "flavor:index": "",
    "flavor:show": "",

    "limits:index": "rule:admin_or_owner",

    "module:create": "rule:admin_or_owner",
    "module:delete": "rule:admin_or_owner",
    "module:index": "rule:admin_or_owner",
    "module:show": "rule:admin_or_owner",
    "module:instances": "rule:admin_or_owner",
    "module:update": "rule:admin_or_owner"
}

1 Eliminate trove-integration and redstack

Fri, 21 Oct 2016 00:00:00

Contents

In the very beginning, when Trove was integrated into OpenStack, one of the subjects of some contention was the redstack tool, and the fact that Trove had a somewhat non-standard structure in the form of a trove-integration repository.

It was, I understand, one of the stipulations of the Technical Committee at the time that this be normalized.

For a number of reasons, (which we will describe in the Problem Description section) this structure is proving to be quite a nusiance and it would be beneficial to the project to do this now.

The goal of this project is to eliminate the trove-integration project and the redstack tool through a normal process of deprecation. That is to say that existing releases, through Newton, will continue to be driven and may (I stress, may) use the tool, but new versions from Ocata will not.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/eliminate-trove-integration-and-redstack

1.1 Problem Description

The trove-integration project was created as a place to store scripts to create, manipulate and test DBaaS vm’s, and to house the scripts used to manage the development environment. The redstack tool was the swiss army knife, it could provision your development environment, make guest images, load them, and later run tests as well.

On the whole, a very useful thing, there is no doubt about that.

It did introduce several serious problems.

trove-integration is not versioned

It has only a master branch. Therefore all code that is version sensitive has to accomplish that through crafty branching constructs.

trove-integration is a repository outside the trove repository

This brings with it dependencies on the trove repository, such as when a new datastore needs to be added, or a new version of a datastore is to be added. Trove ‘elements’ go into trove-integration but get used by trove.

The CI is very painful

Since trove-integration isn’t branched, we have to test each trove-integration branch against not just master, but also stable branches.

The versioning just got more complex

We now have to deal with multiple operating systems (ubuntu and fedora), multiple versions (liberty, mitaka, newton, ocata, …), different versions of operating systems (trusty, and now xenial). The code is now convoluted enough that attempting to add support for xenial (as I had to do for SQL Server) became a nightmare.

There is no easy way to build images

This is the #1 problem faced by users of Trove; how do I build images. A solution was proposed (https://review.openstack.org/#/c/312806/) that would address this through the creation of a trove-image-builder repository. This solution has some merits and demerits, we choose here to propose a different solution (see second bullet in proposed end state) that will address that problem as well.

redstack is ‘non-standard’

The current mode/phase approach provided by devstack is standardized across all projects, and redstack is one of the few (if not the only) non-standard approach. We already support the devstack plugin and redstack (in the provisioning a stack use-case) is nothing more than a thin veneer on devstack.

1.2 Proposed Change

The proposed change is in multiple parts, and several of the sections don’t match the typical headings one expects in a blueprint that is aimed at a new feature or function in Trove. Therefore the sections may be marked as Not Applicable. Additional sections have been added where required.

One thing that is important to highlight is that while this change is highly disruptive, and moves a whole lot of code around, we are largely talking about moving around a couple of hundred files (elements) from one place to another, and leveraging a devstack plugin that we already use extensively. There will be an impact on the tests that are in trove-integration but we already do most of our testing out of the trove repository.

Changes that have been proposed in the trove-integration repository; which are pending review and merge will be negatively impacted. But that will be the case no matter when we choose to make a change like this. Most of the elements are highly parameterized and the changes actually required to make this all work are minimal.

I feel that the best way to describe this proposed change is to begin with a description of the proposed ‘end-state’.

1.2.1 Proposed End State

The elements from trove-integration will get folded into the trove repository
The elements will now be versioned along with the server, they are in the same repository. Tagging the Ocata release will also cause the release of a set of elements for the release and similarly the tagging of the Pike release will cause the release of a set of elements for that release.

A set of tools will be provided that will make this easier, more description of that below.
A set of sample configuration files will be provided that will cause devstack to install and configure Trove. A user can copy these into their own localrc, or use the commands to set the right environment variables before invoking devstack/stack.sh
The trove devstack plugin will have additional code in the stack/extra step that will build and load a set of guest images of the users choosing. This image or images will be built using the elements from the trove repository.

This will cause the standard demo user to be used (not alt_demo).

In running devstack, one must configure a configuration file for devstack and one specifies the datastore whose image is to be built and loaded in that configuration file. I was initially contemplating allowing for the specification of a comma-seperated list but a single datastore may be a good place to start. This datastore would also be set as the default.

One devstack environment can operate with multiple datastores. Three commands (trove-image-builder.bash, load-trove-image.bash and activate-trove-image.bash) will be provided. The first will build the image for you. The second will load it into glance and register it with Trove, the third will set a specified datastore/version to the default.
The trove devstack plugin will have additional code in the stack/post-config phase that will configure the Trove test environment, add the right flavors, and do the other things that are required for Trove testing.
Scenario tests that are already in the trove repository will be run without the use of the redstack tool.
Releases Newton and before may continue to use trove-integration and redstack, releases Ocata and forward shall not use trove-integration and redstack. For example, new elements for Ocata and the future will not go into trove-integration. This means that during this transition (which may last for two releases) some changes may have to go to both trove-integration, and trove.
The Ocata CI will have a new set of jobs that will leverage this new state, no longer using redstack for its provisioning, and the cross version testing that is going on in trove-integration will no longer need to be supported.

1.2.2 Configuration

Some sample configuration files will be provided and these will help a user configure their localrc before running stack.sh. This will cause devstack to completely provision Trove, and additionally build and load guest images as instructed by the user.

1.2.3 How we get there

Getting to the desired end state will be a multi-step exercise.

Step 1

Take everything in the trove-integration repository at some specific point in time and drop it into the trove repository. For convenience, I’m going to drop it into a folder called integration.

It would be nice to do this and preserve history but that is going to be quite a pain to accomplish. An email has been sent to the infra mailing list requesting help with this but in the intermin, the commit https://review.openstack.org/#/c/384195/ does what we need (sans histories).

Step 2

Make changes to the stuff now in the trove repository and make it such that the redstack command (which will be renamed to trovestack) can be executed just as it used to be in the trove-integration branch. This also includes some small amendments to the documentation.

Commit https://review.openstack.org/#/c/384746/ does this.

Step 3

Refactor the CI jobs to use trovestack in the trove tree instead of redstack from the trove-integration tree. This means all the CI jobs (or maybe most) will become legacy jobs and new jobs will be created for Ocata and higher.

Step 4

There is a huge amount of cruft in trovestack that handled the fact that redstack in trove-integration was not versioned and released with each OpenStack version. We should do this cleanup because sooner rather than later as we will soon have to handle the Ubuntu 16.04 transition and having that dimension in addition to the OpenStack version dimension will be very difficult to manage.

Step 5

Take the image build aspect of trovestack (formerly redstack) and make it an independent tool. https://review.openstack.org/#/c/374952/ began this process and took the elements out of the place where they were copied in from trove-integration and starts to make it a standalone tool. We will update trovestack to use this tool and avoid the duplication of the image building logic.

Take the image loading logic (including the setting of the default datastore) and make that a standalone tool and modify trovestack to use that tool.

Step 6

Refactor the CI to automatically build images for different datastores and verify that the image build process is proper. This will be a new job.

Step 7

Refactor the integration tests into the tests directory.

1.2.4 Database

Not applicable

1.2.5 Public API

Not applicable

1.2.6 Public API Security

Not applicable

1.2.7 Python API

Not applicable

1.2.8 CLI (python-troveclient)

Not applicable

1.2.9 Internal API

Not applicable

1.2.10 Guest Agent

The guest agent is already part of the trove repository and will remain there. It will now be in the same versioned repository as the server code which has dependencies on it.

1.2.11 Alternatives

There are several options that have been considered, some at great length.

Do nothing

Well, we’ve done that for three years and it hasn’t been fun. I think we can safely say that there are strong reasons to disqualify this option.

Just get the image build out of redstack, leave the rest

This was part (at least) of the approach behind setting up the trove-image-builder repository. It would be an improvement over the present state but still would have cross repository dependencies. Considering that there is no clear justification for releasing trove-image-builder at a different cadence than trove, I see no justification in having an independent repository.

We know that there is a tight coupling, and there likely will be one for a long time longer, between the server side of Trove and the guest agent/images. Having two repositories just makes handling that a pain.

Use the trove-image-builder approach that was proposed earlier.

We could use the approach that was proposed earlier (https://review.openstack.org/#/c/312806/) but this solution has the same kind(s) of challenges that one faces with trove-integration. Two repositories, releasing them in sync, etc.,

Having elements in the trove repository is a better alternative.

1.3 Dashboard Impact (UX)

No dashboard impact is expected.

The dashboard will be enabled (if the user enables the plugin). Code already exists for that in the trove plugin.

1.4 Implementation

1.4.1 Assignee(s)

Primary assignee:: amrith
Dashboard assignee:: <none>

1.4.2 Milestones

Target Milestone for completion:: Ocata-1

1.4.3 Work Items

Move elements from trove-integration to trove
Write wrapper (trove-image-builder) script for simplicity of use
Create sample configuration file settings
Modify CI jobs

1.5 Upgrade Implications

This change has no upgrade implications as we are talking largely about refactoring a development infrastructure.

1.6 Dependencies

None

1.7 Testing

We will test the creation of each of the images, and putting them through their paces in the CI with our existing scenario tests.

1.8 Documentation Impact

Internal (developer) documentation will need to be updated.

1.9 References

None

1.10 Appendix

None

Datastore-specific API controllers

Tue, 18 Oct 2016 00:00:00

Improve user and database API extensions to handle datastore-specific properties and allow them to work with clusters.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/datastore-specific-api-extensions

Also see 1

Problem Description

User and database commands are currently routed through MySQL-specific API extensions. This causes several problems with input validation which is MySQL-specific and does not work well with other datastores. It also forces MySQL-specific properties on models and views which do not have (or have different) meaning on non-MySQL-like datastores.

Another long-lasting problem has been that the above APIs do not work with cluster instances.

Proposed Change

This document proposes introduction of generic API controllers.

Any datastore-specific parsing, handling and validations would be provided in derived classes. Trove would load the appropriate derived-controller dynamically based on the target datastore of the API call.

The goal is to provide a consistent experience to the Trove user. The base controller implementations would provide the execution flow and handle common Trove validations (e.g. check for existence, duplicates) and notifications.

The derived classes would provide datastore-specific parsing and model validations (e.g. username validation, parsing datastore-specific properties from the payload). Datastore-specific views would map models properties on the output payload.

The default controller would also include cluster interface which would, by default, route all request to one (controller) instance of the cluster leaving the rest on the cluster itself. This strategy proved successful with many if not all currently existing datastores. Datastores may override the node selection or entirely re-implement any of the calls shall it be necessary.

We would avoid introduction of new (cluster) API endpoints (e.g. trove cluster-user-create) at this time. It is not clear whether that is required at all for the above APIs and it draws yet another distinction between clusters and single Trove instances which may not be warranted.

This work would be entirely internal to the API engine. The existing endpoints (guest-agent or user facing) would not be affected.

Configuration

There would be a datastore-specific property for each controller implementation.

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

The CLI would be modified to accept cluster IDs for the related commands.

Internal API

There can be only a single controller per ReST API endpoint.

This would be a Rounting Controller. This controller would be responsible for detecting the target instance’s datastore and loading the derived controller implementation for it (see Configuration). It would then simply pass the request on this derived controller.

The derived controller would extend the Base implementation with the same interface. Generally it would just parse the payload and construct datastore-specific model objects and views. The base implementation would handle the generic validations and execution flow.

The base implementation would also be responsible for detecting the instance type (i.e. single or cluster) and routing the cluster requests to the cluster interface which would then pass it down to one (controller) instance from the cluster. The controller instance would be selected from the nodes exposed by the cluster strategy (i.e. nodes displayed by “trove cluster-instances” command).

Guest Agent

None

Alternatives

None

Dashboard Impact (UX)

None

Implementation

Assignee(s)

Petr Malik <pmalik@tesora.com>

Milestones

Ocata-1

Work Items

Implement the base controllers.

Implement derived controllers.

Switch datastores to use their respective derived controllers.

Upgrade Implications

None

Dependencies

None

Testing

Unittests will be added to test the base and derived controller functionality.

Documentation Impact

None

References

1: Related bug: https://bugs.launchpad.net/trove/+bug/1498573

Appendix

None

Cluster Upgrade

Fri, 12 Aug 2016 00:00:00

Trove currently has support for upgrading since instances from one datastore version to another, but that functionality is lacking for clusters. This blueprint outlines a framework and API for implementing upgrades for clusters.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/cluster-upgrade

Problem Description

Trove does not currently support upgrading clusters to new datastore versions.

Proposed Change

Implement a new cluster-upgrade API for upgrading a cluster to a new datastore version.

This blueprint will outline only the framework and API for implementing cluster upgrades. It will not detail the implementation of upgrade for any specific datastore as the implementation for each datastore may be different.

https://review.openstack.org/#/q/status:open+project:openstack/trove-specs+message:apiimpact,n,z

Code snippets, etc. should be placed in appropriately marked blocks:

# This is a bash command
ls -lf

# sample code
for count in range(1, 10):
    print count

Configuration

No datastore agnostic configuration settings are envisioned. Setting for specific datastores will be detailed in the specifications for each datastore implementation.

Database

No database changes are envisioned.

Public API

A new REST API will be implemented for cluster-upgrade:

Request:

PATCH v1/{tenant_id}/cluster/{cluster_id}
{
    "cluster":
    {
        "datastore_version": "<datastore_version_uuid>"
    }
}

Response:

{
}

REST return codes:

- Accepted.
- BadRequest. Server could not understand request.
- Not Found. <datastore_version_id> not found.

Public API Security

No security implications.

Python API

A new method will be implemented in the trove API. This method will upgrade a cluster to the image specified by the provided datastore_version.

upgrade(cluster, datastore_version)

cluster: the cluster to upgrade
datastore_version: the datastore version, or its id, to which the trove cluster will be upgraded

CLI (python-troveclient)

A new CLI call will be implemented. This new call will upgrade a cluster to the image specified by the provided datastore_version.

trove cluster-upgrade <cluster> <datastore_version>

cluster: the cluster to upgrade
datastore_version: the datastore version to which the instance will be upgraded

Internal API

The implementation of upgrade for single instances will be used to upgrade the guest agent on an instance.

Guest Agent

For the initial implementation it is expected that the existing pre and post upgrade methods will suffice.

Alternatives

Dashboard Impact (UX)

A new cluster action will be implemented to allow a cluster to be upgraded. Said functionality will be similar to the functionality for a single instance.

Implementation

Assignee(s)

Primary assignee:: 6-morgan
Dashboard assignee:: <launchpad-id or None>

Milestones

Target Milestone for completion:: eg. Liberty-1

Work Items

The implementation of this has been posted and is ready for review subject to this spec being approved.

Upgrade Implications

No upgrade implications.

Dependencies

Testing

No int tests will be included with this change as this is only a framework without implementations for specific datastores.

Documentation Impact

References

None.

Appendix

None.

Module Management Ordering

Tue, 12 Jul 2016 00:00:00

Modules currently are applied in any order, however operators may wish to control the order that modules are applied to ensure proper functionality.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/module-management-ordering

Problem Description

Trove now supports module management, in that ‘modules’ containing a payload can be applied to running Trove instances. This allows end users the ability to ‘bring your own license’ for datastores (among other things), as long as a corresponding driver plugin exists. The original implementation doesn’t take into account the case where one module must be applied before another (for example, if multiple licenses must be applied in the correct sequence).

The ability for an operator to ensure that ‘admin’ modules are applied before user modules is also lacking.

This spec addresses these shortcomings.

Proposed Change

A method for specifying ‘priority’ modules, plus a way to rank the order in which modules are applied would be created. Two new attributes ‘priority_apply’ and ‘apply_order’ would need to be added to the payload on create and update. In addition, an is_admin flag will be added as an automatic attribute, set when someone with admin credentials creates a module. This will allow better control on the driver plugin side with regards to security concerns, etc.

The default for ‘priority_apply’ will be False and the default for ‘apply_order’ will be 5. This will allow modules to be ordered before or after if the options are not supplied.

Configuration

No new configuration changes are required.

Database

New columns will be added to the modules table of the Trove schema:

Column

Type

Allow Nulls

Description

priority_apply

tinyint(1)

No

Should this module be applied before all non-priority ones. Admin only option.

apply_order

int(11)

No

Order that modules should be applied in. Value between 0-9 with lower order numbers applied first. Priority modules can also ordered this way.

is_admin

tinyint(1)

No

This module was created or updated by a user with admin credentials. Once this flag is set, only an admin user can subsequently update the module.

Column	Type	Allow Nulls	Description
priority_apply	tinyint(1)	No	Should this module be applied before all non-priority ones. Admin only option.
apply_order	int(11)	No	Order that modules should be applied in. Value between 0-9 with lower order numbers applied first. Priority modules can also ordered this way.
is_admin	tinyint(1)	No	This module was created or updated by a user with admin credentials. Once this flag is set, only an admin user can subsequently update the module.

Creating modules that have ‘priority’ will require admin credentials. All priority modules will be applied before non-priority ones, and will follow the same apply_order sequence.

For example, given modules with the following priority/order, they will be applied in the following order:

==============      ===========
Priority Apply      Apply Order
==============      ===========
Yes                 0
Yes                 4
Yes                 9
No                  0
No                  1
No                  9
==============      ===========

Public API

The following new options will be added to the payload of the module-create and module-update ReST APIs:

Request:

POST /v1.0/modules (or PATCH /v1.0/modules/{module_id})
{
    <current payload>,
    'priority_apply': True,
    'apply_order': 5
}

Response:

{
    "module": {
        <current payload>,
        'priority_apply': True,
        'apply_order': 5
    }
}

Response Codes:

The response codes will remain the same

The is_admin flag will be set automatically, and as such will not need to be passed in the payload. If a module created by a non-admin is updated by an admin, the is_admin flag will be set but only if an ‘admin-only’ option is turned on. Once a module is designated as ‘admin’ then only a user with admin credentials can modify it from that time forward.

Public API Security

This change is not expected to introduce any security concerns.

Python API

New arguments to the module create and update methods will be added to facilitate the ordering.

def module_create(self, module_type, name, description, contents,
                  datastore, datastore_version='all', auto_apply=False,
                  all_tenants=False, visible=True, live_update=False,
                  priority_apply=False, apply_order=5):
    """Create a new module."""

def module_update(self, module, module_type=None, name=None,
                  description=None, contents=None, datastore=None,
                  datastore_version=None, auto_apply=None,
                  all_tenants=None, visible=None, live_update=None,
                  priority_apply=False, apply_order=5):
    """Update an existing module."""

CLI (python-troveclient)

The following Trove CLI commands will support two new arguments, priority_apply and apply_order. priority_apply will be a flag that requires admin credentials, and apply_order will be restricted to an integer between 0-9.

module-create Creates a new module resource.
module-update Updates module details for a particular module
resource.

Internal API

The internal API will change in that new fields will be included in the module structure. No coding changes are anticipated though.

Guest Agent

In the Guest Agent, the modules will be ordered based on the priority and order values. No other changes are anticipated with the exception that the is_admin flag will now be taken from the module payload (if it exists) instead of being inferred from other attributes.

Alternatives

None

Dashboard Impact (UX)

The module detail panel will need to have two new attributes: priority-apply and apply-order. The first as a flag (defaulting to false) and the second restricted to integer values between 0 and 9. These should be added to the python call to create or update a module.

Implementation

Assignee(s)

Primary assignee:: [peterstac]

Milestones

Newton

Work Items

The work will be undertaken with the following tasks:

Client (Python and CLI) changes

Guest Agent changes to ensure the modules are ordered correctly

Upgrade Implications

No upgrade issues are expected, however the Trove database will need to be updated for the feature to work.

Dependencies

None.

Testing

Scenario tests will be enhanced to include ordering (including at least one priority module). It may be difficult to test that the ordering is adhered to (since this would require some sort of dependency and there is only a ping driver plugin) so this may have to be handled by unit tests only.

Documentation Impact

The fact that modules can now be ordered should be added to the documentation.

References

None

Appendix

None

Quota Management

Wed, 22 Jun 2016 00:00:00

A proposal to expose the existing Quota.update API from the management API to the user through the non-management API.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/quota-management

Problem Description

There is a Quota.update API in the management API to allow an operator to change the quota allocation for a resource. Unfortunately, there is no CLI to access that call.

Proposed Change

Add the v1.Quotas class to the v1.client so that an operator can execute a command to change the allocation for a resource.

Configuration

No impact.

Database

No impact.

Public API

No impact (the REST API already exists).

Public API Security

The added call will be restricted to admin users.

Python API

Adds client.quotas.update to the python client:

def update(self, id, quotas):
    """Set limits for quotas."""

This call takes a dict indicating the change to be made:

trove_client.quotas.update(project.id, {‘instances’: 10})

Also adds client.quotas.show to the python client:

def show(self, id):
    """Shows usage information for quota managed resources."""

This call takes the id of the project and returns all resources with their usage:

trove_client.quotas.show(project.id)

CLI (python-troveclient)

Two new shell commands will be added:

$ trove quota-show <project id>

$ trove quota-update <project id> volumes 50

Internal API

No impact.

Guest Agent

No Impact.

Alternatives

Dashboard Impact (UX)

This will be available only via the command line.

Implementation

Assignee(s)

Primary assignee:: 6-morgan

Milestones

Target Milestone for completion:: Newton

Work Items

Patch already delivered to gerrit.

Upgrade Implications

None.

Dependencies

Testing

Tests already exist for this functionality.

Documentation Impact

References

Appendix

Any additional technical information and data.

Scheduled Backups

Tue, 14 Jun 2016 00:00:00

Trove currently has no way to schedule backups of instances. This document proposes to implement scheduled backups by utilizing Mistral workflows.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/scheduled-backup

Problem Description

Trove does not support scheduled backups.

Proposed Change

Scheduled backups will be supported by utilizing Mistral workflows 1. The Trove python API will be extended to provide a Trove centric interface to Mistral workflows.

A Trove workbook will be loaded into Mistral to support calling the trove.backup_create python API via a schedule. Trove python API methods will be added to support scheduling backups and listing scheduled backups and their executions. Using these Trove commands a user will be able to manage backup schedules (create/delete/list, etc.) and view the executions of said schedules. Like is done with Nova flavors however, Trove will not actually execute any commands itself, just pass them down to Mistral. Any errors that occur will be visible when viewing the executions, however will be limited to whatever Mistral exposes.

All changes will be within the python-troveclient module with the exception of the workbook. This will be available in the Trove repo to (possibly) be loaded by the devstack plugin if Mistral is enabled.

See the Appendix for a sample Trove workbook.

Configuration

No configuration changes.

Database

No database changes.

Public API

No changes to the REST API.

Public API Security

No security impact.

Python API

Note: The referenced ‘cron pattern’ is the standard cron expression. 2

def schedule_create(self, instance, pattern, name,
                    description=None, parent_id=None,
                    mistral_client=None):
    """Create a new schedule to backup the given instance.

    :param instance: instance to backup.
    :param pattern: cron pattern for schedule.
    :param name: name for backup.
    :param description: (optional).
    :param parent_id: base for incremental backup (optional).
    :returns: :class:`Backups`
    """

def schedule_list(self, instance, mistral_client=None):
    """Get a list of all backup schedules for an instance.

    :param: instance for which to list schedules.
    :rtype: list of :class:`Schedule`.
    """

def schedule_show(self, schedule, mistral_client=None):
    """Get details of a backup schedule.

    :param: schedule to show.
    :rtype: :class:`Schedule`.
    """

def schedule_delete(self, schedule, mistral_client=None):
    """Remove a given backup schedule.

    :param schedule: schedule to delete.
    """

def execution_list(self, schedule, mistral_client=None,
                   marker='', limit=None):
    """Get a list of all executions of a scheduled backup.

    :param: schedule for which to list executions.
    :rtype: list of :class:`ScheduleExecution`.
    """

def execution_delete(self, execution, mistral_client=None):
    """Remove a given schedule execution.

    :param execution: id of execution to remove.
    """

CLI (python-troveclient)

Load trove workbook

$ mistral workbook-create ~/mwj/mistral/trove.yaml

Create Schedule

$ trove schedule-create m "*/2 * * * *" myback
+---------------------+----------------------------------------------------------------------------------------------------------------+
| Property            | Value                                                                                                          |
+---------------------+----------------------------------------------------------------------------------------------------------------+
| created_at          | 2016-06-14 14:47:16.865731                                                                                     |
| id                  | fb149a29-be9b-49c1-a2f7-ca6c1213896f                                                                           |
| input               | {"instance": "5328f62a-d999-4be3-90bb-83cc6af4469c", "description": null, "parent_id": null, "name": "myback"} |
| instance            | 5328f62a-d999-4be3-90bb-83cc6af4469c                                                                           |
| name                | myback                                                                                                         |
| next_execution_time | 2016-06-14 07:48:00                                                                                            |
| parent_id           | None                                                                                                           |
| pattern             | */2 * * * *                                                                                                    |
+---------------------+----------------------------------------------------------------------------------------------------------------+

List Schedule

$ trove schedule-list m
+--------------------------------------+--------+-------------+---------------------+
| ID                                   | Name   | Pattern     | Next Execution Time |
+--------------------------------------+--------+-------------+---------------------+
| fb149a29-be9b-49c1-a2f7-ca6c1213896f | myback | */2 * * * * | 2016-06-14 07:50:00 |
+--------------------------------------+--------+-------------+---------------------+

Show Schedule

trove schedule-show fb149a29-be9b-49c1-a2f7-ca6c1213896f
+---------------------+----------------------------------------------------------------------------------------------------------------+
| Property            | Value                                                                                                          |
+---------------------+----------------------------------------------------------------------------------------------------------------+
| created_at          | 2016-06-14 14:47:16                                                                                            |
| id                  | fb149a29-be9b-49c1-a2f7-ca6c1213896f                                                                           |
| input               | {"instance": "5328f62a-d999-4be3-90bb-83cc6af4469c", "description": null, "parent_id": null, "name": "myback"} |
| instance            | 5328f62a-d999-4be3-90bb-83cc6af4469c                                                                           |
| name                | myback                                                                                                         |
| next_execution_time | 2016-06-14 07:52:00                                                                                            |
| parent_id           | None                                                                                                           |
| pattern             | */2 * * * *                                                                                                    |
| updated_at          | 2016-06-14 14:49:59                                                                                            |
+---------------------+----------------------------------------------------------------------------------------------------------------+

Delete Schedule

$ trove schedule-delete fb149a29-be9b-49c1-a2f7-ca6c1213896f

List Executions

trove execution-list fb149a29-be9b-49c1-a2f7-ca6c1213896f
+--------------------------------------+---------------------+---------+-------------------------------+
| ID                                   | Execution Time      | State   | Output                        |
+--------------------------------------+---------------------+---------+-------------------------------+
| 38ef2289-4330-4554-8574-4b5351f69713 | 2016-06-14 14:49:59 | SUCCESS | {"status": "Backup complete"} |
| 3713355a-b65e-44e0-ac43-150b863a6e6e | 2016-06-14 14:51:59 | SUCCESS | {"status": "Backup complete"} |
+--------------------------------------+---------------------+---------+-------------------------------+

Delete Executions

$ trove execution-delete 3713355a-b65e-44e0-ac43-150b863a6e6e

Internal API

No internal API changes.

Guest Agent

No guest changes.

Alternatives

The alternative would be for Trove to implement its own scheduling mechanism.

Dashboard Impact (UX)

An action will be added to the instance actions pulldown to show the scheduled backups for an instance. This will lead to a chain of panels for scheduled backups, details of a schedule, and executions of a schedule. The scheduled backups list will have an option to create a new schedule. The panels which list schedules and executions will have an action to delete the corresponding resource.

Implementation

Assignee(s)

Primary assignee:: 6-morgan
Dashboard assignee:: duktesora

Milestones

Target Milestone for completion:: Newton

Work Items

already prototyped 3
implement unit tests

Upgrade Implications

No upgrade implications.

Dependencies

n/a

Testing

If we add support in scenario tests (through the python API) that would mean we’d require Mistral to be installed. As such, no scenario tests will be created.

Documentation Impact

New python API methods and CLI commands would need to be documented.

References

1: https://wiki.openstack.org/wiki/Mistral
2: https://en.wikipedia.org/wiki/Cron#CRON_expression
3: https://review.openstack.org/#/c/329160/

Appendix

This is an idea of what the Trove workbook for Mistral would look like:

---
version: '2.0'

name: trove

description: Trove Workflows

workflows:

  backup_create:
    input: [instance, name, description, parent_id]
    output:
      status: <% $.message %>

    tasks:
      backup_create:
        action: trove.backups_create instance=<% $.instance %> name=<% $.name %> description=<% $.description %> parent_id=<% $.parent_id %>
        publish:
          message: <% 'Backup complete' %>

Instance Upgrade

Wed, 06 Apr 2016 00:00:00

Trove needs to address how guest instances will update their database software, the Trove guest agent, and the underlying operating system. This blueprint proposes a technique which will allow guest instances to be updated by migrating the data volume to a new instance based on an updated guest image.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/instance-upgrade

Problem Description

Trove creates a database instance by creating a nova instance from a provided operating system image and attaching a volume to the instance to hold the user’s data. By using a known image with the database software preconfigured, an operator can ensure that the user will receive a known version of database software, configured with an operating system certified to work with that version of database software. This technique eliminates many of the ambiguities of provisioning applications, allowing an operator to provide a published level of service to the database user.

Unfortunately, running database instances aren’t static over their lifetimes. Over time, it may be necessary for an instance to be upgraded with security patches, new database software, or updates to the Trove guest agent. The cloud operator and the end user must be provided with a way to migrate their databases to new operating environments.

In summary, there are a number of changes that a database instance might be subjected to over time, and this spec addresses applying these types of changes. These include (but are not necessarily limited to):

operating system security patches
database security patches
database version upgrades
Trove guest agent upgrades
OpenStack version upgrades
operating system upgrades

Proposed Change

The traditional approach to system upgrades is to simply execute system utilities such as “apt-get upgrade” to upgrade all or parts of a running system. For large installations, this technique could be automated by tools such as Ansible or Salt to apply the changes to many systems. This approach isn’t ideal for database-as-a-service as there is little control over which components are upgraded, and to which version; an update could leave a database instance in a state where the database does not operate properly, or could even cause the data itself to become corrupted.

It is crucial that Trove develop a method to update instances which provides a high probability of leaving the database in a safe, consistent, and usable state.

To ensure a safe, consistent state, this blueprint proposes a technique for upgrading an instance which transitions it from one known image to a new known image running updated software. The full state of each instance state will be known. At its core, this technique will involve detaching the volume containing the database data and attaching it to a new instance based on a different (updated) image.

The bulk of the “variable” data on a Trove instance exists on the attached volume where the database stores its data. There is also some operating system state which must be preserved during this transition, consisting primarily of networking data (the instance name and ip address), some Trove configuration data (guest_info.conf and trove-guestagent.conf), and the database configuration data (such as /etc/mysql).

The core of this technique will be based on the rebuild API provided by Nova. The Nova rebuild function upgrades a running instance to a new image, ensuring that the networking configuration is applied to the new instance. The newly created instance will come up with the same network configuration (ip, hostname, and Neutron configuration) as the instance from which it was created. Trove guest functionality will be added to allow the guest to capture additional state about the instance (primarily database configuration), and re-apply that state to the system after rebuild. The result will be a running, connectable database provisioned on a new instance running updated software.

The selection of image to upgrade to will be based on the current datastore_version mechanism. Essentially, the user will be allowed to upgrade an instance to a newer datastore_version of its datastore. Safeguards will be put in place to ensure that the user is selecting an appropriate datastore_version to upgrade to; for example, the user would only be allowed to upgrade a mysql instance from datastore_version 5.5 to 5.6, not to 5.0 or 6.7.

Configuration

No configuration changes are planned.

Database

A future blueprint will outline the changes necessary to support the upgrade constraints on datastore_versions; this specification will be limited to implementing the upgrade functionality without the constraints.

Public API

A new REST API call will be added in support of this functionality.

REST API: PATCH /instances/<instance id> REST body:

{
    "instance": {
        "datastore_version": "<datastore_version_uuid>"
    }
}

REST result:

{}

REST return codes:

202 - Accepted. 400 - BadRequest. Server could not understand request. 404 - Not Found. <datastore_version_id> not found.

Public API Security

There are no envisioned security implications.

Python API

A new method will be implemented in the trove API. This method will upgrade a instance to the image specified by the provided datastore_version.

upgrade(instance, datastore_version)

instance: the instance to upgrade
datastore_version: the datastore version, or its id, to which the trove instance will be upgraded

CLI (python-troveclient)

A new CLI call will be implemented. This new call will upgrade a instance to the image specified by the provided datastore_version.

trove upgrade <instance> <datastore_version>

instance: the instance to upgrade
datastore_version: the datastore version to which the instance will be upgraded

Internal API

A new method will be added in support of this functionality.

def upgrade(self, instance_id, datastore_version_id):
    LOG.debug("Making async call to upgrade guest to %s "
              % datastore_version_id)

    cctxt = self.client.prepare(version=self.version_cap)
    cctxt.cast(self.context, "upgrade", instance_id=instance_id,
               datastore_version_id=datastore_version_id)

Guest Agent

Two new operations will be implemented in the guest agent API. It is expected that each datastore will (optionally) override these methods to implement any needed functionality before and after the image upgrade proceeds. Mysql, for example, would use these methods to copy its configuration data from /etc/mysql to the data volume before the image upgrade, copying them back and restarting the mysql server after the image upgrade.

It is expected that the pre_upgrade method will validate that it is possible to perform the requested upgrade; for example, there may be a configuration override specified for an instance which is not compatible with the new datastore_version. In the event that an upgrade cannot be performed, the pre_upgrade method will raise an exception - any exception will cause the taskmanager to abort the upgrade process for that instance.

def pre_upgrade(self):
    """Prepare the guest for upgrade."""
    LOG.debug("Sending the call to prepare the guest for upgrade.")
    return self._call("pre_upgrade", AGENT_HIGH_TIMEOUT, self.version_cap)

def post_upgrade(self, upgrade_info):
    """Recover the guest after upgrading the guest's image."""
    LOG.debug("Recover the guest after upgrading the guest's image.")
    self._call("post_upgrade", AGENT_HIGH_TIMEOUT, self.version_cap)

Two mechanisms are available for the pre_upgrade to communicate information to the post_upgrade. First, the value returned from the pre_upgrade step is passed to the post_upgrade step as the upgrade_info parameter; the guest is free to use this value as it sees fit, though it would typically be used to exchange a dictionary of configuration data. The second mechanism the guest will use to pass information beween phases is by storing configuration data on the data volume; this would typically be used for large configuration files.

It is expected that if the Nova rebuild command does not return an exception before beginning the rebuild process, the rebuild will succeed. Should the rebuild fail, the instance will be moved to an ERROR state and no attempt will be made to recover the instance; operator intervention will be required. Should the post_upgrade return an exception, the instance will be transitioned to an ERROR state.

Alternatives

The alternative to upgrading images is to upgrade via apt-get or yum in the running instance. While this is the normal procedure for upgrading instances, it has undesired implications for Trove. Trove aims to provide a known service level, but upgrading a running instance has the potential to leave an instance in an unknown state. There could also be issues around installations which don’t allow trove instances to access the internet directly as trove would have to provide some mechanism for delivering all of the deb/rpm packages required to update an instance.

Dashboard Impact (UX)

TBD

Implementation

Assignee(s)

Primary assignee:: 6-morgan

Milestones

Target Milestone for completion:: Newton

Work Items

This feature has been already been prototyped. The work required to bring the prototype in line with this spec is:

The prototype uses image id rather than datastore_version
add trove.upgrade.start/end/error notifications
implement unit tests
investigate if int-tests need to be updated for this feature
document the upgrade procedure

Upgrade Implications

Dependencies

n/a

Testing

Unit tests will be added as appropriate.

An int-test will be added that tests upgrading an instance to the image that it is already running.

Documentation Impact

The “trove upgrade” command, and its corresponding python API, will need to be documented.

References

n/a

Appendix

n/a

Persist Error Message

Tue, 29 Mar 2016 00:00:00

Errors that occur in Trove should be easy to retrieve so that the end user can see exactly what is happening with their database instance.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/persist-error-message

Problem Description

Historically it has been very difficult to determine the cause of a failure in Trove. This is due to the fact that errors may be logged in multiple places, none of which are available to the end user. With the advent of Notifications in Trove, however, it is now feasible to persist error messages in the db so that they can be retrieved and displayed.

Proposed Change

Each server will register a callback with the notification framework. Whenever a notification is sent, this callback will be fired off and any errors that occur can then be saved in the database. This information can then be recalled by the user using the ‘trove show’ command.

For errors that occur outside the framework of notifications, a direct call will be made to persist the error. Not all errors will need to be persisted, so an initial set will be proposed that can be enhanced over time as the need arises.

Configuration

No configuration changes are anticipated.

Database

A new table (instance_faults) will be added to the Trove schema:

Column

Type

Allow Nulls

Description

id

varchar(64)

No

ID of fault (autogenerated)

instance_id

varchar(64)

No

ID of instance that the fault occurred on

message

varchar(255)

No

Error message of the fault

details

text(65535)

No

Extra details (i.e. stack trace)

created

DateTime

No

Created date

updated

DateTime

No

Updated date

deleted

tinyint(1)

Yes

Deleted flag

deleted_at

DateTime

Yes

Deleted date

Column	Type	Allow Nulls	Description
id	varchar(64)	No	ID of fault (autogenerated)
instance_id	varchar(64)	No	ID of instance that the fault occurred on
message	varchar(255)	No	Error message of the fault
details	text(65535)	No	Extra details (i.e. stack trace)
created	DateTime	No	Created date
updated	DateTime	No	Updated date
deleted	tinyint(1)	Yes	Deleted flag
deleted_at	DateTime	Yes	Deleted date

Public API

The only change to the public API will be the addition of a ‘fault’ data structure that is returned when requesting instance details. This will look like:

'fault' :
{
    'created': <date>,
    'message': 'error message',
    'details': 'potential stack trace',
},

The ‘details’ value will only be available if the request is done by an admin user.

Public API Security

No security issues are anticipated. Since the messages persisted are all exception messages that are broadcast as notifications, none should contain sensitive information. If any are found to, they should be treated as bugs and modified accordingly (none have been discovered as of yet).

Python API

No changes are anticipated to the python API.

CLI (python-troveclient)

The ‘show’ Trove CLI command may now have new data displayed:

+-------------------+----------------------------------------------------+
| Property          | Value                                              |
+-------------------+----------------------------------------------------+
| created           | 2016-05-06T21:28:53                                |
| datastore         | mysql                                              |
| datastore_version | 5.6                                                |
| fault_date        | 2016-05-06T21:30:06                                |
| fault_details     | Traceback (most recent call last):                 |
|                   |   File "/<snip>/manager.py", line 265, in prepare  |
|                   |     cluster_config, snapshot, modules)             |
|                   |   File "/<snip>/manager.py", line 355, in _prepare |
|                   |     raise RuntimeError("A  guest error occurred")  |
|                   | RuntimeError: A guest error occurred               |
| fault_message     | A guest error occured                              |
| flavor            | 15                                                 |
| id                | 73cfc462-dd59-4dc1-9d32-95954171775f               |
| ip                | 10.66.25.8                                         |
| name              | myinst2                                            |
| status            | ACTIVE                                             |
| updated           | 2016-05-06T21:28:58                                |
| volume            | 1                                                  |
| volume_used       | 0.1                                                |
+-------------------+----------------------------------------------------+

Internal API

No changes need to be made to this API.

Guest Agent

No changes need to be made to the guest agent.

Alternatives

We could continue to require access to the logs and/or Nova instances to determine what happened when an error occurs.

Dashboard Impact (UX)

The relevant fields need to be exposed during the ‘show’ command.

Implementation

Assignee(s)

Primary assignee:: [peterstac]

Milestones

Newton

Work Items

The work will be undertaken within a single task.

Upgrade Implications

No upgrade issues are expected.

Dependencies

None.

Testing

Scenario tests will be enhanced to verify that errors are persisted in the database and can be retrieved.

Documentation Impact

This is a net-new feature, and as such will require documentation.

References

None

Appendix

None

Replication/Cluster Locality

Tue, 29 Mar 2016 00:00:00

Nova has the ability to control whether new instances are created on the same hypervisor (affinity) or on different hypervisors (anti-affinity). This behaviour is useful when setting up replication networks or clusters, so it is proposed to add support for it into Trove.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/replication-cluster-locality

Problem Description

At times it is desirable to control where replicas in a replication set are created. In the case of read slaves, there may be a requirement to ensure that all replicas are on the same hypervisor, whereas for high availability it may be desired that all replicas are created on different hypervisors.

By the same token, clusters would benefit from having instances on different hypervisors.

Replication/cluster locality seeks to address this issue. (Granted, anti-affinity seems more relevant - and desirable - however the two go hand-in-hand as will be seen in the next section.)

Proposed Change

Nova has the capability of placing new instances on the same hypervisor, or on different ones. Within Nova this is handled through the use of ‘server groups.’ 1 Using this feature of Nova will require sending a server_group ‘hint’ to the nova_client.servers.create call. This functionality will be used ‘under the covers’ similar to how secgroups are managed now.

Trove will perform the following actions to accomplish this:

The create_instance method in the Task Manager will be changed to include a ‘locality’ argument.

def create_instance(self, context, instance_id, name, flavor,
                    image_id, databases, users, datastore_manager,
                    packages, volume_size, backup_id, availability_zone,
                    root_password, nics, overrides, slave_of_id,
                    cluster_config, volume_type, locality):

The create method in cluster/models.py will likewise be modified to include a ‘locality’ argument.

@classmethod
def create(cls, context, name, datastore, datastore_version,
           instances, extended_properties, locality):

The locality argument will be used to create a server group with the corresponding policy (affinity or anti-affinity).
The server group details will be converted to a Nova ‘hint.’
This hint will be passed into the Nova client in the create call in taskmanager/models.py.

Note: If affinity is chosen and the hypervisor does not have enough resources, then some of the instances will fail to create. The same goes for choosing anti-affinity and there are not enough available hypervisors. There may be other cases where Nova is unable to create the instance (affinity chosen, but different AZ’s); in these cases the new instance will likely fail to spawn as well.

The Trove show and cluster-show commands will also be modified to show the ‘locality’ value (i.e. the server_group policy), if it exists.

Configuration

No changes

Database

No changes. The server_group id will be obtained from Nova when it’s needed.

Public API

An attribute ‘locality’ will be added to the data payload sent during a ‘create’ command. The request/response will look like:

Request:

POST v1/{tenant_id}/instances
{
    "instance":
    {
        "volume":
        {
            "type": null,
            "size": 1
        },
        "flavorRef": 7,
        "name": "myinst",
        "replica_count": 2,
        "locality": "affinity"
    }
}

Response:

{
    "instance":
    {
        "status": "BUILD",
        "updated": "2015-12-13T12:36:59",
        "name": "myinst",
        "links":
        [
            {
                "href": "https://10.240.64.151:8779/v1.0/<tenant>/instances/<id>",
                "rel": "self"
            },
            {
                "href": "https://10.240.64.151:8779/instances/<id>",
                "rel": "bookmark"
            }
        ],
        "created": "2015-12-13T12:36:59",
        "id": "<id>",
        "volume":
        {
            "size": 1
        },
        "flavor":
        {
            "id": "7",
            "links":
            [
                {
                    "href": "https://10.240.64.151:8779/v1.0/<tenant>/flavors/7",
                    "rel": "self"
                },
                {
                    "href": "https://10.240.64.151:8779/flavors/7",
                    "rel": "bookmark"
                }
            ]
        },
        "datastore":
        {
            "version": "5.6",
            "type": "mysql"
        },
        "locality": "affinity"
    }
}

The ‘show’ and ‘cluster-show’ commands will also now return an attribute ‘locality’ that will look like the one returned from their respective create commands.

Public API Security

No impact

Python API

A new argument ‘locality’ will be added to the Trove create command (this will be passed through to the Nova client as a hint by the Task Manager, after creating the corresponding server group). The new Python API signature will be:

def create(self, name, flavor_id, volume=None, databases=None, users=None,
           restorePoint=None, availability_zone=None, datastore=None,
           datastore_version=None, nics=None, configuration=None,
           replica_of=None, slave_of=None, replica_count=None,
           locality=None):

The Python API signature for cluster-create will be:

def create(self, name, datastore, datastore_version, instances=None,
           locality=None):

CLI (python-troveclient)

The create command will now accept a –locality flag that can be one of two values: affinity and anti-affinity. The command would look like:

trove create my_instance 7 --size 1 --locality affinity

For clusters it will be:

trove cluster-create my_cluster mysql 5.6 --locality affinity \
                                          --instance flavor=10,volume=1 \
                                          --instance flavor=10,volume=1 \
                                          --instance flavor=10,volume=1 \

Replicas can then be created in the usual fashion, with all following the locality setting of the master node. If adding replicas to an existing set, an exception will be thrown if –locality is specified, as this flag cannot be changed once it has been associated with an instance (this is a Nova restriction in that servers can’t be added or removed from a group manually). For example, the following command would fail:

trove create my_replica 7 --size 1 --locality affinity --replica_of <id>

When growing a cluster, the same locality will be applied to any new instances created. Because of this, the locality flag will not be available for cluster-grow.

The show and cluster-show commands will also display the locality value. For example:

> trove show my_instance

+-------------------+-------------------------+
| Property          | Value                   |
+-------------------+-------------------------+
| created           | 2015-12-13T12:36:59     |
| datastore         | mysql                   |
| datastore_version | 5.6                     |
| flavor            | 7                       |
| id                | <id>                    |
| ip                | 10.64.151.6             |
| name              | my_instance             |
| status            | ACTIVE                  |
| updated           | 2015-12-13T12:37:03     |
| volume            | 1                       |
| volume_used       | 0.1                     |
| locality          | affinity                |
+-------------------+-------------------------+

If locality is not specified, then no hint will be sent to Nova and it will follow its default algorithm for deciding what hypervisor to use.

For operator assistance in debugging issues the server group id could be displayed through a management call, but will be kept hidden from end users (just as the Nova instance id is not displayed).

Internal API

The Task Manager is responsible for creating the replication and cluster instances, so it will need to be aware of the locality flag. The relevant methods will be changed to include this, as described above.

Once the flag is converted to a server group, a ‘hint’ will be created to pass to the Nova client. The converted hint data will be equivalent to the corresponding ReST API values:

"os:scheduler_hints":
{
    "group": "<id>"
}

Guest Agent

Since the server group must be created before the Nova instance is created, there are no anticipated Guest Agent changes.

Alternatives

The server_group id could be stored in the Trove database. The trade-off to any speed improvements (from not having to request the information from Nova) would be that Trove would have to manage this field properly. If it got out of sync then Trove wouldn’t function as expected.

Dashboard Impact (UX)

For each of the instance create and cluster create screens, a drop-down for locality will need to be added. This drop-down will have ‘None’, ‘affinity’, and ‘anti-affinity’ in it. The value chosen will need to be passed down into the respective create calls. The show commands will need to be enhanced as well to recognize the new ‘locality’ element.

Implementation

Assignee(s)

Primary assignee:: peterstac

Milestones

Target Milestone for completion:: Newton-1

Work Items

The work will be undertaken with the following tasks:

Client (Python and CLI) changes for replication

Server (API and TaskManager) changes for replication

Client (Python and CLI) changes for clustering

Server (API and TaskManager) changes for clustering

Upgrade Implications

Since this change is completely backwards compatible, no upgrade issues are expected.

Dependencies

None

Testing

The replication scenario test will be modified to use –locality=affinity and the results verified (i.e. that the Trove ‘show’ command returns the right value for the ‘locality’ attribute). A negative test with anti-affinity will also be created (since devstack runs on one hypervisor, this test should always fail to create replicas). As far as it is feasible with respect to resources consumed, this same pattern will be used in the cluster scenario tests.

Documentation Impact

This is a net-new feature, and as such will require documentation.

References

1: Output from running ‘nova help server-group-create’

Appendix

None

Trove Image Builder

Tue, 29 Mar 2016 00:00:00

Contents

The development and user community needs a consistent and automated means of generating Trove guest database images (simply “images”) for different platforms and databases using an open source tool.

Launchpad Blueprint: https://blueprints.launchpad.net/trove-image-builder/+spec/trove-image-build-repo

Problem Description

Trove users (new and experienced) typically run into problems in the last leg of a datastore setup, that is the generation and loading of a compatible image including the database and the Trove guestagent. There are significant variations among the GNU/Linux distros, their native database packages, vendor packages and assumptions built into the current Trove codebase. These differences can together conspire to provide unusable Trove guest database instances. An instance launched from an incorrectly assembled Trove image will not properly prepare and register with the Trove control plane, potentially remaining in BUILD state indefinitely, and the novice user will have little information to go on.

Proposed Change

This specification proposes to create a new Trove repository that will contain source code artifacts for an image generation tool. The new tool will support image generation for the same set of distros currently available from “redstack kick-start”. However, a key difference here is that the new repo will have a clear separation of concern from the trove-integration project: no devstack nor test wrappers, just image building.

The artifacts will essentially be a BASH script main driver which potentially loads distro-specific argument parsing, settings, and functions. The driver script will ultimately invoke DIB (diskimage-builder) 0 with arguments to apply DIB file elements to:

install database packages
install Trove guestagent code from one of several sources: Github tip of master or stable release, distribution packages if applicable, community packages such as RDO, or local file system.
apply any required database configuration path changes
systemd or init.d service enablement
set SELinux labels or AppArmor profiles
perform RHEL subscription registration and attachment
optionally install SSH keys for debugging development and test images

Another parameterized option (e.g., command-line argument) should be whether the script gets a base cloud image from the Internet, uses a cached version (like redstack), or is simply pointed at an existing image on the local filesystem.

Other artifacts in the repository could include:

external repository files for loading packages
systemd scripts or other init files
cloud-init datasource specification

This change proposes to start with existing DIB elements derived from the trove-integration project. These currently comprise Ubuntu and Fedora (F22 or higher). Work is underway to convert the existing Fedora elements to CentOS 7. Proposed supported datastores (specific versions omitted since these will change over time):

datastore	Ubuntu	CentOS 7
Cassandra	Yes	Yes
Couchbase	Yes	Yes
CouchDB	Yes	TBD
DB2	Yes	TBD
MariaDB	Yes	Yes
MongoDB	Yes	Yes
MySQL	Yes	Yes
Percona	Yes	Yes
PostgreSQL	Yes	Yes
PXC	Yes	Yes
Redis	Yes	TBD
Vertica	Yes	TBD

NB: Datastores marked TBD for CentOS 7 currently lack public community yum repos for install.

The image build process should be independent of any specific OpenStack python or other dependencies save for the DIB tool. Ideally, this new repo would not be branched with OpenStack releases, just the same as trove-integration is not branched today. However, trove-integration does rely on per-release requirements files for pip. These are essentially trimmed down requirements to service a pip install of the guestagent code. Some possible solutions:

fetch the existing trove-integration requirement files when doing a pip install in the new builder
going forward, have trove incorporate those files formally instead of maintaining them externally (under a separate blueprint)

Although this project could eventually be integrated with the redstack script for use in development and testing, it should have no dependency on it nor the trove or python-troveclient projects for execution, notwithstanding inclusion of the existing guest requirement files.

Although this proposal will be initially organized around the incumbent DIB tool, it will not preclude additions and enhancements to support other image build technologies as recognized by the OpenStack community. 1 For example, a libguestfs 2 approach could be integrated into this same repository if the community reviews and deems that technology to be advantageous at some point in the future.

Configuration

None, this tool doesn’t make use of the configuration options used by the Trove controller. In other words, it will not parse or otherwise attempt to make use of Trove controller configuration settings for its image generation execution.

Database

None, images created from this tool will be loaded into Glance and registered as a Trove datastore the same as today.

Public API

None, image generation is not part of the Trove API.

Public API Security

None, see above

Python API

None, image generation is not part of the Python API.

CLI (python-troveclient)

None

Internal API

None

Guest Agent

None

Alternatives

Use other image image build and manipulation tools as recognized by OpenStack governance 1, such as libguestfs. 2 However, there are concerns about the ability to successfully adopt these tools in areas such as gate checks where DIB is currently used.
Rely on the limited image options provided by the trove-integration project via the redstack tool. However, these images are specifically generated for development testing and not appropriate for other environments.
Manually follow the DIB-based procedure described in the existing “Building Guest Images” document 3.
Use downstream providers which likely incurs a business relationship for access and use of their proprietary tested images.

Dashboard Impact (UX)

None

Implementation

Assignee(s)

Name	Launchpad Id	IRC	Email
Victoria Martinez de la Cruz	vkmc	vkmc	vimartin@redhat.com
Peter MacKinnon	pmackinn	pmackinn	pmackinn@redhat.com

Milestones

Newton-1

Work Items

Have PTL create new project repository for image creation tool and supporting data files.
Develop and install artifacts into repository.
Update existing docs to guide users to new image creation tool.

Upgrade Implications

None

Dependencies

DIB (diskimage-builder) 0

Testing

Autonomous testing of the image builder likely involves inventory of installed packages, file paths, etc. This could be done using other image tools such as guestfish, virt-ls, and virt-cat. 2 For integration testing, we could investigate configurable changes in redstack where instead of generating an image, redstack invokes the new tool. A new CI job (likely initially a non-voting job) could run devstack with trove, run the image generation, configure the generated image and the related datastore, and run integration tests and/or tempest. When the job is deemed stable by the community, it could be considered for voting.

Documentation Impact

References to the new project should be added to the existing “Building Guest Images” document 3.

References

0(1,2): https://github.com/openstack/diskimage-builder
1(1,2): http://docs.openstack.org/image-guide/modify-images.html
2(1,2,3): http://libguestfs.org/
3(1,2): https://github.com/openstack/trove/blob/master/doc/source/dev/building_guest_images.rst

Appendix

None

Multi-Region Trove

Thu, 17 Mar 2016 00:00:00

Trove is currently able to deploy instances to multiple availability zones within a region, but is limited to deployments within a single Openstack region. This specification outlines a proposal for allowing Trove to deploy instances to multiple Openstack regions.

There are three different approaches to implementing multi-region support in Trove. The first approach would be for each region to have it’s own Trove controller and to implement a consistent view of Trove instances across them; this would allow a user in any region to see all trove instances within their region, regardless of which Trove contoller created them. The second approach would be to have a single trove controller across all regions coordinated by a shared database (such as Galera); this would allow users in any region to see all trove instances in all regions. The third approach would be to have the Trove controller in each region to be independent, but able to create instances in other regions; this would allow a user to see all Trove instances created by the Trove controller in their region, regardless of which region the instance is in, but they would not be able to see Trove instances in their own region created by Trove contollers in other regions.

This specification outlines a proposal that would allow the second and third alternatives to be implemented. It is the author’s belief that the first alternative would be far more complex, difficult to implement, and error prone than the second and third options.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/multi-region

Problem Description

Trove will be modified to be able to use Openstack’s cross-region client access to implement support for creating Trove instances in other regions. Essentially, one Trove controller will be able to access Nova and Cinder services in multiple Openstack regions.

Multi-region support will require that the Keystone service be federated between the regions, effectively allowing the Trove controller to access services in multiple regions using a single, common authentication. This functionality is currently supported in Openstack.

Object storage (such as Swift) may be federated between the regions, or each region may have an independent object storage service. Where regions represent physically distinct data centres, a single shared implementation of object storage may be preferrable as it would allow data to be shared efficiently between the regions, rather than being transferred in it’s entirety upon each access.

In the default configuration, each region will host an independent set of Trove contoller services, with each region having it’s own Trove API, Taskmanager, and Conductor services backed by a separate Trove database for each region. When a Taskmanager in one region needs to allocate resources (compute and block storage) in a different region, it will use the OS_REGION_NAME parameter to the Nova and Cinder clients to access the appropriate Openstack services in the other region. The Trove instances allocated in the second region will only be visible to trove cli commands executed in the first region.

A second configuration will be possible where each region will host a set of Trove services which share a common Trove database implemented via a federated database product such as Galera clustering. This will allow users in each region to see all Trove instances, regardless of in which region they are hosted, but may expose the Trove services to the usual issues associated with running Openstack services on a database which uses optimistic locking. No testing of this alternative is envisioned for the initial implementation of multi-region support.

Implementing multi-region support as described in this document will require that the instances in each region be on a network shared across all regions, and that the instances be able to access the Rabbit network in each region.

Proposed Change

Supporting multiple regions, as laid out in this document, will primary consist of allowing the Trove services (API, Taskmanager, and Guestagent) to provide an OS_REGION_NAME parameter when accessing the clients for Nova and Cinder. Adding this support will encompass the following components:

Add ‘region’ field to the Instances table in the Trove database.
Enhance the CLI and REST API to allow ‘region name’ to be specified for each instance to be created (both single instance and clusters)
Add ‘region name’ to guest agent RPC calls as needed (backup_info parameter of prepare call)
Enhance each datastore to use the ‘region name’ as appropriate

When specifying that an instance be started in a different region, Trove will need to ensure that an appropriate image is available in the target region. To do so, trove will contact the Glance service in the other region to retrieve the metadata for the image of the same name as specified in the datastore (in the first region). The checksums of the images in both regions will be compared to ensure that the same image in installed in each region.

Configuration

No changes to configuration files are envisioned.

Database

A ‘region_name’ field will be added to the Trove ‘instances’ table.

Migration scripts will be provided to add ‘region_name’ parameters to the above listed tables during Openstack release upgrades.

A region_name property will be added to the DBInstance class and shadowed in the SimpleInstance class.

Public API

A “region” parameter will be added to the following REST APIs to indicate the region in which the specified resource should be created. If the “region” parameter is not specified, the resource will be created in the region in which the command is executed.

Operations which do not create new resources, such as the list and show APIs, do not require additional region parameters. For those APIs, the region would be specified via the –os-region-name parameter.

When the Taskmanager is requested to create an instance in a region different than its own, it will need to ensure that a suitable image is available in that region to create the instance (as it will not be possible to tell the Nova in RegionB to use an image from RegionA). To create an instance in RegionB, the Taskmanager in RegionA will proceed as follows:

Retrieve the name of the appropriate image from the appropriate datastore_version in RegionA
Retrieve the checksum from Glance for the image in RegionA
Ensure that trove in RegionB has a datastore_version of the same name, and that datastore_version specifies an image of the same name
Retrieve the checksum of the image from the Glance in RegionB
Ensure that the image in both regions have identical checksums
Follow a procedure similar to that above to ensure that a similarly named flavour exists in both regions and has similar properties
Ask Nova in RegionB to create an instance with the appropriate image name and flavour

Instance Create

Request:

POST v1/<tenant_id>/instances
{
    "instance": {
        "volume": {
            "type": null,
            "size": 1
        },
        "flavorRef": 11,
        "name": "m",
        "replica_count": 1,
        "replica_of": "0d5e5bcc-5c60-4703-b4b3-17f32e0abe72",
        "region": "RegionA"
    }
}

Response:

{
    "instance": {
        "created": "2016-03-08T16:13:30",
        "datastore": {
            "type": "mysql",
            "version": "5.6"
        },
        "flavor": {
            "id": "11",
            "links": [
                {
                    "href": "https://<ip>:8779/v1.0/adbe7218e9f54369a0898f36d9c7a66d/flavors/11",
                    "rel": "self"
                },
                {
                    "href": "https://<ip>:8779/flavors/11",
                    "rel": "bookmark"
                }
            ]
        },
        "id": "0d5e5bcc-5c60-4703-b4b3-17f32e0abe64",
        "links": [
            {
                "href": "https://<ip>:8779/v1.0/adbe7218e9f54369a0898f36d9c7a66d/instances/0d5e5bcc-5c60-4703-b4b3-17f32e0abe64",
                "rel": "self"
            },
            {
                "href": "https://<ip>:8779/instances/0d5e5bcc-5c60-4703-b4b3-17f32e0abe64",
                "rel": "bookmark"
            }
        ],
        "name": "m",
        "status": "BUILD",
        "updated": "2016-03-08T16:13:30",
        "volume": {
            "size": 1
        },
        "region": "RegionA"
    }
}

Cluster Create

Request:

POST /v1.0/<tenant_id>/clusters
{
  "cluster": {
    "name": "products",
    "datastore": {
      "type": "percona",
      "version": "5.5"
    },
    "instances": [
      {
        "flavorRef": "2",
        "volume": {
          "size": 100
        },
        "region": "RegionA",
      },
      {
        "flavorRef": "2",
        "volume": {
          "size": 100
        },
        "region": "RegionA",
      },
      {
        "flavorRef": "2",
        "volume": {
          "size": 100
        },
        "region": "RegionB",
      }
    ],
  }
}

Response:

{
  "cluster": {
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "task": {
      "id": 2,
      "name": "BUILDING",
      "description": "Building the initial cluster."
    },
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "percona",
      "version": "5.5"
    },
    "region": "RegionA",
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "status": "BUILD",
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
        "region": "RegionA",
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
        "status": "BUILD",
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
        "region": "RegionA",
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
        "status": "BUILD",
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
        "region": "RegionB",
      }
    ],
  }
}

cluster-grow

Request:

POST /v1.0/<tenant_id>/clusters/<cluster-id>/action
{
    "grow": [
      {
        "name": "redis-clstr-member-5",
        "instance_type": "master",
        "flavorRef": "2",
        "volume": {
          "size": 2
        },
      },
      {
        "name": "redis-clstr-member-6",
        "instance_type": "slave",
        "related_to": "redis-clstr-member-5",
        "flavorRef": "2",
        "volume": {
          "size": 2
        },
        "region": "RegionB",
      }
    ]
}

Response:

{
  "cluster": {
    "id": "edaac9ca-b5e1-4028-adb7-fa7653e11224",
    "task": {
      "id": 2,
      "name": "BUILDING",
      "description": "Building the initial cluster."
    },
    "name": "redis-clstr",
    "created": "2015-01-29T20:19:23",
    "updated": "2015-01-29T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "redis",
      "version": "3.0"
    },
    "ip": [],
    "region": "RegionA",
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "name": "redis-clstr-member-5",
        "instance_type": "master",
        "status": "BUILD",
        "ip": [],
        "links": [{...}],
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        }
        "region": "RegionA",
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
        "name": "redis-clstr-member-6",
        "instance_type": "slave",
        "related_to": "redis-clstr-member-5",
        "status": "BUILD",
        "ip": [],
        "links": [{...}],
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        }
        "region": "RegionB",
      },
    ]
  }
}

Public API Security

This change should not have security impact.

Python API

“region” parameters will be added to the Instances.create(), Clusters.create(), and Clusters.grow() calls.

CLI (python-troveclient)

A “–region” option will be added to the “trove create” CLI command corresponding to the “region” parameter in the Instances.create() Python API.

A “region” option will be added to the “–instance” option of the “trove cluster-create” and “trove cluster-grow” CLI commands corresponding to the Clusters.create() and Clusters.grow() Python APIs.

Internal API

The “region” parameter will be added to the appropriate Taskmanager calls to support instance creation for both single instance and cluster creation.

The Trove Instance class already has a nova_client property that creates a unique client connection for each guest instance. That call will be enhanced to specify the name of the region in which the instance exists; the create_nova_client() method in remote.py will be enhanced to optionally take a region name parameter.

Guest Agent

The only change to the guest agent should be to support initializing a database with data stored in a different region. This would occur during the prepare process, and is to support creating a replica from a backup of a master in a differnt region.

The guest.prepare() call already takes a structure called backup_info which contains details of the backup to be used to initialize the database. This change will add a member “region” to the backup_info structure which will be the name of the region containing the backup. That region name will be passed to the Swift client to tell Swift in which region the backup was created; note, however, that it is expected that Swift will normally be configured to be shared across regions and so be able to optimize object access from all regions.

When a taskmanager in RegionA creates an instance in RegionB, it will pass a guestagent.conf to the new instance. The new instance in RegionB will use the rabbit configuration parameters in the conf file to determine how to connect to the rabbit broker in RegionA. No changes should be required to the existing guest agent to support this functionality.

Alternatives

As indicated in the introduction, an alternative to the design suggested here would be to have the trove controllers perform their own synchronization giving each controller a view of every Trove instance. This would require that all operations be coordinated with the Trove controllers in every region, either via some form of Two Phase Commit or some Eventual Consistency mechanism. Implementing this would be quite complex and offer little benefit beyond the shared database implementation.

Dashboard Impact (UX)

The user should be able to select the region in which a new instance or cluster is to be created.

Panels which display properties of instances or clusters should be enhanced to display the region name.

Implementation

Assignee(s)

Primary assignee:: 6-morgan

Milestones

Target Milestone for completion:: eg. Liberty-1

Work Items

Already implemented, code awaiting spec approval.

Upgrade Implications

No upgrade implications are envisioned as a result of this change.

Dependencies

No dependecies.

Testing

No int-tests will be developed for this feature due to the difficulty of creating multiple regions within devstack.

Documentation Impact

Documentation will be necessary for the new parameters to the Trove CLI commands.

References

Appendix

None.

Percona XtraDB Cluster Root Enable

Wed, 13 Jan 2016 00:00:00

Contents

For the Percona XtraDB Cluster (PXC) datastore we need the ability to enable root for a cluster to be able to manage the datastore with a privileged user. For example, in order to integrate a PXC datastore cluster with Cloud Foundry it requires a root user to automatically manage the database users and databases on the cluster. This will add root enable to the PXC solution we currently have.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/pxc-root-enable

Problem Description

PXC is lacking support for root enable cluster. This will allow a user to create a root user and either give a new random password or set the given password for the root user across the cluster.

Proposed Change

PXC will use the existing root enable API call. This change will allow the cluster to enable root within a cluster. The call will apply the root user and password to a single node in the cluster and since the cluster replicates to the other nodes they will get the same root user and password set.

Requirements for root enable:

A root enable api to a cluster resource shall apply the root credentials to the cluster.
A user shall be able to connect to any instance in the cluster with the root user and password to manage the datastore.

PXC will need to extend the existing root enable calls into its clustered datastore implementation because currently the root enable calls raise an exception if root enable is called with a cluster since most datastores do not support this feature. This will require a configuration change for the guestagent and will be outlined below.

This change only involves adding the existing feature to the PXC datastore not changing the existing functionality of other datastores. This will not add root enabled at cluster create time because that functionality does not currently exist in the api for cluster create.

Configuration

In order to allow a PXC cluster to enable root we need to update the guestagent api to support this call. After looking over the other cluster enabled datastores the only other cluster datastore that supports this feature is Vertica. Much of this code can easily be reused and applied to PXC as well. We will pull the majority of this code into a common location that can be used by other datastores in the future as well as the PXC datastore today. The Vertica datastore can extend this class with the additional changes it needs specific to its datastore.

The PXC configuration change will be with the root_controller configuration parameter where it uses the DefaultRootController today and instead point to the new class called ClusterRootController.

Database

None

Public API

The following public API calls will be made available to the PXC datastore.

Cluster root enable - The existing call payload will not be changed. Implementing the root enable of a cluster resource will allow a user to enable root across all instances in the cluster, as well as allow the user to specify a password that will be used to enable root on the cluster.

Public API Security

None

Python API

None

CLI (python-troveclient)

Support for the following existing CLI calls.

root-enable with or without the –root_password given by user

No changes should be necessary to accomplish these actions.

Internal API

The Vertica datastore already has this feature and much of the code is reusable so we can move this code to a common location and build off of it.

Move the trove.extensions.vertica.models.VerticaRoot class to trove.extensions.common.models.ClusterRoot
Move the trove.extensions.vertica.service.VericaRootController class to trove.extensions.common.service.ClusterRootController
Make the trove.extensions.vertica.service.VericaRootController class extend the trove.extensions.common.service.ClusterRootController with small change Vertica needs.

Guest Agent

The PXC guest agent will need the methods to enable root as well with or without a password. Since this feature didn’t exist prior to now we only need the enable_root_with_password and not the enable_root method in the manager for PXC. The enable_root_with_password method is the new version of the root enable call. The old version that did not have a password was left for backward compatibility.

Add the enable_root_with_password method to the pxc manager.

No calls will be deprecated in order to complete this.

Alternatives

Not to support root enable of a PXC cluster.

Dashboard Impact (UX)

The dashboard will need an update for this to be enabled for a PXC cluster. This change will need to apply for only a cluster of datastore type pxc as other datastores do not have this ability.

Need a new option on a cluster dropdown to allow a user to enable root for a given cluster. Greyed out if the cluster is not of datastore type pxc.

There should be a dialog that allows the user to enter a password that they would like to use for the root user or allow the system to randomly gererate a password and display the password to them once so that they can copy and paste it to what ever application needs root access.

Implementation

Assignee(s)

Name	Launchpad Id	IRC	Email
Name	Launchpad Id	IRC	Email
Craig Vyvial	cp16net	cp16net	cp16net@gmail.com
Craig Vyvial	cp16net	cp16net	cp16net@gmail.com

Milestones

Targeting this for milestone: Mitaka-3

Work Items

Enable root for a PXC cluster
UX changes needed for root enable of a PXC cluster.

Upgrade Implications

Just the common updates since this is a new feature the guestagent will need to be updated.

Dependencies

None

Testing

There will be unit tests that test the new calls for the strategy.

There will be integration tests added to the scenario tests that will test enabling root on a cluster.

Documentation Impact

We need to update the docs to show support for root enable on PXC clusters.

References

None

Appendix

None

Module Management

Tue, 12 Jan 2016 00:00:00

Historically, Trove has supported open source databases. As more datastores were added it was inevitable that this would eventually change to include proprietary databases as well. Starting with the Liberty release this is the case (support for Vertica and DB2 now being available) and with this comes the issue of managing the licenses of said databases. In addition, operators may find it useful to include other software on their images that may require ‘activation’ by end-users (for example New Relic’s analytical suite). A method of activating this software is also needed.

The concept of applying a ‘license’ or ‘activation’ or ‘configuration’ of this third-party software is what is referred to herein as ‘module’ management.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/module-management

Problem Description

Users of a particular cloud may be willing to purchase a license to use a datastore from the cloud vendor on a pay-as-you-go based model, and this is the model assumed at the moment (as both Vertica and DB2 redstack images include a fully-functional and licensed database). It is also desirable, however, that users be allowed to ‘bring their own license.’ In this scenario the user provides a license file that the database requires, and as such a mechanism needs to be in place to ‘activate’ and/or ‘renew’ the license through the use of the user provided file.

This same problem exists for any other proprietary software that an operator may wish to include in their Trove images. These software packages also typically require activation through the use of a license key or file (such as New Relic 1) or configuration of some kind.

Proposed Change

Trove’s responsiblity towards module management will be restricted to the scope of encrypting and storing the required data file (for example, a license file) and providing a way to apply this data to a new or existing Trove instance or cluster. A mechanism will be put in place to allow end users the ability to manage adding, deleting, listing, viewing and updating these module data files.

Methods to apply, remove, query and retrieve the actual ‘module’ data file on the Trove instance will also be provided.

A repeatable option (–module) will be added to the create and cluster-create commands to allow adhoc module selection. In addition, modules can be set to auto-apply, which will have the effect of the Guest Agent installing that module on any instance created with the relevant datastore combination.

Configuration

The following configuration changes are anticipated.

A way of specifying valid module ‘types’ will be needed for proper validation on module create:

cfg.StrOpt('module_types', default=None,
           help='A list of module types supported.'),

A key will be needed in order to be able to encrypt the module data file before storing it in the database:

cfg.StrOpt('module_aes_cbc_key', default='module_aes_cbc_key',
           help='OpenSSL aes_cbc key for module encryption.'),

Database

A new table (modules) will be added to the Trove schema:

Column

Type

Allow Nulls

Description

id

varchar(36)

No

ID of module (autogenerated)

type

varchar(255)

No

Type of module. This will correlate directly to the required plugin (i.e. a plugin must exist of this ‘type’)

tenant_id

varchar(36)

No

ID of tenant to apply module to. ‘all’ means module applies to all tenants

datastore

varchar(36)

No

Name of datastore to apply module to. ‘all’ means module applies to all datastores

datastore_version

varchar(36)

No

Name of datastore version to apply module to. ‘all’ means module applies to all datastores

name

varchar(255)

No

Name of module

description

varchar(512)

Yes

Description of module

auto_apply

tinyint(1)

No

Should this module be automatically applied during instance/cluster create. Will default to ‘no’ if not provided

visible

tinyint(1)

No

Should this module be visible to non-admin users. Will default to ‘yes’ if not provided

live_update

tinyint(1)

No

Can this module be updated while applied-to instances still exist. If set to ‘no’ all instances must have the corresponding module removed before it can be updated. Defaults to ‘no’

contents

blob

No

Encrypted module contents

md5

varchar(32)

No

MD5 hash of module contents

created

DateTime

No

Created date

updated

DateTime

No

Updated date

deleted

tinyint(1)

Yes

Deleted flag

deleted_at

DateTime

Yes

Deleted date

Column	Type	Allow Nulls	Description
id	varchar(36)	No	ID of module (autogenerated)
type	varchar(255)	No	Type of module. This will correlate directly to the required plugin (i.e. a plugin must exist of this ‘type’)
tenant_id	varchar(36)	No	ID of tenant to apply module to. ‘all’ means module applies to all tenants
datastore	varchar(36)	No	Name of datastore to apply module to. ‘all’ means module applies to all datastores
datastore_version	varchar(36)	No	Name of datastore version to apply module to. ‘all’ means module applies to all datastores
name	varchar(255)	No	Name of module
description	varchar(512)	Yes	Description of module
auto_apply	tinyint(1)	No	Should this module be automatically applied during instance/cluster create. Will default to ‘no’ if not provided
visible	tinyint(1)	No	Should this module be visible to non-admin users. Will default to ‘yes’ if not provided
live_update	tinyint(1)	No	Can this module be updated while applied-to instances still exist. If set to ‘no’ all instances must have the corresponding module removed before it can be updated. Defaults to ‘no’
contents	blob	No	Encrypted module contents
md5	varchar(32)	No	MD5 hash of module contents
created	DateTime	No	Created date
updated	DateTime	No	Updated date
deleted	tinyint(1)	Yes	Deleted flag
deleted_at	DateTime	Yes	Deleted date

A unique index will be created from the (datastore, datastore_version, name) fields, to allow easy determination of the correct module to apply to a specific instance.

An MD5 hash of the module contents will be stored in the module record as well. This hash will be reported back when querying the module from a running instance, as the original module record could have been modified with new contents after it was initially applied.

On installing the module contents on a given instance, a file will be created in a know location using <datastore>-<datastore_version>-<name>.lic as a pattern.

Creating modules that apply to ‘all’ tenants or ‘all’ datastores and ones that are auto-applied will require admin credentials.

Setting a module to ‘not’ visible is also an admin-only option. This will allow administrators to ‘hide’ modules from users if they so desire. Modules that are marked visible=False will not be returned in commands such as list or show unless requested by an admin user. Non-admin users won’t be able to apply a non-visible module, however they will still be auto-applied if so designated.

A new table (instance_modules) will be added to the Trove schema to track which modules have been applied to each instance:

Column

Type

Allow Nulls

Description

id

varchar(36)

No

ID of association (autogenerated)

instance_id

varchar(36)

No

ID of instance

module_id

varchar(36)

No

ID of module

md5

varchar(32)

No

MD5 hash of module contents

created

DateTime

No

Created date

updated

DateTime

No

Updated date

deleted

tinyint(1)

Yes

Deleted flag

deleted_at

DateTime

Yes

Deleted date

Column	Type	Allow Nulls	Description
id	varchar(36)	No	ID of association (autogenerated)
instance_id	varchar(36)	No	ID of instance
module_id	varchar(36)	No	ID of module
md5	varchar(32)	No	MD5 hash of module contents
created	DateTime	No	Created date
updated	DateTime	No	Updated date
deleted	tinyint(1)	Yes	Deleted flag
deleted_at	DateTime	Yes	Deleted date

Public API

New ReST API calls will be added to the Trove infrastructure. These fall into two categories - ones to manage the maintenance of the actual modules, and ones to handle the instance interactions.

In addition, the create and cluster-create calls will be enhanced.

Module Maintenance

To retrieve a list of all modules that can be applied, the following request would be made:

Request:

GET v1/modules

Response:

{
    'modules' : [
        {
            'id': <id>,
            'type': 'vertica_license',
            'tenant': <id>,
            'datastore': 'vertica',
            'datastore_version': 'all',
            'name': '100GB',
            'description': 'Vertica license for 100GB',
            'auto_apply': False,
            'visible': True,  # returned for admin only
            'live_update': False,
            'md5': <md5>,
            'created': <date>,
            'updated': <date>,
        },
        {
            'id': <id>,
            'type': 'new_relic_activation',
            'tenant': <id>,
            'datastore': 'all',
            'datastore_version': 'all',
            'name': 'new_relic',
            'description': 'New Relic activation',
            'auto_apply': True,
            'visible': True,  # returned for admin only
            'live_update': True,
            'md5': <md5>,
            'created': <date>,
            'updated': <date>,
        },
    ]
}

Response Codes:

200  Success

Note that an admin user will receive the modules for all tenants, whereas regular users will see modules for their tenant only.

To retrieve a list of valid modules that can be applied to a specific datastore, the following request would be made:

Request:

GET v1/datastores/{datastore_id}/modules

Response:

{
    'modules' : [
        {
            'id': <id>,
            'type': 'new_relic_activation',
            'tenant': <id>,
            'datastore': 'all',
            'datastore_version': 'all',
            'name': 'new_relic',
            'description': 'New Relic activation',
            'auto_apply': True,
            'visible': True,  # returned for admin only
            'live_update': True,
            'md5': <md5>,
            'updated': <date>,
        },
    ]
}

Response Codes:

200  Success

To show the details of a particular module, the following request would be made:

Request:

GET v1/modules/<id>

Response:

{
    'id': <id>,
    'type': 'new_relic_activation',
    'tenant': <id>,
    'datastore': 'all',
    'datastore_version': 'all',
    'name': 'new_relic',
    'description': 'New Relic activation',
    'auto_apply': True,
    'visible': True,  # returned for admin only
    'live_update': True,
    'md5': <md5>,
    'created': <date>,
    'updated': <date>,
}

Response Codes:

200  Success
404  Not Found

To create a module, the following request would be made:

Request:

POST /v1.0/modules
{
    'type': 'vertica_license',
    'tenant': <id>,
    'datastore': 'vertica',
    'datastore_version': 'all',
    'name': '100GB',
    'description': 'Vertica license for 100GB',
    'auto_apply': False,
    'visible': False,  # admin-only option
    'live_update': True,
    'contents': <module_contents>,
}

Response:

{
    "module": {
        'id': <id>,
        'type': 'vertica_license',
        'tenant': <id>,
        'datastore': 'vertica',
        'datastore_version': 'all',
        'name': '100GB',
        'description': 'Vertica license for 100GB',
        'auto_apply': False,
        'visible': False,  # returned for admin only
        'live_update': True,
        'md5': <md5>,
        'created': <date>,
        'updated': <date>,
    }
}

Response Codes:

200  Success
400  Bad Request

To update a module, the following request would be made:

Request:

PATCH /v1.0/modules/{module_id}
{
    'type': 'new_type',
    'tenant': <id>,
    'datastore': 'new_datastore',
    'datastore_version': 'new_datastore_version',
    'name': 'new_name',
    'description': 'new_description',
    'auto_apply': True,
    'visible': False,  # admin-only option
    'live_update': True,
    'contents': <module_contents>,
}

Response:

{
    "module": {
        'id': <id>,
        'type': 'new_type',
        'tenant': <id>,
        'datastore': 'new_datastore',
        'datastore_version': 'new_datastore_version',
        'name': 'new_name',
        'description': 'new_description',
        'auto_apply': True,
        'visible': False,  # returned for admin only
        'live_update': True,
        'md5': <new_md5>,
        'created': <date>,
        'updated': <date>,
    }
}

Response Codes:

Success
Bad Request
Not Found

To delete a module, the following request would be made:

Request:

DELETE /v1.0/modules/{module_id}
{
}

Response:

This operation has no response body

Response Codes:

200  Success
404  Not Found

To query which instances have a particular module applied, the following request would be made:

Request:

GET v1/modules/{module_id}/instances
{
}

Response:

{
    'instance': <id>,
    'modules' : [
        {
            'name': '100GB',
            'id': <id>,
            'md5': <md5>,
            'installed': <date>,
        },
        {
            'name': 'new_relic',
            'id': <id>,
            'md5': <md5>,
            'installed': <date>,
        },
    ]
}

Response Codes:

200  Success
404  Not Found

Instance Interaction

To apply modules to an instance, the following request would be made:

Request:

POST v1/{tenant_id}/instances/{instance_id}/modules
{
    'modules' : [
        {
            "id": <id>,
        },
    ]
}

Response:

{
    'type': 'vertica_license',
    'datastore': 'vertica',
    'datastore_version': 'all',
    'name': '100GB',
    'md5': <md5>,
}

Response Codes:

Success
Bad Request
Not Found

To query an instance about installed modules, the following request would be made:

Request:

GET v1/{tenant_id}/instances/{instance_id}/modules
{
}

Response:

{
    'modules' : [
        {
            'type': 'vertica_license',
            'datastore': 'vertica',
            'datastore_version': 'all',
            'name': '100GB',
            'filename': 'vertica-all-100GB.lic',
            'md5': <md5>,
            'installed': <date>,
            'status': 'OK',
            'error_message': None,
        },
        {
            'type': 'new_relic_activation',
            'datastore': 'all',
            'datastore_version': 'all',
            'name': 'new_relic',
            'filename': 'all-all-new_relic.lic',
            'md5': <md5>,
            'installed': <date>,
            'status': 'FAILED',
            'error_message': 'New Relic binaries not found',
        },
    ]
}

Response Codes:

200  Success
404  Not Found

To retrieve a module from an instance, the following request would be made:

Request:

GET v1/{tenant_id}/instances/{instance_id}/modules/{module_id}
{
}

Response:

{
    'filename': 'vertica-all-100GB.lic',
    'contents': <module_contents>,
    'md5': <md5>,
}

Response Codes:

200  Success
404  Not Found

To delete a module from an instance, the following request would be made:

Request:

DELETE v1/{tenant_id}/instances/{instance_id}/modules/{module_id}
{
}

Response:

This operation has no response body

Response Codes:

202  Success
404  Not Found

Creation Enhancements

The instance create API will be enhanced to include a module field, containing a list of modules to apply. These will be sent down during the normal ‘prepare’ call and the appropriate plugin called once this instance has been provisioned correctly.

{
    'modules' : [
        {
            "id": <id>,
        },
    ]
}

In a similar manner, the cluster create API will also be enhanced to include module information in the instances field, as is currently done with flavors, AZs, etc.

Public API Security

Since the file will be transmitted clear text across the management network, there is a chance that the module can be intercepted if the network is compromised.

It should be ensured that each plugin created does not ‘execute’ the contents of the supplied module data file, as this would present the opportunity for a security breach. This seems unlikely though (and will not be the case for the proposed implementations) as most module data files will be passed to another process for validation, and it is up to that process to ensure proper security is maintained. Code reviews will be vital to make sure no plugin accidentally executes this data.

Python API

New methods will be added to the Python API to facilitate the licensing. A few existing methods will need to be extended as well.

Module Maintenance

def module_list(self, datastore=None):
    """Get a list of all modules that can be applied. Return only
    those that apply to the datastore if it is passed in.
    """

def module_list_instances(self, module):
    """Get a list of all instances that have a given module applied."""

def module_show(self, module):
    """Show the details of the module."""

def module_create(self, module_type, name, description, contents,
                  datastore, datastore_version='all', auto_apply=False,
                  all_tenants=False, visible=True, live_update=False):
    """Create a new module."""

def module_update(self, module, module_type=None, name=None,
                  description=None, contents=None, datastore=None,
                  datastore_version=None, auto_apply=None,
                  all_tenants=None, visible=None, live_update=None):
    """Update an existing module."""

def module_delete(self, module):
    """Delete a module."""

Instance Interaction

def module_apply(self, instance, modules):
    """Apply modules to an instance."""

def module_query(self, instance):
    """Query an instance about installed modules."""

def module_retrieve(self, instance, module=None, filename=None):
    """Retrieve the module data file from an instance and save it in
    filename.  If module is not supplied, retrieve all the modules.
    If filename is not supplied, use the generated filename found
    on the instance.
    """

def module_remove(self, instance, module):
    """Remove a module from an instance."""

Creation Enhancements

For instance.create, the modules field will be added to the call:

def create(self, name, flavor_id, volume=None, databases=None, users=None,
           restorePoint=None, availability_zone=None, datastore=None,
           datastore_version=None, nics=None, configuration=None,
           replica_of=None, slave_of=None, replica_count=None,
           modules=None):
    """Create (boot) a new instance."""

For cluster.create, the modules field will be added to the [‘cluster’][‘instances’] data structure that is already being passed in.

CLI (python-troveclient)

The following Trove CLI commands (upon completion) will be fully functional

module-list Displays all modules for the tenant.
module-show Shows details for a particular module resource.
module-create Creates a new module resource.
module-update Updates module details for a particular module
resource.
module-delete Delete a module resource.
module-apply Apply the given modules to a Trove instance.
module-query Query the given Trove instance for any installed
modules.
module-retrieve Retrieves the current modules from a Trove instance.
module-remove Remove a module from a Trove instance.
create –module [–module]
Creates a new instance and applies the given modules.
cluster-create –instance=module=<id>[,module=<id>]
Creates a new cluster and applies the given modules to each instance.

Internal API

Changes also need to be made to the internal API to include any module IDs as a part of the message body that is sent to the task manager.

The API server will need to make calls to the Guest Agent for the instance interaction type commands.

Guest Agent

In the Guest Agent, the modules will be managed with a plugin style architecture based on the stevedore.driver.DriverManager paradym. Each plugin will need to implement ‘apply’, ‘query’ and ‘remove’ actions. The ‘query’ action will need to report the status of the module ‘apply’ action. This would report (at a minimum) ‘OK’ or ‘FAILED’ plus any other state that seems reasonable for users of the relevant software. If possible, the ‘error_message’ field should be filled with useful information if an error occurs.

A simple plugin ‘base class’ that defines the contract will be provided. It will also provide functionality such as placing the file contents into a specified location and retrieving the file will be added. This can be used as the basis for all other plugins.

The Guest Agent code will use the module ‘type’ to determine if a plugin exists for the given module. If no plugin can be found, then an error will be written to the log and processing stopped.

To provide a concrete, real-world plugin implementation, a Vertica license module plugin will be created to allow licenses to be applied to a Vertica datastore. A New Relic plugin will also be created to illustrate activation of other third party software on a guest image.

Alternatives

None

Dashboard Impact (UX)

A multi-dropdown will need to be added to the instance create dialog that contains all modules for the selected datastore. These modules, along with any auto-apply ones, will need to be sent along on the create call. The same will be needed for the cluster create dialog.

A module detail panel will need to be created. This panel will have fields representing the attributes of a module (see module-create command).

A ‘modules’ list panel will need to be created. This will have buttons for ‘delete’ and ‘update’ and will have a link to the detail page for each listed module. This will be a high-level panel, similar to ‘Instances.’

The instance list panel will need to have a new action added: ‘apply module.’ This will cause a pop-up where the available modules are displayed. The selected module will then be passed in to the module-apply command.

The instance detail panel will need to run ‘module-query’ and display the results in a new section ‘modules.’ Alternately, a link could be placed here that would open a module list panel with the results of the ‘module-query’ call. Here, buttons for ‘module-remove’ and ‘module-retrieve’ would be needed.

Implementation

Assignee(s)

Primary assignee:: [peterstac]

Milestones

Mitaka

Work Items

The work will be undertaken with the following tasks:

Client (Python and CLI) changes

Server (API) changes

Guest Agent module plugin infrastructure

Vertica/New Relic plugin implementation

Upgrade Implications

Since this change is net-new, no upgrade issues are expected.

Dependencies

None.

Testing

Generic int-tests will be written, however these will not be run under MySQL testing as it requires no module-based handling.

Documentation Impact

This is a net-new feature, and as such will require documentation.

References

1: nrsysmond-config –set license_key=<new_relic_key>.

Appendix

None

Implement cassandra cluster provisioning

Mon, 11 Jan 2016 00:00:00

Implement cassandra cluster provisioning.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/cassandra-cluster

Problem Description

Cassandra is by nature a distributed database consisting of ‘equal’ nodes. The ‘equality’ of nodes means that any node in the cluster can receive and service a request. There are no query routers or configuration servers like in some other databases (MongoDB). There are no restrictions on the hardware configuration of individual nodes (i.e. the nodes may have different flavors).

When a node receives a request (it becomes so called a coordinator for that request) it uses the partition key (part of the primary key) to compute which nodes (replicas) contain the requested data (data is distributed based on the hash of the partition key). It then passes the request to the replicas and collects the results. Its job is then compose the response to the client. Each field (value) stored in the database is accompanied by timestamp (or tombstone marker in case of deleted data). The coordinator uses these timestamps to compile the most current view of the data into a result set. That is why it is absolutely critical for the nodes’ clock to be synchronized.

The Cassandra has been designed such that it can survive failure of any given node(s) and still be available. This was achieved by relaxing the consistency requirements. Cassandra offers so called eventual consistency. Both the fault tolerance and consistency can be tuned. Fault tolerance can be configured on the keyspace level (Replication Factor). Consistency can be controlled globally or on per-request level (Consistency Level).

The replication factor determines how many replicas will hold data from a given keyspace. Cassandra can group replicas into ‘racks’. A ‘rack’ is a set of nodes (replicas) that share a common point of failure. Cassandra always tries to distribute data across multiple racks, so that failure of one does not render the data inaccessible.

Example:: Let’s have a three-node system (nodes #1, #2 and #3) and a keyspace with replication factor (RC) 2. Now let’s suppose the user issues an insert statement which lands on node #3 (coordinator). The coordinator uses the partition key to determine which of the nodes the data belong to, say it would be #1. It therefore proceeds to store the data on node #1 and because RC=2 also on the next node #2. Now suppose both node #1 and #2 share a common failure point and are placed into a single rack. The coordinator therefore stores the second replica on the next available node not in the rack - that is #3 (itself).

Cassandra also has a concept of data centers. Data centers are generally geographically distant groups of nodes with their own set of racks. They very much behave like separate clusters.

Example:: Let’s suppose that the example above took place on datacenter DC1, but the keyspace in question is also configured to replicate to DC2 (the replication factor for DC1 does not have to be the same as for DC1). The coordinator then fires an asynchronous request to one of the nodes in DC2 which then proceeds handling the it in the same way.

Note that it is not uncommon for a keyspace to have replication factor RC=0 for a specific data center - one reason could be legal regulation which require certain data be stored within a particular geographical region.

Each node is configured with a cluster name. All nodes within a single cluster must share the same cluster name. Individual nodes in a cluster exchange so called gossip - basic information on the cluster’s topology. Newly added nodes learn about all the other nodes the same way. They just need to be provided a set of initial gossip seeds (already active nodes from within the cluster). It is recommended that the seed nodes come from multiple racks across all DCs (in case one is down).

All these configurations are stored in the standard ‘cassandra.yaml’ file which is already managed by the guest agent 2

The node membership (rack/dc) is stored in ‘cassandra-rackdc.properties’ file at the same location as the main configuration file.

Proposed Change

This specification proposes the following cluster-related actions:

create
grow
shrink
delete

Only rack support will be implemented as a part of this patch. All nodes will be placed into a single DC (‘dc1’). The racks will be mapped on the Availability Zones (AZ) passed from the client. The nodes will be placed into a single default rack (‘rack1’) if no AZ is specified.

Create Cluster

Provision a requested number of instances.
Wait for the instances to become active.
Select the seed nodes. They should include at least one node from each data center and rack.
Configure each cluster node with the list of seeds. Note that the seed nodes must have the automatic bootstrap disabled during initial startup of a new empty cluster. Once all nodes are configured, start the seed nodes one at a time (automatic bootstrap is disabled at this time) followed by the rest of the nodes (automatic bootstrap may be enabled as the seed nodes are already running).
Create the in-database (‘os_admin’) user via the first node. The remaining nodes will replicate in-database changes automatically. Only update the local authentication file on the other nodes.

Grow Cluster

Provision a requested number of instances.
Wait for the instances to become active.
Recompute the seed nodes based on the updated cluster geometry and configure each cluster node with the updated list of seeds.
Retrieve the superuser credentials from one previously existing node and save it on the newly added nodes.
Start any seeds from the added nodes first one at a time followed by the rest of the nodes.
Run nodetool cleanup on each of the previously existing nodes to remove the keys that no longer belong to those nodes (they now on belong to some of the added nodes).

Put the node into BLOCKED state first and then initiate the cleanup. Restore the node’s state once the cleanup finishes. The taskmanager can poll for the node state change and proceed to the next node when ready.

The operation has to be run sequentially on all previously existing nodes and can take an excessive amount of time. Cleanup can generally be safely postponed for low-usage hours.

Shrink Cluster

Recompute the seed nodes based on the updated cluster geometry if any of the existing seed nodes was removed.
Update the list of seeds on remaining nodes if necessary.
Run nodetool decommission on the removed nodes. Cassandra will stream data from decommissioned nodes to the remaining ones. Shutdown the database once completed.
Wait for the removed nodes to go SHUTDOWN.
Delete decommissioned instances.

Configuration

The following configuration values will be implemented in the Cassandra configuration group:

cfg.BoolOpt('cluster_support', default=True,
            help='Enable clusters to be created and managed.'),
cfg.StrOpt('api_strategy',
           default='trove.common.strategies.cluster.experimental.'
           'cassandra.api.CassandraAPIStrategy',
           help='Class that implements datastore-specific API logic.'),
cfg.StrOpt('taskmanager_strategy',
           default='trove.common.strategies.cluster.experimental'
           '.cassandra.taskmanager.CassandraTaskManagerStrategy',
           help='Class that implements datastore-specific task manager '
                'logic.'),
cfg.StrOpt('guestagent_strategy',
           default='trove.common.strategies.cluster.experimental'
           '.cassandra.guestagent.CassandraGuestAgentStrategy',
           help='Class that implements datastore-specific Guest Agent API '
                'logic.'),

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

The following methods will be implemented in the CassandraGuestAgentAPI:

def get_data_center(self):
    LOG.debug("Retrieving the data center for node: %s" % self.id)
    return self._call("get_data_center", guest_api.AGENT_LOW_TIMEOUT,
                      self.version_cap)

def get_rack(self):
    LOG.debug("Retrieving the rack for node: %s" % self.id)
    return self._call("get_rack", guest_api.AGENT_LOW_TIMEOUT,
                      self.version_cap)

def set_seeds(self, seeds):
    LOG.debug("Configuring the gossip seeds for node: %s" % self.id)
    return self._call("set_seeds", guest_api.AGENT_LOW_TIMEOUT,
                      self.version_cap, seeds=seeds)

def get_seeds(self):
    LOG.debug("Retrieving the gossip seeds for node: %s" % self.id)
    return self._call("get_seeds", guest_api.AGENT_LOW_TIMEOUT,
                      self.version_cap)

def set_auto_bootstrap(self, enabled):
    LOG.debug("Setting the auto-bootstrap to '%s' for node: %s"
              % (enabled, self.id))
    return self._call("set_auto_bootstrap", guest_api.AGENT_LOW_TIMEOUT,
                      self.version_cap, enabled=enabled)

def cluster_complete(self):
    LOG.debug("Sending a setup completion notification for node: %s"
              % self.id)
    return self._call("cluster_complete", guest_api.AGENT_LOW_TIMEOUT,
                      self.version_cap)

def node_cleanup_begin(self):
    LOG.debug("Signaling the node to prepare for cleanup: %s" % self.id)
    return self._call("node_cleanup_begin", guest_api.AGENT_LOW_TIMEOUT,
                      self.version_cap)

def node_cleanup(self):
    LOG.debug("Running cleanup on node: %s" % self.id)
    return self._cast('node_cleanup', self.version_cap)

def node_decommission(self):
    LOG.debug("Decommission node: %s" % self.id)
    return self._cast("node_decommission", self.version_cap)

def cluster_secure(self, password):
    LOG.debug("Securing the cluster via node: %s" % self.id)
    return self._call(
        "cluster_secure", guest_api.AGENT_HIGH_TIMEOUT,
        self.version_cap, password=password)

def get_admin_credentials(self):
    LOG.debug("Retrieving the admin credentials from node: %s" % self.id)
    return self._call("get_admin_credentials", guest_api.AGENT_LOW_TIMEOUT,
                      self.version_cap)

def store_admin_credentials(self, admin_credentials):
    LOG.debug("Storing the admin credentials on node: %s" % self.id)
    return self._call("store_admin_credentials",
                      guest_api.AGENT_LOW_TIMEOUT, self.version_cap,
                      admin_credentials=admin_credentials)

Guest Agent

Functionality for writing the ‘cassandra-rackdc.properties’ file will be implemented in addition to the above methods.

The node membership (rack/dc) will be included in the cluster_info dictionary passed into the prepare method.

Alternatives

None

Dashboard Impact (UX)

Will need to enable Cassandra as a clustering datastore.

Implementation

Assignee(s)

Primary assignee:: Petr Malik <pmalik@tesora.com>

Milestones

Mitaka

Work Items

The work will be delivered as a single commit.

Upgrade Implications

None

Dependencies

This implementation heavily depends on work done as a part of:

blueprint cassandra-database-user-functions 1
blueprint cassandra-configuration-groups 2
blueprint cassandra-backup-restore 3

Testing

Manager unittests will be added where appropriate.
The scenario tests already cover implemented functionality.

Documentation Impact

Datastore documentation for Cassandra will need to be updated to reflect clustering support.

References

1: Cassandra user/database implementation review: https://review.openstack.org/#/c/206739/
2(1,2): Cassandra configuration review: https://review.openstack.org/#/c/206740/
3: Cassandra backup/restore review: https://review.openstack.org/#/c/206751/

Appendix

None

Support for ‘root’ actions to Cassandra

Fri, 08 Jan 2016 00:00:00

Implement root enable (with password)/disable and show calls in the Cassandra guest agent. Also include datastore-agnostic scenario tests for the root actions.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/cassandra-root-enable

Problem Description

Continuing the work on add user/database support for Cassandra guests 1 we should also enable the Trove users to create superuser accounts.

Proposed Change

The Cassandra’s superuser is called ‘cassandra’. It comes with any new Cassandra installation, but it can be deleted.

In Cassandra only SUPERUSERS can create other users and grant permissions to database resources. Trove uses the ‘os_admin’ superuser to perform its administrative tasks. 1 The users it creates are all ‘normal’ (NOSUPERUSER) accounts.

The built-in ‘cassandra’ superuser is proactively removed on prepare as it is not needed.

During normal operation (no ‘root’ enabled), there should be only one superuser in the system (os_admin). The ‘root’ is therefore not enabled.

Configuration

None

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

None

Guest Agent

The ‘enable’ action will be implemented by creating a new superuser (‘cassandra’) and granting it full access to all keyspaces. The username (‘cassandra’) and password (user-supplied or random otherwise) will be returned back to the client.

Now on, there will be more than one superuser accounts in the system and the ‘root’ will hence be reported as ‘enabled’.

The ‘disable’ action will merely reset the user’s password to a new random string without exposing it to the end-user. The account itself will not be removed so the root-show will keep reporting root as ‘enabled’ as it should.

When a backup is restored the presence of superusers other than ‘os_admin’ will be checked and the root-status of the new instance will be reported as ‘enabled’ if there are any.

Note that a superuser cannot remove its super privileges or delete itself. It should therefore not be possible to bypass the root check by creating a new superuser, deleting the old one and restoring the state into a new instance.

Once having the superuser access to the database the end-user could of course alter/drop the ‘os_admin’, but doing that would render the instance unmanageable from Trove.

Alternatives

None

Dashboard Impact (UX)

None

Implementation

Assignee(s)

Primary assignee:: Petr Malik <pmalik@tesora.com>

Milestones

Mitaka

Work Items

Implement root-related calls in the Cassandra manager.
Add datastore-agnostic scenario tests to exercise the new functionality.

Upgrade Implications

None

Dependencies

This implementation largely depends on work done as a part of blueprint cassandra-database-user-functions 1

Testing

Manager unittests will be added where appropriate.
Datastore-agnostic scenario tests to exercise the related functionality will be implemented.

Documentation Impact

Document the newly enabled functionality for the Cassandra datastore.

References

1(1,2,3): Cassandra user/database implementation review: https://review.openstack.org/#/c/206739/

Appendix

None

1 Ceilometer Integration

Wed, 06 Jan 2016 00:00:00

The current implementation of trove supports 4 different operation notifications to Ceilometer: trove.instance.create, trove.instance.delete, trove.instance.modify_volume, and trove.instance.modify_flavor. An additional notification, trove.instance.exists is generated periodically to provide status of active instances.

This spec outlines how Trove can be extended to bring its notification support to be more in line with the rest of Openstack by generating start|end|fail notifications for all operation which result in a change of state of the system.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/ceilometer-integration

Contents

1 Ceilometer Integration

1.1 Problem Description

Trove does not support sufficient notifications for billing or error monitoring.

1.2 Proposed Change

Each Trove API call which results in a change of state will be enhanced to generate notifications in line with other Openstack projects. Taskmanager tasks and asynchronous guest agent calls will be supplemented with additional parameters to support the generation of appropriate notifications. A new RPC method will be added to the Conductor to allow the guest to forward the results of asynchronous calls as the guest agent may not have access to the control plane where the Ceilometer collectors operate.

Each API event handler will generate appropriate start, end, and (if necessary) error notifications. For asynchronous calls to the taskmanager or guest agent, the event handler will emit the start notification and pass along sufficient information in the Trove Context to allow subsequent components to generate the end and error notifications. Notification events generated by the Trove guest will be forwarded to the Conductor to be placed on the Trove control plane RabbitMQ broker.

1.2.1 Trove Status Notifications

These notifications have an event type of “trove.<action>”. They are the notifications currently emitted by trove, enhanced to include the cluster_id where appropriate.

All Trove status notifications will have the following information:

Message

event_type: trove.*

traits: &trove_base_traits

instance_type:

fields: payload.instance_type

user_id:

fields: payload.user_id

resource_id:

fields: payload.instance_id

instance_type_id:

fields: payload.instance_type_id

launched_at:

type: datetime

fields: payload.launched_at

instance_name:

fields: payload.instance_name

state:

fields: payload.state

state_description:

fields: payload.state_description

nova_instance_id:

fields: payload.nova_instance_id

service_id:

fields: payload.service_id

created_at:

type: datetime

fields: payload.created_at

region:

fields: payload.region

instance_type: name of the flavor of the instance
user_id: the id of the Openstack user who invoked the action
resource_id: trove instance_id of the instance
instance_type_id: flavor id of the flavor of the instance
launched_at: same as created_at
instance_name: name of the trove instance
state: current state of the instance
state_description: description of the current state of the instance
nova_instance_id: id of the underlying nova instance
service_id: notification service id for datastore
created_at: creation time of the instance
region: Openstack region

Trove status notifications for trove.create, trove.delete, trove.modify_volume and trove.modify_flavor will have the following additional information:

Message

event_type: [‘trove.instance.create’, ‘trove.instance.modify_volume’,

‘trove.instance.modify_flavor’, ‘trove.instance.delete’]

traits: &trove_common_traits

name:

fields: payload.name

availability_zone:

fields: payload.availability_zone

instance_size:

type: int

fields: payload.instance_size

volume_size:

type: int

fields: payload.volume_size

nova_volume_id:

fields: payload.nova_volume_id

cluster_id:

fields: payload.cluster_id

name: name of the Trove instance
availability_zone: Openstack availability zone
instance_size: ram allocated to instance
volume_size: size of volume (if volume support enabled)
nova_volume_id: nove volume id (if volume support enabled)
cluster_id: id of cluster (if instance in cluster)

1.2.1.1 trove.instance.create

Message

event_type: trove.instance.create

traits:

<<: [*trove_base_traits, *trove_common_traits]

1.2.1.2 trove.instance.delete

Message

event_type: trove.instance.delete

traits:

<<: [*trove_base_traits, *trove_common_traits]

deleted_at:

type: datetime

fields: payload.deleted_at

deleted_at: time at which instance deletion is complete

1.2.1.3 trove.instance.modify_volume

Message

event_type: trove.instance.modify_volume

traits:

<<: [*trove_base_traits, *trove_common_traits]

old_volume_size:

type: int

fields: payload.old_volume_size

modify_at:

type: datetime

fields: payload.modify_at

old_volume_size: volume size prior to resize
modify_at: time at which volume resize completed

1.2.1.4 trove.instance.modify_flavor

Message

event_type: trove.instance.modify_flavor

traits:

<<: [*trove_base_traits, *trove_common_traits]

old_instance_size:

type: int

fields: payload.old_instance_size

modify_at:

type: datetime

fields: payload.modify_at

old_instance_size: memory of instance prior to resize
modify_at: time at which resize completed

1.2.1.5 trove.instance.exists

A periodic event to update statistics about each Trove instance. This notification is emitted only if configured via the exists_notification_transformer config value.

Message

event_type: trove.instance.exists

traits:

<<: *trove_base_traits

display_name:

fields: payload.display_name

audit_period_beginning:

type: datetime

fields: payload.audit_period_beginning

audit_period_ending:

type: datetime

fields: payload.audit_period_ending

display_name: name of the instance
audit_period_beginning: audit period start time
audit_period_ending: audit period end time

1.2.1.6 dbaas.quotas

The dbaas.quotas notification will be periodically emitted to reflect the current state of utilization of the Trove quotas. The frequency of notification events will be defined by the quota_notification_interval configuration value, but at each interval multiple events may be emitted for each resource due to Taskmanager deployment configuration - each Taskmanager will emit a notification for each resource.

Message

event_type: trove.quota

traits:

resource:

fields: payload.resource

in_use:

type: int

fields: payload.in_use

reserved:

type: int

fields: payload.reserved

limit:

type: int

fields: payload.limit

updated:

type: datetime

fields: payload.updated

resource: name of resource (currently “instances”, “volumes”, or “backups”)
in_use: the number of resource instances currently in use
reserved: the number of resource instances currently reserved
limit: the maximum allocation of resource
updated: the time of last resource allocation event

1.2.2 Trove API Notifications

See the Appendix at the end of this document for the individual API notifications.

1.2.3 Configuration

A new configuration value will be added to support implementation of periodic notifcations for quotas.

quota_notification_interval: Seconds between periodic quota notifications (<0 to disable) 1.

1.2.4 Database

No changes will be made to the Trove database.

1.2.5 Public API

No changes will be made to the Trove REST API.

1.2.6 Public API Security

n/a

1.2.7 Python API

No changes will be made to the Trove Python API.

1.2.8 CLI (python-troveclient)

No changes will be made to the Trove CLI. n/a

1.2.9 Internal API

Notification object instances representing the REST API calls that are responsible for generating API Notification events will be created in the Trove API process. These notification objects will be created for each REST API call which expects to change the state of the Trove system, or one of the guest agents. At the time the notification object is created, the appropriate “.start” event will be generated and published to the notification.info queue.

For operations which are processed completely within the trove-api process, the appropriate “.end” or “.error” event will be published upon completion of the operation.

For operations which require support from the Taskmanager or a guest agent, the notification object will be attached to the TroveContext instance for the operation. The trove-taskmanager or trove-guestagent will be able to retrieve the notification instance from the TroveContext instance and use it to generate the appropriate notification events. The trove-taskmanager will place the notification events directly on the RabbitMQ notification.info queue, whereas the guest agent will forward the notification object to the trove-conductor to be placed on the queue.

A new method will be added to the API for the Conductor. The following RPC method instructs the trove-conductor to send an end or error notification with the appropriate values:

def notify_exc_info(self, etype, value, trace):

etype, value, trace: results from the sys.exc_info exception call.

1.2.10 Guest Agent

There will be no changes to the synchronous guest-agent RPC methods.

Asynchronous guest-agent methods will be modified to send “.end” or”.error” notifications to the trove-conductor to reflect the completion status of each asynchronous RPC method.

It is expected that the current upgrade procedure where the guests are upgraded before the trove controller components will be followed. In this scenario, no notifications will be added to the TroveContext, so the guest agent will not send any notifications to the (yet to be upgraded) trove-conductor.

1.2.11 Alternatives

An alternative implementation would be to have the guest agent place notifications directly on the notifications queue, but this would require the guest agent to have access to the same RabbitMQ broker as the trove-api and trove-taskmanager. The proposed design is a step in the direction of allowing the guest to operate in an isolated environment.

1.3 Dashboard Impact (UX)

TBD (section added after approval)

1.4 Implementation

1.4.1 Assignee(s)

Primary assignee:: 6-morgan

1.4.2 Milestones

Target Milestone for completion:: eg. Mitaka

1.4.3 Work Items

Develop framework for new notifications
Re-implement existing notifications in new framework (for consistency)
Implement new notification messages
Implement TroveContext changes
Integrate messages with synchronous API calls in trove-api
Implement conductor RPC methods
Integrate notifications with async Taskmanager RPC methods
Integrate notifications with asynch guest agent RPC methods
Update ceilometer message definitions for new notifications
Unit tests for new code
Modify existing tests to verify notifications
Provide documentation for new notifications

1.5 Upgrade Implications

tenant-id will be added to ceilometer definition for existing Trove notifications
new notifications will be added for all state changing REST APIs

1.6 Dependencies

n/a

1.7 Testing

Existing unit tests will test changes to trove.* notifications.

New unit tests will be developed to test the new notifications.

The notification mechanism, when propely configured, adds notifications to queues to be consumed by other Openstack, or non-Openstack, components. As these consumers are outside the scope of Trove, integration tests do not test notifications.

1.8 Documentation Impact

Documentation will be generated for the new notifications.

1.9 References

1: http://docs.openstack.org/developer/oslo-incubator/api/openstack.common.periodic_task.html

1.10 Appendix

1.10.1 Dbaas API Notification Definitions

These are new notifications emitted by trove to reflect in invocation of the various Trove APIs.

All DBAAS notifications will have an event type of “dbaas.<action>.(start|end|error)” and include the following information:

Message

event_type: dbaas.*

traits: &dbaas_base_traits

tenant_id:

fields: payload.tenant_id

client_ip:

fields: payload.client_ip

server_ip:

fields: payload.server_ip

server_type:

fields: payload.server_type

request_id:

fields: payload.request_id

tenant_id: id of the configured tenant
client_ip: ip:port of trove client which generated API invocation
server_ip: ip:port of trove service which generated notification
server_type: service which generated notification (api|task|conductor)
request_id: Identifier used to correlate start|end|error

1.10.2 dbaas.instance_create.start

Python API: instances.create(name, flavor_id, volume=None, databases=None, users=None, restorePoint=None, availability_zone=None, datastore=None, datastore_version=None, nics=None, configuration=None, replica_of=None, slave_of=None, replica_count=None)
REST API: POST v1/{tenant_id}/instances/create
Event Type: dbaas.instance_create.start
Queue: notification.info
Message

event_type: dbaas.instance_create.start

traits:

<<: *dbaas_base_traits

name:

fields: payload.name

flavor:

fields: payload.flavor

datastore:

fields: payload.datastore

databases:

fields: payload.databases

users:

fields: payload.users

volume_size:

type: int

fields: payload.volume_size

restore_point:

fields: payload.restore_point

availability_zone:

fields: payload.availability_zone

replica_of:

fields: payload.replica_of

replica_count:

fields: payload.replica_count

name: Name of new instance
flavor: Flavor id of instance
datastore: Datastore of instance
datastore_version: Datastore_version of instance
databases: Comma separated list of databases to create (if specified)
users: Comma separated list of users to create (if specified)
volume_size: Size of volume to create (if specified)
restore_point: Reference to backup to use to initialize instance (if specified)
availability_zone: AZ of instance (if specified)
replica_of: Master instance from which to create replica (if specified)
replica_count: Number of replicas to create (if specified)

1.10.3 dbaas.instance_create.end

Python API: instances.create(name, flavor_id, volume=None, databases=None, users=None, restorePoint=None, availability_zone=None, datastore=None, datastore_version=None, nics=None, configuration=None, replica_of=None, slave_of=None, replica_count=None)
REST API: POST v1/{tenant_id}/instances/create
Event Type: dbaas.instance_create.end
Queue: notification.info
Message

event_type: dbaas.instance_create.end

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

instance_id: id of created instance

1.10.4 dbaas.instance_create.error

Python API: instances.create(name, flavor_id, volume=None, databases=None, users=None, restorePoint=None, availability_zone=None, datastore=None, datastore_version=None, nics=None, configuration=None, replica_of=None, slave_of=None, replica_count=None)
REST API: POST v1/{tenant_id}/instances/create
Event Type: dbaas.instance_create.error
Queue: notification.error
Message

event_type: dbaas.instance_create.error

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

message:

fields: payload.message

exception:

fields: payload.exception

instance_id: id of instance
message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.5 dbaas.instance_restart.start

Python API: instances.restart(instance)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_restart.start
Queue: notification.info
Message

event_type: dbaas.instance_restart.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

instance_id: id of instance to restart

1.10.6 dbaas.instance_restart.end

Python API: instances.restart(instance)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_restart.end
Queue: notification.info
Message

event_type: dbaas.instance_restart.end

traits:

<<: *dbaas_base_traits

1.10.7 dbaas.instance_restart.error

Python API: instances.restart(instance)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_restart.error
Queue: notification.error
Message

event_type: dbaas.instance_restart.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.8 dbaas.instance_resize_volume.start

Python API: instances.resize_volume(instance, volume_size)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_resize_volume.start
Queue: notification.info
Message

event_type: dbaas.instance_resize_volume.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

new_size:

type: int

fields: payload.new_size

instance_id: id of instance to resize_volume
new_size: size to which volume will be resized

1.10.9 dbaas.instance_resize_volume.end

Python API: instances.resize_volume(instance, volume_size)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_resize_volume.end
Queue: notification.info
Message

event_type: dbaas.instance_resize_volume.end

traits:

<<: *dbaas_base_traits

1.10.10 dbaas.instance_resize_volume.error

Python API: instances.resize_volume(instance, volume_size)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_resize_volume.error
Queue: notification.error
Message

event_type: dbaas.instance_resize_volume.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.11 dbaas.instance_resize_instance.start

Python API: instances.resize_instance(instance, flavor_id)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_resize_instance.start
Queue: notification.info
Message

event_type: dbaas.instance_resize_instance.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

new_flavor_id:

fields: payload.new_flavor_id

instance_id: id of instance to resize_instance
new_flavor_id: id of flavor to which instance will be resized

1.10.12 dbaas.instance_resize_instance.end

Python API: instances.resize_instance(instance, flavor_id)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_resize_instance.end
Queue: notification.info
Message

event_type: dbaas.instance_resize_instance.end

traits:

<<: *dbaas_base_traits

1.10.13 dbaas.instance_resize_instance.error

Python API: instances.resize_instance(instance, flavor_id)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_resize_instance.error
Queue: notification.error
Message

event_type: dbaas.instance_resize_instance.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.14 dbaas.instance_promote.start

Python API: instances.promote_to_replica_source(instance)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_promote.start
Queue: notification.info
Message

event_type: dbaas.instance_promote.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

instance_id: id of instance to promote

1.10.15 dbaas.instance_promote.end

Python API: instances.promote_to_replica_source(instance)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_promote.end
Queue: notification.info
Message

event_type: dbaas.instance_promote.end

traits:

<<: *dbaas_base_traits

1.10.16 dbaas.instance_promote.error

Python API: instances.promote_to_replica_source(instance)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_promote.error
Queue: notification.error
Message

event_type: dbaas.instance_promote.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.17 dbaas.instance_eject.start

Python API: instances.eject_replica_source(instance)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_eject.start
Queue: notification.info
Message

event_type: dbaas.instance_eject.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

instance_id: id of instance to eject

1.10.18 dbaas.instance_eject.end

Python API: instances.eject_replica_source(instance)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_eject.end
Queue: notification.info
Message

event_type: dbaas.instance_eject.end

traits:

<<: *dbaas_base_traits

1.10.19 dbaas.instance_eject.error

Python API: instances.eject_replica_source(instance)
REST API: POST v1/{tenant_id}/instances/{id}/action
Event Type: dbaas.instance_eject.error
Queue: notification.error
Message

event_type: dbaas.instance_eject.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.20 dbaas.instance_delete.start

Python API: instances.delete(instance)
REST API: DELETE v1/{tenant_id}/instances/{id}
Event Type: dbaas.instance_delete.start
Queue: notification.info
Message

event_type: dbaas.instance_delete.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

instance_id: id of instance to delete

1.10.21 dbaas.instance_delete.end

Python API: instances.delete(instance)
REST API: DELETE v1/{tenant_id}/instances/{id}
Event Type: dbaas.instance_delete.end
Queue: notification.info
Message

event_type: dbaas.instance_delete.end

traits:

<<: *dbaas_base_traits

1.10.22 dbaas.instance_delete.error

Python API: instances.delete(instance)
REST API: DELETE v1/{tenant_id}/instances/{id}
Event Type: dbaas.instance_delete.error
Queue: notification.error
Message

event_type: dbaas.instance_delete.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.23 dbaas.instance_detach.start

Python API: instances.edit(detach_replica_source=True)
REST API: PATCH v1/{tenant_id}/instances/{id}
Event Type: dbaas.instance_detach.start
Queue: notification.info
Message

event_type: dbaas.instance_detach.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

instance_id: id of instance to detach

1.10.24 dbaas.instance_detach.end

Python API: instances.edit(detach_replica_source=True)
REST API: PATCH v1/{tenant_id}/instances/{id}
Event Type: dbaas.instance_detach.end
Queue: notification.info
Message

event_type: dbaas.instance_detach.end

traits:

<<: *dbaas_base_traits

1.10.25 dbaas.instance_detach.error

Python API: instances.edit(detach_replica_source=True)
REST API: PATCH v1/{tenant_id}/instances/{id}
Event Type: dbaas.instance_detach.error
Queue: notification.error
Message

event_type: dbaas.instance_detach.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.26 dbaas.instance_attach_configuration.start

Python API: instances.modify(instance, configuration)
REST API: PATCH v1/{tenant_id}/instances/{id}
Event Type: dbaas.instance_attach_configuration.start
Queue: notification.info
Message

event_type: dbaas.instance_attach_configuration.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

configuration_id:

fields: payload.configuration_id

instance_id: id of instance to attach_configuration
configuration_id: id of configuration to attach

1.10.27 dbaas.instance_attach_configuration.end

Python API: instances.modify(instance, configuration)
REST API: PATCH v1/{tenant_id}/instances/{id}
Event Type: dbaas.instance_attach_configuration.end
Queue: notification.info
Message

event_type: dbaas.instance_attach_configuration.end

traits:

<<: *dbaas_base_traits

1.10.28 dbaas.instance_attach_configuration.error

Python API: instances.modify(instance, configuration)
REST API: PATCH v1/{tenant_id}/instances/{id}
Event Type: dbaas.instance_attach_configuration.error
Queue: notification.error
Message

event_type: dbaas.instance_attach_configuration.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.29 dbaas.instance_detach_configuration.start

Python API: instances.edit(instance, configuration, remove_configuration=True)
REST API: PATCH v1/{tenant_id}/instances/{id}
Event Type: dbaas.instance_detach_configuration.start
Queue: notification.info
Message

event_type: dbaas.instance_detach_configuration.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

instance_id: id of instance to detach_configuration

1.10.30 dbaas.instance_detach_configuration.end

Python API: instances.edit(instance, configuration, remove_configuration=True)
REST API: PATCH v1/{tenant_id}/instances/{id}
Event Type: dbaas.instance_detach_configuration.end
Queue: notification.info
Message

event_type: dbaas.instance_detach_configuration.end

traits:

<<: *dbaas_base_traits

1.10.31 dbaas.instance_detach_configuration.error

Python API: instances.edit(instance, configuration, remove_configuration=True)
REST API: PATCH v1/{tenant_id}/instances/{id}
Event Type: dbaas.instance_detach_configuration.error
Queue: notification.error
Message

event_type: dbaas.instance_detach_configuration.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.32 dbaas.cluster_create.start

Python API: clusters.create(name, datastore, datastore_version, instances=None)
REST API: POST v1/{tenant_id}/clusters
Event Type: dbaas.cluster_create.start
Queue: notification.info
Message

event_type: dbaas.cluster_create.start

traits:

<<: *dbaas_base_traits

name:

fields: payload.name

datastore:

fields: payload.datastore

datastore_version:

fields: payload.datastore_version

name: name of cluster
datastore: datastore of cluster
datastore_version: datastore_version of cluster

1.10.33 dbaas.cluster_create.end

Python API: clusters.create(name, datastore, datastore_version, instances=None)
REST API: POST v1/{tenant_id}/clusters
Event Type: dbaas.cluster_create.end
Queue: notification.info
Message

event_type: dbaas.cluster_create.end

traits:

<<: *dbaas_base_traits

cluster_id:

fields: payload.cluster_id

cluster_id: id of newly created cluster

1.10.34 dbaas.cluster_create.error

Python API: clusters.create(name, datastore, datastore_version, instances=None)
REST API: POST v1/{tenant_id}/clusters
Event Type: dbaas.cluster_create.error
Queue: notification.error
Message

event_type: dbaas.cluster_create.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.35 dbaas.cluster_delete.start

Python API: clusters.delete(cluster)
REST API: DELETE v1/{tenant_id}/clusters/{id}
Event Type: dbaas.cluster_delete.start
Queue: notification.info
Message

event_type: dbaas.cluster_delete.start

traits:

<<: *dbaas_base_traits

cluster_id:

fields: payload.cluster_id

cluster_id: id of cluster to delete

1.10.36 dbaas.cluster_delete.end

Python API: clusters.delete(cluster)
REST API: DELETE v1/{tenant_id}/clusters/{id}
Event Type: dbaas.cluster_delete.end
Queue: notification.info
Message

event_type: dbaas.cluster_delete.end

traits:

<<: *dbaas_base_traits

1.10.37 dbaas.cluster_delete.error

Python API: clusters.delete(cluster)
REST API: DELETE v1/{tenant_id}/clusters/{id}
Event Type: dbaas.cluster_delete.error
Queue: notification.error
Message

event_type: dbaas.cluster_delete.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.38 dbaas.cluster_add_shard.start

Python API: clusters.add_shard(cluster)
REST API: POST v1/{tenant_id}/clusters/{id}
Event Type: dbaas.cluster_add_shard.start
Queue: notification.info
Message

event_type: dbaas.cluster_add_shard.start

traits:

<<: *dbaas_base_traits

cluster_id:

fields: payload.cluster_id

cluster_id: id of cluster to which to add a shard

1.10.39 dbaas.cluster_add_shard.end

Python API: clusters.add_shard(cluster)
REST API: POST v1/{tenant_id}/clusters/{id}
Event Type: dbaas.cluster_add_shard.end
Queue: notification.info
Message

event_type: dbaas.cluster_add_shard.end

traits:

<<: *dbaas_base_traits

1.10.40 dbaas.cluster_add_shard.error

Python API: clusters.add_shard(cluster)
REST API: POST v1/{tenant_id}/clusters/{id}
Event Type: dbaas.cluster_add_shard.error
Queue: notification.error
Message

event_type: dbaas.cluster_add_shard.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.41 dbaas.cluster_grow.start

Python API: clusters.grow(cluster, instances)
REST API: POST v1/{tenant_id}/clusters/{id}
Event Type: dbaas.cluster_grow.start
Queue: notification.info
Message

event_type: dbaas.cluster_grow.start

traits:

<<: *dbaas_base_traits

cluster_id:

fields: payload.cluster_id

cluster_id: id of cluster to which to add a shard

1.10.42 dbaas.cluster_grow.end

Python API: clusters.grow(cluster, instances)
REST API: POST v1/{tenant_id}/clusters/{id}
Event Type: dbaas.cluster_grow.end
Queue: notification.info
Message

event_type: dbaas.cluster_grow.end

traits:

<<: *dbaas_base_traits

1.10.43 dbaas.cluster_grow.error

Python API: clusters.grow(cluster, instances)
REST API: POST v1/{tenant_id}/clusters/{id}
Event Type: dbaas.cluster_grow.error
Queue: notification.error
Message

event_type: dbaas.cluster_grow.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.44 dbaas.cluster_shrink.start

Python API: clusters.shrink(cluster, instances)
REST API: POST v1/{tenant_id}/clusters/{id}
Event Type: dbaas.cluster_shrink.start
Queue: notification.info
Message

event_type: dbaas.cluster_shrink.start

traits:

<<: *dbaas_base_traits

cluster_id:

fields: payload.cluster_id

cluster_id: id of cluster to which to add a shard

1.10.45 dbaas.cluster_shrink.end

Python API: clusters.shrink(cluster, instances)
REST API: POST v1/{tenant_id}/clusters/{id}
Event Type: dbaas.cluster_shrink.end
Queue: notification.info
Message

event_type: dbaas.cluster_shrink.end

traits:

<<: *dbaas_base_traits

1.10.46 dbaas.cluster_shrink.error

Python API: clusters.shrink(cluster, instances)
REST API: POST v1/{tenant_id}/clusters/{id}
Event Type: dbaas.cluster_shrink.error
Queue: notification.error
Message

event_type: dbaas.cluster_shrink.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.47 dbaas.backup_create.start

Python API: backups.create(name, instance=None, description=None, parent_id=None, backup=None)
REST API: POST v1/{tenant_id}/backups
Event Type: dbaas.backup_create.start
Queue: notification.info
Message

event_type: dbaas.backup_create.start

traits:

<<: *dbaas_base_traits

name:

fields: payload.name

instance_id:

fields: payload.instance_id

description:

fields: payload.description

parent_id:

fields: payload.parent_id

name: name of backup
instance_id: id of instance of which to create backup
description: description of backup
parent_id: id of backup of which to take incremental backup

1.10.48 dbaas.backup_create.end

Python API: backups.create(name, instance=None, description=None, parent_id=None, backup=None)
REST API: POST v1/{tenant_id}/backups
Event Type: dbaas.backup_create.end
Queue: notification.info
Message

event_type: dbaas.backup_create.end

traits:

<<: *dbaas_base_traits

backup_id:

fields: payload.backup_id

backup_id: id of newly created backup

1.10.49 dbaas.backup_create.error

Python API: backups.create(name, instance=None, description=None, parent_id=None, backup=None)
REST API: POST v1/{tenant_id}/backups
Event Type: dbaas.backup_create.error
Queue: notification.error
Message

event_type: dbaas.backup_create.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.50 dbaas.backup_delete.start

Python API: backups.delete(backup_id)
REST API: DELETE v1/{tenant_id}/backups/{id}
Event Type: dbaas.backup_delete.start
Queue: notification.info
Message

event_type: dbaas.backup_delete.start

traits:

<<: *dbaas_base_traits

backup_id:

fields: payload.backup_id

backup_id: id of backup to delete

1.10.51 dbaas.backup_delete.end

Python API: backups.delete(backup_id)
REST API: DELETE v1/{tenant_id}/backups/{id}
Event Type: dbaas.backup_delete.end
Queue: notification.info
Message

event_type: dbaas.backup_delete.end

traits:

<<: *dbaas_base_traits

1.10.52 dbaas.backup_delete.error

Python API: backups.delete(backup_id)
REST API: DELETE v1/{tenant_id}/backups/{id}
Event Type: dbaas.backup_delete.error
Queue: notification.error
Message

event_type: dbaas.backup_delete.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.53 dbaas.database_create.start

Note that a single call to databases.create would generate a separate notification for each database to be created.

Python API: databases.create(instance, databases)
REST API: POST v1/{tenant_id}/databases
Event Type: dbaas.database_create.start
Queue: notification.info
Message

event_type: dbaas.database_create.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

dbname:

fields: payload.dbname

instance_id: id of instance of which to create database
dbname: name of database to be created

1.10.54 dbaas.database_create.end

Python API: databases.create(instance, databases)
REST API: POST v1/{tenant_id}/databases
Event Type: dbaas.database_create.end
Queue: notification.info
Message

event_type: dbaas.database_create.end

traits:

<<: *dbaas_base_traits

1.10.55 dbaas.database_create.error

Python API: databases.create(instance, databases)
REST API: POST v1/{tenant_id}/databases
Event Type: dbaas.database_create.error
Queue: notification.error
Message

event_type: dbaas.database_create.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.56 dbaas.database_delete.start

Python API: databases.delete(database_id)
REST API: DELETE v1/{tenant_id}/databases/{id}
Event Type: dbaas.database_delete.start
Queue: notification.info
Message

event_type: dbaas.database_delete.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

dbname:

fields: payload.dbname

instance_id: id of instance from which to delete database
dbname: name of database to delete

1.10.57 dbaas.database_delete.end

Python API: databases.delete(database_id)
REST API: DELETE v1/{tenant_id}/databases/{id}
Event Type: dbaas.database_delete.end
Queue: notification.info
Message

event_type: dbaas.database_delete.end

traits:

<<: *dbaas_base_traits

1.10.58 dbaas.database_delete.error

Python API: databases.delete(database_id)
REST API: DELETE v1/{tenant_id}/databases/{id}
Event Type: dbaas.database_delete.error
Queue: notification.error
Message

event_type: dbaas.database_delete.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.59 dbaas.user_create.start

Note that a single call to users.create would generate a separate notification for each user to be created.

Python API: users.create(instance, users)
REST API: POST v1/{tenant_id}/instances/{instance}/users
Event Type: dbaas.user_create.start
Queue: notification.info
Message

event_type: dbaas.user_create.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

username:

fields: payload.username

instance_id: id of instance for which to create user
username: name of user to be created

1.10.60 dbaas.user_create.end

Python API: users.create(instance, users)
REST API: POST v1/{tenant_id}/instances/{instance}/users
Event Type: dbaas.user_create.end
Queue: notification.info
Message

event_type: dbaas.user_create.end

traits:

<<: *dbaas_base_traits

1.10.61 dbaas.user_create.error

Python API: users.create(instance, users)
REST API: POST v1/{tenant_id}/instances/{instance}/users
Event Type: dbaas.user_create.error
Queue: notification.error
Message

event_type: dbaas.user_create.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.62 dbaas.user_delete.start

Python API: users.delete(instance, username, hostname)
REST API: DELETE v1/{tenant_id}/instances/{instance}/users/{id}
Event Type: dbaas.user_delete.start
Queue: notification.info
Message

event_type: dbaas.user_delete.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

username:

fields: payload.username

instance_id: id of instance from which to delete user
username: name of user to delete

1.10.63 dbaas.user_delete.end

Python API: users.delete(instance, username, hostname)
REST API: DELETE v1/{tenant_id}/instances/{instance}/users/{id}
Event Type: dbaas.user_delete.end
Queue: notification.info
Message

event_type: dbaas.user_delete.end

traits:

<<: *dbaas_base_traits

1.10.64 dbaas.user_delete.error

Python API: users.delete(instance, username, hostname)
REST API: DELETE v1/{tenant_id}/instances/{instance}/users/{id}
Event Type: dbaas.user_delete.error
Queue: notification.error
Message

event_type: dbaas.user_delete.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.65 dbaas.user_update_attributes.start

Python API: users.update_attributes(self, instance, username, newuserattr=None, hostname=None)
REST API: PUT v1/{tenant_id}/instances/{instance}/users/{id}
Event Type: dbaas.user_update_attributes.start
Queue: notification.info
Message

event_type: dbaas.user_update_attributes.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

username:

fields: payload.username

instance_id: id of instance for which to update attributes
username: name of user for whom to update attributes

1.10.66 dbaas.user_update_attributes.end

Python API: users.update_attributes(self, instance, username, newuserattr=None, hostname=None)
REST API: PUT v1/{tenant_id}/instances/{instance}/users/{id}
Event Type: dbaas.user_update_attributes.end
Queue: notification.info
Message

event_type: dbaas.user_update_attributes.end

traits:

<<: *dbaas_base_traits

1.10.67 dbaas.user_update_attributes.error

Python API: users.update_attributes(self, instance, username, newuserattr=None, hostname=None)
REST API: PUT v1/{tenant_id}/instances/{instance}/users/{id}
Event Type: dbaas.user_update_attributes.error
Queue: notification.error
Message

event_type: dbaas.user_update_attributes.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.68 dbaas.user_grant.start

Note that a single call to users.grant would generate a set of notifications for each database to which the user is granted access.

Python API: users.grant(instance, username, databases, hostname=None)
REST API: PUT v1/{tenant_id}/instances/{instance}/users/{id}/databases
Event Type: dbaas.user_grant.start
Queue: notification.info
Message

event_type: dbaas.user_grant.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

username:

fields: payload.username

database:

fields: payload.database

instance_id: id of instance from which to grant user
username: name of user to grant access to database
database: name of database to which access will be granted

1.10.69 dbaas.user_grant.end

Python API: users.grant(instance, username, databases, hostname=None)
REST API: PUT v1/{tenant_id}/instances/{instance}/users/{id}/databases
Event Type: dbaas.user_grant.end
Queue: notification.info
Message

event_type: dbaas.user_grant.end

traits:

<<: *dbaas_base_traits

1.10.70 dbaas.user_grant.error

Python API: users.grant(instance, username, databases, hostname=None)
REST API: PUT v1/{tenant_id}/instances/{instance}/users/{id}/databases
Event Type: dbaas.user_grant.error
Queue: notification.error
Message

event_type: dbaas.user_grant.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.71 dbaas.user_revoke.start

Python API: users.revoke(instance, username, database, hostname=None)
REST API: PUT v1/{tenant_id}/instances/{instance}/users/{id}/databases/{database}
Event Type: dbaas.user_revoke.start
Queue: notification.info
Message

event_type: dbaas.user_revoke.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

username:

fields: payload.username

database:

fields: payload.database

instance_id: id of instance from which to revoke user
username: name of user to revoke access to database
database: name of database to which access will be revoked

1.10.72 dbaas.user_revoke.end

Python API: users.revoke(instance, username, databases, hostname=None)
REST API: PUT v1/{tenant_id}/instances/{instance}/users/{id}/databases/{database}
Event Type: dbaas.user_revoke.end
Queue: notification.info
Message

event_type: dbaas.user_revoke.end

traits:

<<: *dbaas_base_traits

1.10.73 dbaas.user_revoke.error

Python API: users.revoke(instance, username, databases, hostname=None)
REST API: PUT v1/{tenant_id}/instances/{instance}/users/{id}/databases/{database}
Event Type: dbaas.user_revoke.error
Queue: notification.error
Message

event_type: dbaas.user_revoke.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.74 dbaas.user_change_password.start

Note that a single call to users.change_password would result in a separate notification event for each changed user.

Python API: users.change_password(instance, users)
REST API: PUT v1/{tenant_id}/instances/{instance}/users
Event Type: dbaas.user_change_password.start
Queue: notification.info
Message

event_type: dbaas.user_change_password.start

traits:

<<: *dbaas_base_traits

instance_id:

fields: payload.instance_id

username:

fields: payload.username

instance_id: id of instance from which to change_password user
username: name of user for which password changed

1.10.75 dbaas.user_change_password.end

Python API: users.change_password(instance, users)
REST API: PUT v1/{tenant_id}/instances/{instance}/users
Event Type: dbaas.user_change_password.end
Queue: notification.info
Message

event_type: dbaas.user_change_password.end

traits:

<<: *dbaas_base_traits

1.10.76 dbaas.user_change_password.error

Python API: users.change_password(instance, users)
REST API: PUT v1/{tenant_id}/instances/{instance}/users
Event Type: dbaas.user_change_password.error
Queue: notification.error
Message

event_type: dbaas.user_change_password.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.77 dbaas.configuration_create.start

Python API: configurations.create(name, values, description=None, datastore=None, datastore_version=None)
REST API: POST v1/{tenant_id}/configurations
Event Type: dbaas.configuration_create.start
Queue: notification.info
Message

event_type: dbaas.configuration_create.start

traits:

<<: *dbaas_base_traits

name:

fields: payload.name

datastore:

fields: payload.datastore

datastore_version:

fields: payload.datastore_version

name: name of configuration to create
datastore: datastore of configuration
datastore_version: datastore_version of configuration

1.10.78 dbaas.configuration_create.end

Python API: configurations.create(name, values, description=None, datastore=None, datastore_version=None)
REST API: POST v1/{tenant_id}/configurations
Event Type: dbaas.configuration_create.end
Queue: notification.info
Message

event_type: dbaas.configuration_create.end

traits:

<<: *dbaas_base_traits

configuration_id:

fields: payload.configuration_id

configuration_id: id of newly created configuration

1.10.79 dbaas.configuration_create.error

Python API: configurations.create(name, values, description=None, datastore=None, datastore_version=None)
REST API: POST v1/{tenant_id}/configurations
Event Type: dbaas.configuration_create.error
Queue: notification.error
Message

event_type: dbaas.configuration_create.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.80 dbaas.configuration_delete.start

Python API: configurations.delete(configuration_id)
REST API: DELETE v1/{tenant_id}/configurations/{id}
Event Type: dbaas.configuration_delete.start
Queue: notification.info
Message

event_type: dbaas.configuration_delete.start

traits:

<<: *dbaas_base_traits

configuration_id:

fields: payload.configuration_id

configuration_id: id of configuration to delete

1.10.81 dbaas.configuration_delete.end

Python API: configurations.delete(configuration_id)
REST API: DELETE v1/{tenant_id}/configurations/{id}
Event Type: dbaas.configuration_delete.end
Queue: notification.info
Message

event_type: dbaas.configuration_delete.end

traits:

<<: *dbaas_base_traits

1.10.82 dbaas.configuration_delete.error

Python API: configurations.delete(configuration_id)
REST API: DELETE v1/{tenant_id}/configurations/{id}
Event Type: dbaas.configuration_delete.error
Queue: notification.error
Message

event_type: dbaas.configuration_delete.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.83 dbaas.configuration_update.start

Python API: configurations.update(configuration_id, values, name=None, description=None)
REST API: PUT v1/{tenant_id}/configurations/{id}
Event Type: dbaas.configuration_update.start
Queue: notification.info
Message

event_type: dbaas.configuration_update.start

traits:

<<: *dbaas_base_traits

configuration_id:

fields: payload.configuration_id

name:

fields: payload.name

description:

fields: payload.description

configuration_id: id of configuration to update
name: new group name (if specified)
description: new group description (if specified)

1.10.84 dbaas.configuration_update.end

Python API: configurations.update(configuration_id, values, name=None, description=None)
REST API: PUT v1/{tenant_id}/configurations/{id}
Event Type: dbaas.configuration_update.end
Queue: notification.info
Message

event_type: dbaas.configuration_update.end

traits:

<<: *dbaas_base_traits

1.10.85 dbaas.configuration_update.error

Python API: configurations.update(configuration_id, values, name=None, description=None)
REST API: PUT v1/{tenant_id}/configurations/{id}
Event Type: dbaas.configuration_update.error
Queue: notification.error
Message

event_type: dbaas.configuration_update.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

1.10.86 dbaas.configuration_edit.start

Python API: configurations.edit(configuration_id, values)
REST API: PATCH v1/{tenant_id}/configurations/{id}
Event Type: dbaas.configuration_edit.start
Queue: notification.info
Message

event_type: dbaas.configuration_edit.start

traits:

<<: *dbaas_base_traits

configuration_id:

fields: payload.configuration_id

configuration_id: id of configuration to edit

1.10.87 dbaas.configuration_edit.end

Python API: configurations.edit(configuration_id, values)
REST API: PATCH v1/{tenant_id}/configurations/{id}
Event Type: dbaas.configuration_edit.end
Queue: notification.info
Message

event_type: dbaas.configuration_edit.end

traits:

<<: *dbaas_base_traits

1.10.88 dbaas.configuration_edit.error

Python API: configurations.edit(configuration_id, values)
REST API: PATCH v1/{tenant_id}/configurations/{id}
Event Type: dbaas.configuration_edit.error
Queue: notification.error
Message

event_type: dbaas.configuration_edit.error

traits:

<<: *dbaas_base_traits

message:

fields: payload.message

exception:

fields: payload.exception

message: message of exception (duplicated from stack trace)
exception: stack trace of exception

CouchDB Backup & Restore

Wed, 06 Jan 2016 00:00:00

Add support for CouchDB backup and restore functionality in Trove.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/couchdb-backup-restore

Problem Description

CouchDB is one of the NoSQL databases that Trove supports. While Trove provides support for doing backups and restores, this functionality has not been implemented for CouchDB.

Proposed Change

The recommended method for doing full backups in CouchDB is to do a simple filesystem copy of the data files 1. Since CouchDB does not officially support incremental backups, this feature will not be implemented in this release.

CouchDB is a document based database system where each database consist of several documents and each document is a JSON structure. In CouchDB, each document is associated with a revision number. Each database is stored as a single .couch file which is an append-only file (Every time a change is made to the document, it does not overwrite the existing document, instead it creates a new document with an updated version number). This makes it easy to do a full backup by simply copying this .couch file.

The following steps give a high level overview of the full backup and restore process for CouchDB.

Backup process is explained as follows:

Locate the database files to be backed up. By default, the database files are located under /var/lib/couchdb directory (this location will be specified in the couchdb config file under /etc/couchdb directory). In CouchDB, each database is wholly contained in a single append-only file. This means that we can take a backup while its being written to.
Compress and/or encrypt the database files and stream it to Swift storage.

Restore process is explained as follows:

Stream and uncompress backup from storage to the database file directory (this location is specified in the /etc/couchdb config file)
Change ownership of the files to the CouchDB user and group

Configuration

The following configuration options will be updated with the corresponding values for CouchDB:

backup namespace

restore namespace

backup_strategy

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

The following commands will be functional for CouchDB after these changes:

backup-create

backup-delete

backup-list

create –backup

Internal API

None

Guest Agent

The CouchDB guestagent will be modified to add support for full backup and restore functionality. In addition, new strategies will also be added to support these new features.

The following guestagent file will be modified:

trove/guestagent/datastore/experimental/couchdb/manager.py

and the following new strategy files will be added for CouchDB:

trove/guestagent/strategies/backup/experimental/couchdb_impl.py
trove/guestagent/strategies/restore/experimental/couchdb_impl.py

Alternatives

Since CouchDB does not officially support incremental backups, an alternative to doing this is to use the CouchDB replication interface. CouchDB’s replication interface supports master-master replication model and replicates at the database level. It uses the REST API to trigger replication between databases on a source and target systems. If the databases are not on the same system, the source/target should be specified in a URI format:

http://<address>:<port>/<database name>

An example of a request for replicating database ‘db1’ from a local server to a remote server on x.x.x.x server would be:

curl -X POST -d ‘{“source”:”db1”,”target”:”http://x.x.x.x:5984:db1”}’: http://localhost:5984/_replicate

When replication is enabled for a database from a source to target system, it uses the document revision numbers to track differences and detect conflicts and resolve it.

Dashboard Impact (UX)

TBD (section added after approval)

Implementation

Assignee(s)

Mariam John (mariamj@us.ibm.com)

Milestones

Mitaka

Work Items

There will only be one work item for this feature. This includes implementing the strategies for CouchDB backup and restore and make the necessary guestagent changes. This will also include test-cases necessary to test the new functionalities.

Upgrade Implications

None

Dependencies

None

Testing

Add new test cases to test backup and restore functionality for CouchDB:
- Add new unit tests to test the newly implemented functionality
- Add a new CouchDB helper class to the existing integration-test framework.

Documentation Impact

The datastore documentation should be updated to reflect the enabled features.

References

1: Filesystem Backup: http://wiki.apache.org/couchdb/FilesystemBackups?action=show

Appendix

None

Trove Logging

Wed, 06 Jan 2016 00:00:00

Provide end-user access to various type of logs on guest instances.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/datastore-log-operations

Problem Description

In the current implementation, it is not possible for the user to retrieve any logs from the guest agent without ssh access to the instance. The user should be able to retrieve logs as defined by the datastore.

Proposed Change

This document outlines a proposal to provide access to guest logs by storing them in a Swift container.

Each datastore will define a number of log files which it can make available to the user. Logs can be either system logs such as the Trove guest-agent log which are always enabled, or the log can be user logs such as the MySQL Slow Query Log which can be enabled or disabled by the user.

The contents of the appropriate log files will be copied from the guest instance to a Swift container via execution of a log-publish command, and may then be streamed to the user. Subsequent invocations of the log-publish command will only copy log entries made since the last log-publish command to efficiently utilize network resources and minimize impact on database performance.

Swift Log Container

Each log will be composed of multiple objects stored in a Swift container (with a predefined prefix), with each object being a subset of the logging information. The entire log will be reconstructed by concatenating the objects in the container that pertain to the specified log; order can be determined by examining the object metadata for each object, or simply by alphabetically sorting the objects by filename (assuming a suitable naming convention). The prefix will be generated by using a know pattern. All the corresponding files can then be retrieved from Swift using this prefix. The suggested prefix will be: ‘%(instance_id)s/%(datastore)s-%(log)s/’

Each log will store a metadata file in the container, which will have the following information associated with it:

Key	Value
Log Name	Name of log
Log Type	SYS or USER
Log File	File where data is published from
Log Hash	Hash of log file
Log Size	Log file size at last publish
Log Lines	Log file line count at last publish

A list of all the instances with logs in the container could be viewed with the following command:

$ swift list database_logs --delimiter='/'

Response:

fa8c452e-6568-438a-9f04-b2878290fb90/

A list of all the logs for an instance could be viewed with the following command:

$ swift list database_logs --delimiter='/' --prefix fa8c452e-6568-438a-9f04-b2878290fb90/

Response:

fa8c452e-6568-438a-9f04-b2878290fb90/mysql-guest/
fa8c452e-6568-438a-9f04-b2878290fb90/mysql-guest_metafile

A list of all the actual log files associated with a specific datastore log could be viewed with the following command:

$ swift list database_logs --delimiter='/' --prefix fa8c452e-6568-438a-9f04-b2878290fb90/mysql-guest/

Response:

fa8c452e-6568-438a-9f04-b2878290fb90/mysql-guest/log-2015-11-18T21:28:09.975754

Configuration

Datastore CONF settings:

guest_log_exposed_logs - Used to configure which logs will be exposed by the logging component
guest_log_container_name - pattern used to generate the name of the contain which will hold the logs
guest_log_long_query_time - the time to set to identify if a query is taking a long time
guest_log_limit - max size (in bytes) for any chunk pushed to Swift
guest_log_expiry - time in seconds after which the log component is removed from the Swift container (0 for no expiry)

guest_log_exposed_logs

A guest_log_exposed_logs config value will enumerate the logs available for a datastore. By default, the guestagent log would be supported by all datastores plus any other logs defined and supported by the datastore’s guestagent manager.

Excerpt from trove-guestagent.conf:

guest_log_exposed_logs = error,guest,slow_query,general

Any USER log not mentioned in the guest_log_exposed_logs list will be disabled by default. A special value of “ALL” will enable all logs supported by the guest agent manager.

An admin user will always be able to see all the logs, and to publish and view the contents, regardless of whether they are ‘exposed’ or not.

guest_log_container_name

The name of the container used to store log components can be specified in the configuration file.

Excerpt from trove-guestagent.conf:

guest_log_container_name = database_logs

The value shown above is the default value.

guest_log_long_query_time

The amount of time to set for ‘slow_query’ logging. This value is datastore specific, and may mean different things for different datastores. For example, MySQL has a slow query log that these queries are written into, whereas PostgreSQL would use the field to decide what queries to write into its general log.

Log Rotation

Many systems will use log rotation to ensure that logs do not exceed the amount of available disk space on a system. At any point in time, the current log file could be renamed to “<logfile>.1” (or some other name) and a new log file started for ongoing log messages.

To account for this, the logging feature will keep track of a hash of the first line of the current log file that exists during a log-publish operation. The current hash value will be stored in the x-container-meta-log-header-digest value associated with the log file container. Subsequent log-publish operations will use the hash value to determine whether the log has indeed been rotated. If so, the current container will be purged and the new log file published to it.

Database

n/a

Public API

For log-list:

Request:

GET v1/instance/{id}/log

Response:

{
    'logs' : [
        {
            'name': 'guest',
            'type': 'SYS',
            'status': 'Ready',
            'published': '0',
            'pending': '4234',
            'container': 'None'
            'prefix': 'None',
            'metafile': '<id>/mysql-guest_metafile',
        },
        {
            'name': 'general',
            'type': 'USER',
            'status': 'Disabled',
            'published': '0',
            'pending': '0',
            'container': 'None'
            'prefix': 'None',
            'metafile': '<id>/mysql-general_metafile',
        },
        {
            'name': 'slow_query',
            'type': 'USER',
            'status': 'Partial',
            'published': '1009',
            'pending': '304',
            'container': 'database_logs'
            'prefix': '<id>/mysql-slow_query/',
            'metafile': '<id>/mysql-slow_query_metafile',
        },
    ]
}

For log-show:

Request:

POST v1/instance/{id}/log
{ 'name': 'general' }

Response:

{
    'log': {
        'name': 'guest',
        'type': 'SYS',
        'status': 'Partial',
        'published': 218913,
        'pending': 2636234
        'container': 'database_logs',
        'prefix': '<id>/mysql-guest/',
        'metafile': '<id>/mysql-guest_metafile',
    }
}

For log-enable:

Request:

POST v1/instance/{id}/log
{ 'name': 'general', 'enable': 'True' }

Response:

{
    'log': {
        'name': 'general',
        'type': 'USER',
        'status': 'Enabled',
        'published': '0',
        'pending': '0',
        'container': 'None'
        'prefix': 'None',
        'metafile': '<id>/mysql-general_metafile',
        }
    ]
}

For log-disable:

Request:

POST v1/instance/{id}/log
{ 'name': 'general', 'disable': 'True' }

Response:

{
    'log': {
        'name': 'general',
        'type': 'USER',
        'status': 'Disabled',
        'published': '30103',
        'pending': '0',
        'container': 'log-mysql-general-<id>'
        'prefix': '<id>/mysql-general/',
        'metafile': '<id>/mysql-general_metafile',
        }
    ]
}

For log-publish: (Note that ‘publish’ will automatically ‘enable’ a log)

Request:

POST v1/instance/{id}/log
{ 'name': 'general', 'publish': 'True' }

Response:

{
    'log': {
        'name': 'guest',
        'type': 'SYS',
        'status': 'Published',
        'published': '443',
        'pending': '0',
        'container': 'log-mysql-guest-<id>'
        'prefix': '<id>/mysql-guest/',
        'metafile': '<id>/mysql-guest_metafile',
        }
    ]
}

For log-discard

Request:

POST v1/instance/{id}/log
{ 'name': 'general', 'discard': 'True' }

Response:

{
    'log': {
        'name': 'general',
        'type': 'USER',
        'status': 'Ready',
        'published': '0',
        'pending': '30103',
        'container': 'None'
        'prefix': 'None',
        'metafile': '<id>/mysql-general_metafile',
        }
    ]
}

Python API

def log_list(self, instance):
    """Get a list of all guest logs.

    :param instance: The :class:`Instance` (or its ID) of the database
    instance to get the log from.
    :rtype: list of :class:`DataStoreLog`.
    """

def log_show(self, instance, log):
    """Show details of a log.

    :param instance: The :class:`Instance` (or its ID) of the database
    instance to get the log from.
    :param log: The type of <log> to enable
    :rtype: List of :class:`DataStoreLog`.
    """

def log_enable(self, instance, log):
    """Enable the writing of a log.

    :param instance: The :class:`Instance` (or its ID) of the database
    instance to get the log from.
    :param log: The type of <log> to enable
    :rtype: List of :class:`DataStoreLog`.
    """

def log_disable(self, instance, log):
    """Disable the writing of a log.

    :param instance: The :class:`Instance` (or its ID) of the database
    instance to get the log from.
    :param log: The type of <log> to disable
    :rtype: List of :class:`DataStoreLog`.
    """

def log_publish(self, instance, log, disable=None, discard=None):
    """Publish guest log to Swift container.

    :param instance: The :class:`Instance` (or its ID) of the database
    instance to get the log from.
    :param log: The type of <log> to publish
    :param disable: Turn off <log>
    :param discard: Delete the associated container
    :rtype: List of :class:`DataStoreLog`.
    """

def log_discard(self, instance, log):
    """Discard (delete) the published log container in Swift.

    :param instance: The :class:`Instance` (or its ID) of the database
    instance to get the log from.
    :param log: The type of <log> to discard
    :rtype: List of :class:`DataStoreLog`.
    """

def log_generator(self, instance, log, publish=None, lines=50):
    """Return generator to yield the last <lines> lines of guest log.

    :param instance: The :class:`Instance` (or its ID) of the database
    instance to to get the log from.
    :param log: The type of <log> to publish
    :param publish: Publish updates before displaying log
    :param lines: Display last <lines> lines of log (0 for all lines)
    :rtype: generator function to yield log as chunks.
    """

def log_save(self, instance, log, publish=None, filename=None):
    """Saves a guest log to a file.

    :param instance: The :class:`Instance` (or its ID) of the database
    instance to get the log from.
    :param log: The type of <log> to publish
    :param publish: Publish updates before displaying log
    :rtype: Filename to which log was saved
    """

CLI (python-troveclient)

Log List

The log-list command provides information about each log available on a given Trove instance.

$ trove log-list <instance>
+------------+------+-------------+-----------+---------+---------------+---------------------+
| Name       | Type | Status      | Published | Pending | Container     | Prefix              |
+------------+------+-------------+-----------+---------+---------------+---------------------+
| error      | SYS  | Unavailable |         0 |       0 | None          |                     |
| general    | USER | Published   |      1009 |       0 | database_logs | <id>/mysql-general/ |
| guest      | SYS  | Ready       |         0 |  499850 | None          |                     |
| slow_query | USER | Disabled    |         0 |       0 | None          |                     |
+------------+------+-------------+-----------+---------+---------------+---------------------+

Column	Description
Column	Description
Name	Name of the log component
Name	Name of the log component
Type	SYS: System log, always on
	SYS: System log, always on
	USER: Managed by user
	USER: Managed by user
Status	Disabled: Inital state of USER log
	Disabled: Inital state of USER log
	Enabled: Initial state of a SYS log or a USER log with no data in it


	Unavailable: SYS log that has no data in it
	Unavailable: SYS log that has no data in it
	Ready: Log has data available for publishing
	Ready: Log has data available for publishing
	Published: Log file has been fully published
	Published: Log file has been fully published
	Partial: Log file has been partially published
	Partial: Log file has been partially published
	Rotated: Log file has rotated, so next publish will delete the container first


	Restart Required: Datastore requires a restart in order to begin writing to the log file


	Restart Completed: Internal state so the guest log knows to begin reporting the actual state again


Published	Amount of data published to container
Published	Amount of data published to container
Pending	Amount of data available to be published by log-publish
Pending	Amount of data available to be published by log-publish
Container	Swift container that holds the components of the log
Container	Swift container that holds the components of the log
Prefix	Prefix to send to Swift to get just the relevant log parts
Prefix	Prefix to send to Swift to get just the relevant log parts

Note: Where the values for ‘Container’ and ‘Prefix’ for the logs in the example above are ‘None,’ this signifies that that log has not had a log-publish operation executed against it

Log Show

The log-show command provides full information about a specific log available on a given Trove instance.

$ trove log-show <instance> general
 +--------------+-----------------------------+
 | Property     | Value                       |
 +--------------+-----------------------------+
 | name         | slow_query                  |
 | type         | USER                        |
 | status       | Enabled                     |
 | published    | 135                         |
 | pending      | 2156                        |
 | container    | database_logs               |
 | prefix       | <id>/mysql-slow_query/      |
 | metafile     | <id>/mysql-general_metafile |
 +--------------+-----------------------------+

Log Enable

The log-enable command will instruct the guest agent to begin writing information to the specified log file. Only ‘USER’ logs can be enabled as ‘SYS’ logs are enabled by default (and cannot be disabled). Depending on the datastore, this may cause the log to go into a ‘Restart Required’ state where it will remain until the datastore is restarted. This can be configured on a per-datastore basis, and should only be done if there is no way to dynamically start the logging (i.e. PostgreSQL must be restarted in order to change logging, so it would require this configuration).

$ trove log-enable <instance> general
 +--------------+-----------------------------+
 | Property     | Value                       |
 +--------------+-----------------------------+
 | name         | general                     |
 | type         | USER                        |
 | status       | Enabled                     |
 | published    | 0                           |
 | pending      | 0                           |
 | container    | None                        |
 | prefix       | None                        |
 | metafile     | <id>/mysql-general_metafile |
 +--------------+-----------------------------+

Log Disable

The log-disable command will instruct the guest agent to stop writing information to the specified log file. Only ‘USER’ logs can be disabled. As with log-enable, this may cause the log to go into a ‘Restart Required’ state. See log-enable for more details.

$ trove log-disable <instance> general
 +--------------+-----------------------------+
 | Property     | Value                       |
 +--------------+-----------------------------+
 | name         | general                     |
 | type         | USER                        |
 | status       | Disabled                    |
 | published    | 34658                       |
 | pending      | 2532                        |
 | container    | database_logs               |
 | prefix       | <id>/mysql-general/         |
 | metafile     | <id>/mysql-general_metafile |
 +--------------+-----------------------------+

Log Publish

The log-publish command will instruct the guest agent to push any updates to the specified log to the Swift container, which will be created if required. One log-publish command could result in multiple objects being pushed to the Swift container in order to keep each object below the maximum object size as configured by the guest_log_limit CONF value.

The log-publish command will execute asynchronously. When the log-publish command is executed, the Trove instance will be put in the LOGGING state, returning to ACTIVE when objects have been pushed to the logging container so as to successfully finish execution of the command.

When an object is pushed to the Swift container, an X-Delete-After header is used to specify a time-to-live for the container object. This will result in objects automatically being removed from the container after a period of time as specified by the log_expiry CONF value.

An optional –disable parameter will be supported to disable logging for a particular USER log. An optional –discard parameter will be supported to first discard (delete) the associated container.

$ trove log-publish <instance> slow_query
 +--------------+--------------------------------+
 | Property     | Value                          |
 +--------------+--------------------------------+
 | name         | slow_query                     |
 | type         | USER                           |
 | status       | Published                      |
 | published    | 43242                          |
 | pending      | 0                              |
 | container    | database_logs                  |
 | prefix       | <id>/mysql-slow_query/         |
 | metafile     | <id>/mysql-slow_query_metafile |
 +--------------+--------------------------------+

Log Discard

The log-discard command will discard (delete) the container where the current log information resides.

$ trove log-discard <instance> general
 +--------------+-----------------------------+
 | Property     | Value                       |
 +--------------+-----------------------------+
 | name         | general                     |
 | type         | USER                        |
 | status       | Enabled                     |
 | published    | 0                           |
 | pending      | 37190                       |
 | container    | None                        |
 | prefix       | None                        |
 | metafile     | <id>/mysql-general_metafile |
 +--------------+-----------------------------+

Log Tail

By default, log-tail outputs the 50 lines at the end of the log. With the –lines=n option, log-tail will output the last n lines of the log. If n is negative, output will start past line n and continue to the end; –lines=0 will output the entire log.

With the –publish option, log-tail will first execute a log-publish command and wait for the log to be published before beginning output.

It should be noted that the actual display of the log will take place in the Python API only. There will be no REST APIs to facilitate display of the log; such APIs would put undue stress on the system due to the requirements of buffering and streaming from Swift.

$ trove log-tail <instance> slow_query --publish
/usr/local/mysql/libexec/mysqld, Version: 3.23.54-log, started with:
Tcp port: 3306  Unix socket: /tmp/mysql.sock
Time                 Id Command    Argument
# Time: 030207 15:03:33
# User@Host: wsuser[wsuser] @ localhost.localdomain [127.0.0.1]
# Query_time: 13  Lock_time: 0  Rows_sent: 117  Rows_examined: 234
use wsdb;
SELECT l FROM un WHERE ip='209.xx.xxx.xx';

Log Save

Like the log-tail command, the log-save command will execute in the python-troveclient, but where log-tail will output the log to the console, the log-save command will save the log into a file in the filesystem. This will allow the user to download extremely large log files without overwhelming the client or browser.

With the –file option, the log will be saved to the named file. Without the –file option, the log will be output to <logname>.log in the current directory.

With the –publish option, log-tail will first execute a log-publish command and wait for the log to be published before beginning output.

It should be noted that the actual saving of the log will take place in the Python API only. There will be no REST APIs to facilitate display of the log; such APIs would put undue stress on the system due to the requirements of buffering and streaming from Swift.

$ trove log-save <instance> slow_query --file=/tmp/my.log --publish

Public API Security

The Swift containers will be created with the same security credentials as used for backup.

Internal API

Appropriate API methods will be added to both the task manager and the guest.

No changes to existing APIs are foreseen.

Guest Agent

No compatibility issues are foreseen.

Alternatives

Alternate API

An alternative API could be implemented that does away with some of the simpler commands.

Note: This alternate section was modified after describing the process to documentation personnel, where it was determined that having the ‘simple’ commands made it significantly easier to understand the logging process.

Openstack functionality:

API	Description
log-list	As above
log-show	Removed
log-enable	Removed
log-disable	Removed
log-publish	As above
log-discard	Removed
log-tail	As above
log-save	As above

–publish by default

There has also been a suggestion to make the –publish option the default on the log-tail and log-save commands. If this were done, it seems unlikely that anyone would ever execute –no-publish and the log-publish command would really only ever execute “log-publish –disable”, so it would be better to just eliminate the –publish/–no-publish options, have log-tail and log-save always publish first, and replace the log-publish command with log-disable.

Admin override for guest log

It is possible for the operator to exclude the guest agent log from the list of logs returned to the user. Doing so, however, will not prevent the operator from seeing the guest log, as they would access it through the admin account. Currently the admin user will still stream the logs to the same tenant as the Trove user. This could be enhanced (in the future) to have the admin user provide a tenant that could be used to host the log containers.

Dashboard Impact (UX)

TBD (section added after approval)

Implementation

Assignee(s)

Primary assignee:: vgnbkr (morgan@tesora.com)
Secondary assignees:: peterstac (peter@tesora.com) atomic77 (atomic@tesora.com)
Documentation:: laurelm (lmichaels@tesora.com)

Milestones

Target Milestone for completion:: Mitaka-1

Work Items

This component has been largely implemented.

Upgrade Implications

As this is entirely new functionality, no upgrade implications are foreseen.

Dependencies

n/a

Testing

An integration scenario test will be added to retrieve the guest log, execute a command, retrieve the guest log again and confirm that additional logging details were captured. The log will then be deleted via the log-discard command and the removal of the files from the container from Swift will be verified. The TestHelper class will be enhanced to include the ‘default’ log list (which is simply ‘guest’), and the MySQL one the additional defined USER logs.

Documentation Impact

New documentation describing the added functionality will need to be written.

References

https://etherpad.openstack.org/p/trove-2015-vancouver-logging

Appendix

None

Refactor the Datastore Manager

Wed, 06 Jan 2016 00:00:00

There is a large amount of boiler-plate and/or copy-and-paste code in each datastore manager. As more managers are added, the time involved in maintaining all this code will continue to grow. To alleviate this and the bugs it promotes, a base manager class needs to be created where all common code can reside.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/datastore-manager-refactor

Problem Description

When bugs are discovered and fixed in the manager code of one datastore, these changes are seldom transferred to all the other datastores. This causes a ‘drift’ in the stability of each experimental datastore as currently there are no third-party CI’s. In addition, when ‘new improved’ ways to solve a problem are implemented they are done differently across each manager, if at all. It is also difficult to transfer features (and coding knowledge) from one datastore to another, and the problem is exacerbated as code is copied-and-pasted when a new datastore manager is implemented.

A current example of this problem is the issue where instances flip from BUILD to ACTIVE and back again 1, or from BUILD->SHUTDOWN->ACTIVE. 2 Fixing this means changing how prepare works, and currently this change would be required to be made in each datastore manager. Implementing this in a uniform way would be almost impossible with the current architecture of the manager code base.

Proposed Change

A new ‘Manager’ class will be created that will be the base class for all datastore managers. In order to keep the scope reasonable, only the barest minimum of functionality will be pulled back into the base class in the initial implementation - this will include methods that are currently ‘boiler-plate’ code (such as the response to the rpc_ping and get_fileststem_stats calls, and the periodic task, update_status).

A mechanism for encapsulating functionality will also be needed in order for the base manager to be able to execute datastore-specific instructions. This will be accomplished through the use of properties that can be overridden by each descendant. Some required ones (such as the ‘status’ object) will be declared abstract and must be present; others (such as the new configuration manager, and potentially a future dictionary of strategies) will be optional.

The new Manager class will reside in the datastore module:

trove/guestagent/datastore/manager.py

This is alongside the existing service.py file (which contains the existing BaseDbStatus class). The directory structure of the MySQL derived classes will also be tidied a bit and will end up looking like the following:

trove
+-- guestagent
    +-- datastore
        +-- __init__.py
        +-- manager.py          <-- new 'base' manager
        +-- service.py
        +-- experimental
         +-- __init__.py
         +-- cassandra
          +-- __init__.py
          +-- manager.py
          +-- service.py
          +-- system.py

        <other experimental datastore modules>

         +-- redis
          +-- __init__.py
          +-- manager.py
          +-- service.py
          +-- system.py
         +-- vertica
             +-- __init__.py
             +-- manager.py
             +-- service.py
             +-- system.py
        +-- mysql_common        <-- new module
         +-- __init__.py
         +-- manager.py      <-- renamed from mysql/manager_base.py
         +-- service.py      <-- renamed from mysql/service_base.py
        +-- mysql
         +-- __init__.py
         +-- manager.py
         +-- service.py
        +-- technical-preview
            +-- __init__.py

Within the context of this refactor (as a proof-of-concept), the issue with having instances flip in-and-out of BUILD state will be addressed properly. The prepare method will be moved into the base class, which will seamlessly implement the code required to ensure that no notifications are sent until it is guaranteed that prepare has finished successfully. The existing prepare methods will be renamed ‘do_prepare’ and will be called from within the base prepare method.

This manner of determining whether prepare has finished will be accomplished by having the manager write a file at the start and at the successful conclusion of the prepare operation. Any errors (exceptions) will be trapped and logged and the instance set into the FAILED state.

A sample of what this will look like in the base manager is as follows, although additional properties may be defined as they are deemed necessary (and others can be added in future cleanup work):

class Manager(periodic_task.PeriodicTasks):
    """This is the base class for all datastore managers.  Over time, common
    functionality should be pulled back here from the existing managers.
    """

    def __init__(self, manager_name):

        super(Manager, self).__init__(CONF)

        # Manager properties
        self.__manager_name = manager_name
        self.__manager = None
        self.__prepare_error = False

    @property
    def manager_name(self):
        """This returns the passed-in name of the manager."""
        return self.__manager_name

    @property
    def manager(self):
        """This returns the name of the manager."""
        if not self.__manager:
            self.__manager = CONF.datastore_manager or self.__manager_name
        return self.__manager

    @property
    def prepare_error(self):
        return self.__prepare_error

    @prepare_error.setter
    def prepare_error(self, prepare_error):
        self.__prepare_error = prepare_error

    @property
    def configuration_manager(self):
        """If the datastore supports the new-style configuration manager,
        it should override this to return it.
        """
        return None

    @abc.abstractproperty
    def status(self):
        """This should return an instance of a status class that has been
        inherited from datastore.service.BaseDbStatus.  Each datastore
        must implement this property.
        """
        return None

    ################
    # Status related
    ################
    @periodic_task.periodic_task
    def update_status(self, context):
        """Update the status of the trove instance. It is decorated with
        perodic_task so it is called automatically.
        """
        LOG.debug("Update status called.")
        self.status.update()

    def rpc_ping(self, context):
        LOG.debug("Responding to RPC ping.")
        return True

    #################
    # Prepare related
    #################
    def prepare(self, context, packages, databases, memory_mb, users,
                device_path=None, mount_point=None, backup_info=None,
                config_contents=None, root_password=None, overrides=None,
                cluster_config=None, snapshot=None):
        """Set up datastore on a Guest Instance."""
        LOG.info(_("Starting datastore prepare for '%s'.") % self.manager)
        self.status.begin_install()
        post_processing = True if cluster_config else False
        try:
            self.do_prepare(context, packages, databases, memory_mb,
                            users, device_path, mount_point, backup_info,
                            config_contents, root_password, overrides,
                            cluster_config, snapshot)
        except Exception as ex:
            self.prepare_error = True
            LOG.exception(_("An error occurred preparing datastore: %s") %
                          ex.message)
            raise
        finally:
            LOG.info(_("Ending datastore prepare for '%s'.") % self.manager)
            self.status.end_install(error_occurred=self.prepare_error,
                                    post_processing=post_processing)
        # At this point critical 'prepare' work is done and the instance
        # is now in the correct 'ACTIVE' 'INSTANCE_READY' or 'ERROR' state.
        # Of cource if an error has occurred, none of the code that follows
        # will run.
        LOG.info(_('Completed setup of datastore successfully.'))

        # We only create databases and users automatically for non-cluster
        # instances.
        if not cluster_config:
            try:
                if databases:
                    LOG.debug('Calling add databases.')
                    self.create_database(context, databases)
                if users:
                    LOG.debug('Calling add users.')
                    self.create_user(context, users)
            except Exception as ex:
                LOG.exception(_("An error occurred creating databases/users: "
                                "%s") % ex.message)
                raise

        try:
            LOG.debug('Calling post_prepare.')
            self.post_prepare(context, packages, databases, memory_mb,
                              users, device_path, mount_point, backup_info,
                              config_contents, root_password, overrides,
                              cluster_config, snapshot)
        except Exception as ex:
            LOG.exception(_("An error occurred in post prepare: %s") %
                          ex.message)
            raise

    @abc.abstractmethod
    def do_prepare(self, context, packages, databases, memory_mb, users,
                   device_path, mount_point, backup_info, config_contents,
                   root_password, overrides, cluster_config, snapshot):
        """This is called from prepare when the Trove instance first comes
        online.  'Prepare' is the first rpc message passed from the
        task manager.  do_prepare handles all the base configuration of
        the instance and is where the actual work is done.  Once this method
        completes, the datastore is considered either 'ready' for use (or
        for final connections to other datastores) or in an 'error' state,
        and the status is changed accordingly.  Each datastore must
        implement this method.
        """
        pass

    def post_prepare(self, context, packages, databases, memory_mb, users,
                     device_path, mount_point, backup_info, config_contents,
                     root_password, overrides, cluster_config, snapshot):
        """This is called after prepare has completed successfully.
        Processing done here should be limited to things that will not
        affect the actual 'running' status of the datastore (for example,
        creating databases and users, although these are now handled
        automatically).  Any exceptions are caught, logged and rethrown,
        however no status changes are made and the end-user will not be
        informed of the error.
        """
        LOG.debug('No post_prepare work has been defined.')
        pass

    #####################
    # File System related
    #####################
    def get_filesystem_stats(self, context, fs_path):
        """Gets the filesystem stats for the path given."""
        # TODO(peterstac) - note that fs_path is not used in this method.
        mount_point = CONF.get(self.manager).mount_point
        LOG.debug("Getting file system stats for '%s'" % mount_point)
        return dbaas.get_filesystem_volume_stats(mount_point)

    #################
    # Cluster related
    #################
    def cluster_complete(self, context):
        LOG.debug("Cluster creation complete, starting status checks.")
        self.status.end_install()

The base service class will be enhanced to contain the necessary methods to set a flag denoting whether prepare has finished of not. This will look something like the following (only new code is shown):

class BaseDbStatus(object):

    GUESTAGENT_DIR = '~'
    PREPARE_START_FILENAME = '.guestagent.prepare.start'
    PREPARE_END_FILENAME = '.guestagent.prepare.end'

    def __init__(self):
        self._prepare_completed = None

    @property
    def prepare_completed(self):
        if self._prepare_completed is None:
            # Force the file check
            self.prepare_completed = None
        return self._prepare_completed

    @prepare_completed.setter
    def prepare_completed(self, value):
        # Set the value based on the existence of the file; 'value' is
        # ignored
        # This is required as the value of prepare_completed is cached,
        # so this must be referenced any time the existence of the
        # file changes
        self._prepare_completed = os.path.isfile(
            guestagent_utils.build_file_path(
                self.GUESTAGENT_DIR, self.PREPARE_END_FILENAME))

    def begin_install(self):
        """Called right before DB is prepared."""
        prepare_start_file = guestagent_utils.build_file_path(
            self.GUESTAGENT_DIR, self.PREPARE_START_FILENAME)
        operating_system.write_file(prepare_start_file, '')
        self.prepare_completed = False

        self.set_status(instance.ServiceStatuses.BUILDING, True)

    def end_install(self, error_occurred=False, post_processing=False):
        """Called after prepare completes."""

        # Set the "we're done" flag if there's no error and
        # no post_processing is necessary
        if not (error_occurred or post_processing):
            prepare_end_file = guestagent_utils.build_file_path(
                self.GUESTAGENT_DIR, self.PREPARE_END_FILENAME)
            operating_system.write_file(prepare_end_file, '')
            self.prepare_completed = True

        final_status = None
        if error_occurred:
            final_status = instance.ServiceStatuses.FAILED
        elif post_processing:
            final_status = instance.ServiceStatuses.INSTANCE_READY

        if final_status:
            LOG.info(_("Set final status to %s.") % final_status)
            self.set_status(final_status, force=True)
        else:
            self._end_install_or_restart(True)

    def end_restart(self):
        self.restart_mode = False
        LOG.info(_("Ending restart."))
        self._end_install_or_restart(False)

    def _end_install_or_restart(self, force):
        """Called after DB is installed or restarted.
        Updates the database with the actual DB server status.
        """
        real_status = self._get_actual_db_status()
        LOG.info(_("Current database status is '%s'.") % real_status)
        self.set_status(real_status, force=force)

    @property
    def is_installed(self):
        """This is for compatibility - it may be removed during further
        cleanup.
        """
        return self.prepare_completed

    def set_status(self, status, force=False):
        """Use conductor to update the DB app status."""

        if force or self.is_installed:
            LOG.debug("Casting set_status message to conductor "
                      "(status is '%s')." % status.description)
            context = trove_context.TroveContext()

            heartbeat = {'service_status': status.description}
            conductor_api.API(context).heartbeat(
                CONF.guest_id, heartbeat, sent=timeutils.float_utcnow())
            LOG.debug("Successfully cast set_status.")
            self.status = status
        else:
            LOG.debug("Prepare has not completed yet, skipping heartbeat.")

Configuration

No configuration changes are anticipated.

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

The ServiceStatuses.BUILD_PENDING state has been renamed to ServiceStatuses.INSTANCE_READY to better reflect the instance’s actual state. The value displayed will remain ‘BUILD’ so that there should be no outward differences and thus backwards compatibility will be maintained.

Guest Agent

This change should not affect any behavior on the Guest Agent. The current tests should be adequate to ensure that the change is fully compatible with the rest of the code base.

Alternatives

One suggestion as to how to fix the prepare issue was to use Nova metadata, which is available on the guest instance. If it is decided that this would be useful, it could be added in addition to the proposed method as a means of notification only.

Dashboard Impact (UX)

TBD (section added after approval)

Implementation

Assignee(s)

Primary assignee:: <peterstac>

Milestones

Target Milestone for completion:: eg. Mitaka-1

Work Items

All changes will be done in the context of a single task.

Upgrade Implications

None are anticipated.

Dependencies

None

Testing

The unittests will be modified as necessary, however minimal changes will be made in this regard as well. In order to keep the changes as small as possible, refactoring the tests will also be done in stages with only the bare minimum done to start and the remainder being left to a future date. The future work would include testing the base class thoroughly and then removing all the corresponding tests from the derived classes.

The int-tests should continue to run as always, and will be used to determine that no fundamental changes have been made to the implementation, with the exception of the bug fixes (and they should just lead to greater stability in the test infrastructure).

Documentation Impact

Since all the changes are implementation related, no documentation changes are foreseen.

References

1: https://bugs.launchpad.net/trove/+bug/1482795
2: https://bugs.launchpad.net/trove/+bug/1487233

Appendix

None

DB2 Backup And Restore

Wed, 06 Jan 2016 00:00:00

The DB2 Express C datastore in Trove lacks backup and restore functionality at the moment. This blueprint aims at adding the feature of backup and restore to DB2 in Trove.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/db2-backup-restore

Problem Description

Currently, there is no way to create backups of DB2 databases using Trove nor is it possible to perform a restore from a backup. Since this functionality is core to a data model and typical use case, it is imperative for Trove to add it.

Proposed Change

The simplest form of database backup and restore will be implemented for DB2 creating a backup image in a default location. Since by default, DB2 databases are in a circular logging mode, it isn’t possible to take an online backup. Therefore a full offline backup functionality will be implemented by this feature 1 . At this time, incremental backups for db2 will not be supported. The LIST UTILITIES command will be used to monitor the process. 2

Backup

The backup procedure for DB2 will follow a process as below. 3

Determine the directory where the backups are written. For DB2 databases the default directory location can be found in the config file (/etc/db2/db2 .conf)
Check if sufficient disk space is available for the backup on the instance.
Initiate a DB2 backup through the cli. The BACKUP DATABASE command will be used for this purpose.
Once the backup is complete, compress/encrypt the backup inside the volume storage. The name of the backup file will have the following format - DB_alias.Type.Inst_name.NODEnnnn.CATNnnnn.timestamp.Seq_num
The db2ckbkp command will be used to display information about existing backup images. This will verify the newly created database backup.
The compressed/encrypted file can then be stored in Swift under the database_backups container.
The old backup files can then be deleted on the VM once stored in Swift.

Since db2 only supports streaming to named pipes, the backup cannot be performed in a single step and will first need to store the backup file on the VM. 4

Restore

In order to restore a DB2 database from backup, the following procedure will be followed. 5

Determine the directory where the backup files are stored
Ensure that no other applications are running against the database
Retrieve the backup from storage to the proper location
Use the RESTORE DATABASE command to recover the database

Configuration

The default configuration values of the following will need to change to correspond to the respective DB2 locations.

backup_strategy
backup_namespace
restore_namespace

Database

None

Public API

The implmentation of the DB2 backup and restore functionality will allow users to use several CLI commands for the DB2 datastore.

Public API Security

None

Python API

None

CLI (python-troveclient)

The following commands will be functional for DB2

backup-create
backup-delete
backup-list
backup-list-instance
backup-show
create –backup

Internal API

None

Guest Agent

The DB2 guest agent will be modified to support backup and restore. In particular the following files will have added components-

guestagent/strategies/backup/experimental/db2_impl.py
guestagent/strategies/restore/experimental/db2_impl.py

The following existing files will be updated:

guestagent/datastore/experimental/db2/manager.py

It will be backwards compatible with API and Task Manager.

Alternatives

The other type of backup that can be implmented is online backup so the database won’t need to shut down before taking backups. However, for online backups to be implemented, the database must have archive logging options. At this time it is not possible to specify logging options for DB2 databases in Trove. Once users can configure other logging options like archive logging, online backup can be implemented.

Dashboard Impact (UX)

TBD (section added after approval)

Implementation

Assignee(s)

Primary assignee:: imandhan

Milestones

Target Milestone for completion:

Mitaka-1

Work Items

Implement backup and restore API calls.
Write associated test cases.

Upgrade Implications

None

Dependencies

None

Testing

Unit tests will be added as necessary for the backup and restore functionality. Also, a db2 helper will be added to the existing int-test framework.

Documentation Impact

The DB2 Trove documentation should be updated to indicate that backup and restore is supported.

References

1: Offline Backup: http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.ha.doc/doc/c0051343.html
2: Monitoring Backup: http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.ha.doc/doc/t0060260.html
3: DB2 Backup procedure: http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.ha.doc/doc/c0006150.html
4: DB2 backing up to named pipe: https://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.5.0/com.ibm.db2.luw.admin.ha.doc/doc/t0006202.html
5: DB2 Restore procedure: http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.ha.doc/doc/c0006237.html

Appendix

None

Move to oslo_db

Wed, 06 Jan 2016 00:00:00

Replace the trove.db.sqlalchemy package with the oslo_db library.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/move-to-oslo-db

Problem Description

The current code uses the outdated incubator package for database connectivity. The old package has some bugs such as not handling concurrent threads correctly. This is a good opportunity to migrate to the oslo_db library, replacing the trove.db.sqlalchemy package.

A similar move has been done by other OS projects such as nova 1.

Proposed Change

oslo_db is a library that handles the common database connection and controls. Change the trove.db.sqlalchemy engine and session handling to use the oslo_db library. This transparently handles connections and sessions, and protects against mismanaged connections in Trove’s threaded environment.

The only external effect will be the exposure of oslo_db configuration options.

Configuration

Add the configuration options for oslo_db to the [database] section 2.

Database

The database connection is the only change. Data models and queries are not affected.

Public API

N/A

Public API Security

N/A

Python API

N/A

CLI (python-troveclient)

N/A

Internal API

N/A

Guest Agent

N/A

Alternatives

Determine the root cause of the database connection mismanagement 3. Previous efforts were unsuccessful.

Dashboard Impact (UX)

TBD (section added after approval)

Implementation

Remove the engine and session management code and replace it with a facade from oslo_db.

Assignee(s)

Matthew Van Dijk

Milestones

Mitaka-1

Work Items

A single task for code changes
Update docs with the new configuration settings

Upgrade Implications

N/A

Dependencies

oslo_db

Testing

Do not break existing tests - especially the fake mode tests.

Documentation Impact

Describe the config variables that are in the database section.

References

1: Nova’s migration commit.
2: oslo_db configuration options.
3: Associated bug on Launchpad.

Appendix

None

PostgreSQL Incremental Backup and Restore

Wed, 06 Jan 2016 00:00:00

Trove currently only supports full backup and restore with the PostgreSQL guest agent.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/postgresql-incremental-backup

Problem Description

Currently, backups in PostgreSQL are done with the pg_dump and pg_restore tools, which produce efficient, logical backups. Unfortunately, they cannot be used as the basis for incremental backups or point-in-time recovery.

Proposed Change

PostgreSQL Logging

In contrast to MySQL, PostgreSQL uses the same foundation for recovery, replication and incremental backups. Central to this is the write-ahead log (WAL) 1. With each update to the database, a WAL entry is created. The WAL contents are periodically checkpointed to the main data files in the database directory and are eventually purged depending on the system configuration.

In the event of a system crash, recovery proceeds by replaying the contents of the WAL since the last checkpoint. This same mechanism is used, in a “continuous” sense, for replication, and up until a specific WAL entry, for point-in-time recovery 2.

So in order to support incremental backup, two parts are required:

A base backup, a snapshot of the filesystem contents of the pgsql data directory
The WAL files written since the previous backup up to the desired point in time

Base Backups

Base backups are taken with the pg_basebackup filesystem-level backup tool 3. pg_basebackup creates a copy of the binary files in the PostgreSQL cluster data directory and enough WAL segments to allow the database to be brought back to a consistent state. Associated with each backup is a log location, normally indicated by the WAL file name and the position inside the file.

Point-in-time Recovery and WAL Archiving

PostgreSQL can be made to support Trove’s notion of incremental backup and restore by bootstrapping the recovery system. To restore an incremental backup from a previous backup, in PostgreSQL, is to replay the WAL entries to a designated point in time – all that is required is the most recent base backup, and all WAL files that were written since.

During normal operation of the database, as WAL data is written and then checkpointed to the data files, unneeded files are purged. PostgreSQL can also be configured to automatically archive WAL files, which can then be used for the purposes of point-in-time recovery.

PostgreSQL has two important configuration parameters that manage WAL archiving and WAL retrieval for recovery: archive_command and recovery_command.

The parameters allow a great deal of flexibility in how WAL files are handled – all that is required is that the command returns 0 only in the event that the file was successfully transferred. A simple example:

archive_command = 'test ! -f /mnt/arch/%f && cp %p /mnt/arch/%f'

The above command checks for the existence of the file in the archive directory /mnt/arch (to avoid overwritting an existing file) and then copies it in.

Similarly, for recovery:

restore_command = 'cp /mnt/arch/%f "%p"'

The above commands assume that /mnt/arch is accessible and, for recovery, contains the appropriate WAL files.

In principle archive_command can be anything, with a few important caveats:

The speed of the archive_command must be fast enough to keep up with the on-going generation of WAL files
The restore command must be able to reverse whatever operation is applied to the WAL file on archive

Assuming that an appropriate mechanism to archive and retrieve the WAL files is in place, incremental backup and restore become simple operations: an incremental “backup” is done by creating a restore point, using pg_start_backup('my_restore_pt'), and the equivalent restore is done by restarting the server with a recovery_target_name = 'my_restore_pt' in the recovery.conf file.

Incremental Collection of WAL Files

Consider the following series of WAL files that may be found in the PostgreSQL WAL archive directory:

$ ls -lh /mnt/wal_arch
-rw------- 1 postgres postgres  16M Oct 27 19:11 000000010000000000000016
-rw------- 1 postgres postgres  16M Oct 27 19:11 000000010000000000000017
-rw------- 1 postgres postgres  16M Oct 27 19:12 000000010000000000000018
-rw------- 1 postgres postgres  16M Oct 27 19:12 000000010000000000000019
-rw------- 1 postgres postgres  16M Oct 27 19:12 00000001000000000000001A

The WAL entry stream can be visualized as:

--------------------------------------------------------
|   16   |    17    |   18   |    19    |    1A   | ...
--------------------------------------------------------
     ^                  ^                    ^
    B1                  I1                   I2

Suppose that base backup B1 was taken at file 16 position 48, incremental backup I1 at file 18 position 30, and incremental backup I2 at file 1A position 20.

I1 would consist of WAL files 16 through 18: the entries after position 48 in file 16, where B1 was taken, would be needed, along with the contents of file 18 up until position 30.

I2 would consist of WAL files 18 through 1A.

This approach has the benefit of being consistent with the current paradigm used by Trove for incremental backup and restore. The main downside is that WAL data must be archived in two stages: once by PostgreSQL, local to the instances, and a second time by Trove, to object storage. This introduces more complexity and increases storage requirements on the instance.

Configuration

The new incremental backup and restore strategies will need to be added.

Database

None.

Public API

None.

Public API Security

None.

Python API

None.

CLI (python-troveclient)

After implementation, the following CLI commands will work:

trove backup-create <inst> <inc_backup> --parent <backup_id>
trove create <inst> .. --backup <inc_backup_id>

Internal API

None.

Guest Agent

A new backup and restore strategy based on pg_basebackup will be added.

Alternatives

Tighter Postgresql - Swift Integration

A simpler approach would be to bypass the two-stage archive process, and have PostgreSQL automatically manage the WAL archiving process to and from object storage. For example, the archive command could run the WAL file through a fast compressor such as Snappy or LZOP, encrypt and then ship the file to Swift directly, with an equivalent reverse procedure for recovery.

This has the benefit of being simpler to implement, as most log handling is pushed down to PostgreSQL, but has the significant side-effect of introducing a relatively continuous stream of WAL data from the PostgreSQL guest to Swift, something potentially unexpected and not consistent with the approach to incremental backups on other datastores.

Dashboard Impact (UX)

TBD (section added after approval)

Implementation

Assignee(s)

Primary assignee:: atomic77

Milestones

mitaka-1

Work Items

pg_basebackup full backup
incremental backup
integration tests

Upgrade Implications

Backups taken with the old PgDump strategy will not be compatible with this new strategy.

Dependencies

Ability to create pgsql instances in the generic int-test framework.

Testing

Int-tests for incremental backup currently do not exist in the new generic int-test framework and will be added. Unit tests will be added as necessary.

Documentation Impact

The documentation will need to be updated to reflect the new backup and restore strategy.

References

1: http://www.postgresql.org/docs/9.4/static/wal-configuration.html
2: http://www.postgresql.org/docs/9.4/static/continuous-archiving.html
3: http://www.postgresql.org/docs/current/static/app-pgbasebackup.html

Appendix

N/A

Postgres Configuration Groups

Wed, 06 Jan 2016 00:00:00

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/postgres-configuration-groups

Problem Description

The Postgres guestagent currently does not support configuration groups.

Proposed Change

The patch set will implement configuration groups for Postgres 9.3 or above.

Postgres stores its configuration 1 in ‘postgresql.conf’ file. Configuration directives have key-value format: “keyword = value”

In addition to the ‘postgresql.conf’ file already mentioned, PostgreSQL uses two other configuration files (‘pg_hba.conf’ and ‘pg_ident.conf’), which control client authentication.

These will be written directly by the guest-agent and will not be exposed via the configuration groups mechanism.

Configuration changes can be loaded on the fly without interrupting any open connections by the special ‘pg_reload_conf’ call 2. Note that Postgres also has a ‘SET’ command, but it only affects the current session values.

Most configuration properties will be available via configuration groups. Some, however, do not make sense in the Trove context.

These include (will be documented in the configuration template):

guestagent specific (e.g. file paths, passwords, file access rules)

items that Trove needs to control (replication/clustering/log properties)

See Available Configuration Properties for the full list of supported options.

Configuration

None

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

CLI (python-troveclient)

This work will enable the following client commands:

configuration-attach

configuration-create

configuration-default

configuration-delete

configuration-detach

configuration-instances

configuration-list

configuration-parameter-list

configuration-parameter-show

configuration-patch

configuration-show

configuration-update

Guest Agent

Implement update_overrides and apply_overrides in the manager and service modules.
A configuration template and validation rules will with changes noted in Available Configuration Properties will be provided.

The existing ‘PropertiesCodec’ implementation will be reused to handle text-file operations. Configuration overrides will be implemented using the ‘OneFileOverrideStrategy’ of the guestagent configuration manager.

Alternatives

None

Dashboard Impact (UX)

TBD (section added after approval)

Implementation

Assignee(s)

Petr Malik <pmalik@tesora.com>

Milestones

Mitaka

Work Items

All changes will be done in the context of a single task.

Implement configuration-related manager API calls:

def update_overrides(self, context, overrides, remove=False)
def apply_overrides(self, context, overrides)

Upgrade Implications

None

Dependencies

None

Testing

The change is largely covered by the existing configuration tests. Unit tests will be added to validate any Postgres-specific codepaths. A ‘postgres_helper’ will be contributed to the scenario tests to enable configuration coverage on this datastore.

Documentation Impact

The datastore documentation should be updated to reflect the enabled features.

References

1: Documentation on Postgres configuration: http://www.postgresql.org/docs/9.3/static/runtime-config.html
2: Notes on Postgres system administration functions: http://www.postgresql.org/docs/9.3/static/functions-admin.html

Appendix

Available Configuration Properties

The properties configurable by the user via the Trove API:

max_connections

superuser_reserved_connections

bonjour

bonjour_name

authentication_timeout

password_encryption

db_user_namespace

tcp_keepalives_idle

tcp_keepalives_interval

tcp_keepalives_count

shared_buffers

huge_pages

temp_buffers

max_prepared_transactions

work_mem

maintenance_work_mem

autovacuum_work_mem

max_stack_depth

dynamic_shared_memory_type

temp_file_limit

max_files_per_process

vacuum_cost_delay

vacuum_cost_page_hit

vacuum_cost_page_miss

vacuum_cost_page_dirty

vacuum_cost_limit

bgwriter_delay

bgwriter_lru_maxpages

bgwriter_lru_multiplier

effective_io_concurrency

max_worker_processes

fsync

synchronous_commit

wal_sync_method

full_page_writes

wal_log_hints

wal_buffers

wal_writer_delay

commit_delay

commit_siblings

checkpoint_segments

checkpoint_timeout

checkpoint_completion_target

checkpoint_warning

wal_keep_segments

wal_sender_timeout

synchronous_standby_names

vacuum_defer_cleanup_age

hot_standby

max_standby_archive_delay

max_standby_streaming_delay

wal_receiver_status_interval

hot_standby_feedback

wal_receiver_timeout

enable_bitmapscan

enable_hashagg

enable_hashjoin

enable_indexscan

enable_indexonlyscan

enable_material

enable_mergejoin

enable_nestloop

enable_seqscan

enable_sort

enable_tidscan

seq_page_cost

random_page_cost

cpu_tuple_cost

cpu_index_tuple_cost

cpu_operator_cost

effective_cache_size

geqo

geqo_threshold

geqo_effort

geqo_pool_size

geqo_generations

geqo_selection_bias

geqo_seed

default_statistics_target

constraint_exclusion

cursor_tuple_fraction

from_collapse_limit

join_collapse_limit

log_truncate_on_rotation

log_rotation_age

log_rotation_size

client_min_messages

log_min_messages

log_min_error_statement

log_min_duration_statement

debug_print_parse

debug_print_rewritten

debug_print_plan

debug_pretty_print

log_checkpoints

log_connections

log_disconnections

log_duration

log_error_verbosity

log_hostname

log_line_prefix

log_lock_waits

log_statement

log_temp_files

log_timezone

track_activities

track_counts

track_io_timing

track_functions

track_activity_query_size

log_parser_stats

log_planner_stats

log_executor_stats

log_statement_stats

autovacuum

log_autovacuum_min_duration

autovacuum_max_workers

autovacuum_naptime

autovacuum_vacuum_threshold

autovacuum_analyze_threshold

autovacuum_vacuum_scale_factor

autovacuum_analyze_scale_factor

autovacuum_freeze_max_age

autovacuum_multixact_freeze_max_age

autovacuum_vacuum_cost_delay

autovacuum_vacuum_cost_limit

search_path

default_tablespace

temp_tablespaces

check_function_bodies

default_transaction_isolation

default_transaction_read_only

default_transaction_deferrable

session_replication_role

statement_timeout

lock_timeout

vacuum_freeze_min_age

vacuum_freeze_table_age

vacuum_multixact_freeze_min_age

vacuum_multixact_freeze_table_age

bytea_output

xmlbinary

xmloption

datestyle

intervalstyle

timezone

timezone_abbreviations

extra_float_digits

client_encoding

lc_messages

lc_monetary

lc_numeric

lc_time

default_text_search_config

deadlock_timeout

max_locks_per_transaction

max_pred_locks_per_transaction

array_nulls

backslash_quote

default_with_oids

escape_string_warning

lo_compat_privileges

quote_all_identifiers

sql_inheritance

standard_conforming_strings

synchronize_seqscans

transform_null_equals

exit_on_error

restart_after_crash

Guestagent-controlled properties:

data_directory

hba_file

ident_file

external_pid_file

listen_addresses

port

unix_socket_directories

unix_socket_group

unix_socket_permissions

wal_level

archive_mode

archive_command

archive_timeout

log_destination

logging_collector

log_directory

log_filename

log_file_mode

update_process_title

Template for New Trove Specs

Wed, 06 Jan 2016 00:00:00

Introduction paragraph – what is the motivation for the spec/blueprint? (Don’t forget to change the title above to something more relevant.)

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/name-of-blueprint

Problem Description

A detailed description of the problem.

Proposed Change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

https://review.openstack.org/#/q/status:open+project:openstack/trove-specs+message:apiimpact,n,z

Code snippets, etc. should be placed in appropriately marked blocks:

# This is a bash command
ls -lf

# sample code
for count in range(1, 10):
    print count

Configuration

Does this impact any configuration files? If so, which ones?

Database

Does this impact any existing tables? If so, which ones? Are the changes forward and backward compatible? Be sure to include the expected migration process

Public API

Does this change any API that an end-user has access to? Are there any exceptions in terms of consistency with other APIs?

Public API Security

If this change proposes a new API, or if this change relates to security on an existing API, provide details here.

What are the expectations of, and implications to security on the Public API.

Python API

Does this change the Python API? If anything was removed, has it been properly marked as deprecated?

CLI (python-troveclient)

Will the Trove CLI need to be modified? If the CLI will just implement the changes mentioned in the Python API section, it may be enough to just mention it here.

Internal API

Does this change any internal messages between API and Task Manager or Task Manager to Guest?

Guest Agent

Does this change behavior on the Guest Agent? If so, is it backwards compatible with API and Task Manager?

Alternatives

This is an optional section, where it does apply we’d just like a demonstration that some thought has been put into why the proposed approach is the best one.

Dashboard Impact (UX)

This section should detail how the dashboard (Horizon) should display the new changes, if relevant. For example, if adding cluster support for Redis, this section could say:

Enabling Redis clustering will simply reuse the existing Launch Cluster
dialog. The Redis datastore will be in the datastore pulldown. When the
user selects Redis the Launch Cluster dialog fields will dynamically
change to display the default Launch Cluster fields.

There will be a new detail overview panel with Redis cluster specific
information.

There are no additional actions to be added at this point for the Redis
cluster.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a spec where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Dashboard assignee:: <launchpad-id or None>

Can list additional ids if they intend on doing substantial implementation work on this spec.

Milestones

Target Milestone for completion:: eg. Liberty-1

Work Items

Upgrade Implications

In this section, describe the upgrade implications (if any) of the proposed change. This could include such details as:

changes to location of files, or layout of the source tree if this impacts configuration files,
invalidates old backups,
changes the CLI in a manner that could impact existing scripting,
eliminates or adds new notifications (events),
any changes that an operator or user must perform as part of the upgrade.

If the change has upgrade implications, also remember to:

add the DocImpact keyword to the commit, and
provide sufficient information in the commit message or in the documentation bug that gets created.

For more information about the DocImpact keyword, refer to https://wiki.openstack.org/wiki/Documentation/DocImpact

Dependencies

Include specific references to specs and/or blueprints in Trove, or in other projects, that this one either depends on or is related to.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Documentation Impact

References

1: Links to mailing list or IRC discussions
2: Links to notes from a summit session
3: Links to relevant research, if appropriate
4: Anything else you feel it is worthwhile to refer to

Appendix

Any additional technical information and data.

Vertica Cluster Grow and Shrink Support

Mon, 04 Jan 2016 00:00:00

The Vertica database has elastic grow/shrink capabilities which are not currently supported by the Vertica guest agent for Trove.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/vertica-grow-shrink-cluster

Problem Description

The Vertica guest agent currently does not leverage the underlying elastic capabilities of Vertica. This will enable a user to grow a cluster in the event that they wish to accommodate more data or enable faster query performance, while scaling down helps avoid costs associated with overprovisioning.

Proposed Change

As Vertica was architected from the ground up to be a clustered system, adding and removing nodes is relatively simple in comparison to other datastores.

Configuration

A minimum k-safety configuration option will be added for vertica to allow the operator to decide their desired level of fault tolerance.

Database

None

Public API

The following public API calls will be made available to the Vertica datastore.

Cluster Grow - The existing call payload will not be changed. Implementing the grow cluster feature will add the new instances to the existing cluster.
Cluster Shrink - The existing call payload will not be changed. Implementing the shrink cluster feature will allow a user to remove instances from their existing cluster.

Public API Security

None

Python API

None

CLI (python-troveclient)

Support for the following existing CLI calls.

cluster-grow
cluster-shrink

No changes should be nessesary to accomplish these actions.

Internal API

None.

Guest Agent

To enable more efficient grow and shrink, local data segmentation will be enabled on Vertica 1. This creates additional local, logical segments of data on a node to enable easier shipping of data between nodes. The number of local segments is configurable with the scaling factor variable. Local segmentation has the drawback of making tables with many hundreds of projections less efficient 2.

Grow

Growing a cluster involves two main steps 3.

First, a new “host” must be added to the cluster, which in the case of trove would mean a new instance. The update_vertica script is then called, similar to the install_vertica script, which handles installation of the vertica binaries.

Second, the host must be added as a node to the database. The adminTools utility is called with the db_add_node command to register the host with the database.

Shrink

Removing a node from a Vertica cluster proceeds inversely to addition, with an extra check to ensure that the minimum k-safety level of the system is maintained.

If a user attempts to remove a node that would lower the k-safety level below the configured level, an error will be thrown.

After the k-safety check, the host is removed from the database 4. Similarly as with grow, the adminTools utility will be called using the db_remove_node command.

Then, the host to be removed is removed from the cluster, using the same update_vertica script but with the –remove-hosts option.

K-safety

Vertica defines three K-safety levels for the number of nodes K that could fail while allowing the cluster to continue to operate: K=0 for clusters with 1 or 2 nodes, K=1 for clusters with 3 or 4 nodes, and K=2 for 5 or more 5 6.

Rather than prevent a user from removing nodes that would result in a lower k-safety value, it is up to the operator to define a minimum level of safety she is willing to accept. For example, in some cases it may be that the costs associated with overprovisioning the cluster outweigh the risk of data being unavailable.

Alternatives

Trove could enforce a minimum k-safety level to ensure the integrity of the cluster, but this could be too restrictive.

Implementation

Assignee(s)

atomic77

Milestones

Mitaka-3

Work Items

Grow cluster
Shrink cluster

Upgrade Implications

None.

Dependencies

None

Testing

Integration tests will be added or modified as needed in order to test grow/shrink with the new int-test framework.

Documentation Impact

The documentation should be updated to reflect the fact that grow and shrink is supported for Vertica clusters.

Dashboard Impact (UX)

There will be some minor changes to the UI to support grow and shrink buttons for the cluster.

References

1: https://my.vertica.com/docs/7.1.x/HTML/index.htm#Authoring/AdministratorsGuide/ClusterManagement/ElasticCluster/LocalDataSegmentation.htm
2: The Vertica documentation recommends local data segmentation be done with numbers of nodes that are a power of two. Some experimentation will be required to see what is whether violating this recommendation is still worthwhile compared to not using local data segmentation at all
3: https://my.vertica.com/docs/7.1.x/HTML/index.htm#Authoring/AdministratorsGuide/ManageNodes/AddingNodes.htm
4: https://my.vertica.com/docs/7.1.x/HTML/index.htm#Authoring/AdministratorsGuide/ManageNodes/RemovingNodes.htm
5: https://my.vertica.com/docs/7.1.x/HTML/index.htm#Authoring/AdministratorsGuide/ManageNodes/LoweringTheK-SafetyLevelToAllowForNodeRemoval.htm
6: https://my.vertica.com/docs/7.1.x/HTML/Content/Authoring/ConceptsGuide/Components/HighAvailabilityAndRecovery.htm

Appendix

None

Vertica Configuration Groups

Thu, 24 Dec 2015 00:00:00

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/vertica-configuration-groups

Problem Description

The Vertica guestagent currently does not support configuration groups for the single instance case.

Proposed Change

While Vertica has minimal required configuration as one of its design goals, it is nonetheless possible to configure a large number of parameters 1.

Databases in Trove and Vertica

While Trove thinks of instances as being the parent of one or more databases, following the model of single-server MySQL, Vertica supports one or more nodes natively, and databases can span multiple nodes. Configuration options can also be applied at both the node and database level.

The Vertica guest agent currently does not support multiple Vertica databases for both the single instance and clustered cases, so only configuration options valid at the database level will be exposed.

Configuration Changes

Vertica recommends against writing configuration changes to a configuration file for version 7.1. It is unclear if this applies to 7.2, or has been remedied, but for the purposes of providing support for both 7.1 and 7.2, writing to this file will be avoided 3. Configurations can be applied with the use of the ALTER DATABASE command, and current configuration settings can be retrieved via the CONFIGURATION_PARAMETERS system view 2.

Vertica supports 360 different options as of Vertica 7.2.1. Options related to authentication and big data support have been excluded. For a complete list of the options that will be supported, see the Appendix section.

Configuration

None

Database

None

Public API

None

Public API Security

None

Python API

None (empty section added after merging)

CLI (python-troveclient)

None (empty section added after merging)

Internal API

CLI (python-troveclient)

This work will enable the following client commands:

configuration-attach

configuration-create

configuration-default

configuration-delete

configuration-detach

configuration-instances

configuration-list

configuration-parameter-list

configuration-parameter-show

configuration-patch

configuration-show

configuration-update

Note that the configuration groups API is not currently supported for clusters, so only support for single-instance Vertica will be provided.

Guest Agent

Implement update_overrides and apply_overrides in the manager and service modules.

The following existing files will be updated:

guestagent/datastore/experimental/vertica/manager.py
guestagent/datastore/experimental/vertica/service.py
guestagent/datastore/experimental/vertica/system.py

Alternatives

None

Implementation

Assignee(s)

Alex Tomic <atomic@tesora.com>

Milestones

mitaka-3

Work Items

provide support for viewing and modifying the vertica database-level configuration options

implement configuration-related manager API calls:

def update_overrides(self, context, overrides, remove=False)
def apply_overrides(self, context, overrides)

Upgrade Implications

None.

Dependencies

None.

Testing

Unit tests will be added to validate implemented functions, and integration tests added or modified from the new scenario testing framework as needed.

Documentation Impact

The datastore documentation should be updated to reflect the enabled features.

Dashboard Impact (UX)

None.

Appendix

The following is the list of parameters that will be supported:

ActivePartitionCount AddressCollectorInterval AdvanceAHMInterval AHMBackupManagement AllowNonAsciiNames AnalyzeRowCountInterval AnalyzeStatsPlanMaxColumns AnalyzeStatsSampleBands ARCCommitPercentage AuditConfidenceLevel AuditErrorTolerance BasicVerticaOptions BlockCacheSize BufferQueryOutputForPossibleRetry CachePositionIndex CascadeResourcePoolAlwaysReplan CatalogCheckpointChunkSizeKB CatalogCheckpointMinLogSizeKB CatalogCheckpointPercent CatalogDeindexRename CheckCRCs CheckDataTargetSortOrder ClusterRecoveryWait CollationExpansion CompressCatalogOnDisk CompressDistCalls CompressNetworkData ComputeApproxNDVsDuringAnalyzeStats ContainersPerProjectionLimit CopyFromVerticaWithIdentity DatabaseHeartbeatInterval DBDCorrelationSampleRowCount DBDCorrelationSampleRowPct DBDCountDistinctSampleRowCount DBDCountDistinctSampleRowPct DBDDeploymentParallelism DBDDynamicSampling DBDEncodingSampleRowCount DBDEncodingSampleRowPct DBDLargestTableRowCountBoundary DBDLogInternalDesignProcess DBDMaxConcurrencyForEncodingExperiment DBDRepLargeRowCountPct DBDRepSmallRowCountPct DBDSampleStorageBandCount DBDSkewDetectionSampleRowCount DBDSkewDetectionSampleRowPct DBDUseOnlyDesignerResourcePool DefaultIntervalStyle DefaultSessionLocale DisableInheritedPrivileges DisableLocalResegmentation DisableNodeDownOptimization DisablePrejoinProjections DisallowMars DiskSpacePollingInterval DMLCancelTM EEVerticaOptions EnableAccessPolicy EnableAllRolesOnLogin EnableApportionLoad EnableAutoDMLStats EnableBlockMemoryManager EnableCooperativeParse EnableDataCollector EnabledCipherSuites EnableEEThreadPool EnableEMMJMultiblockSIPS EnableExprsInProjections EnableForceOuter EnableGroupByProjections EnableJIT EnableNewPrimaryKeysByDefault EnableNewUniqueKeysByDefault EnableParallelHashBuild EnableParallelSort EnablePatternMatchingAnyRow EnablePlanStability EnablePlanStabilityLookup EnableResourcePoolCPUAffinity EnableRuntimePriorityScheduler EnableSSL EnableStorageBundling EnableStrictTimeCasts EnableTopKProjections EnableUDTProjections EnableUniquenessOptimization EnableVirtualCoreCount EpochMapInterval EscapeStringWarning EvaluateDeletePerformanceSampleStorageBandCount EvaluateDeletePerformanceSampleStorageCount ExcludeEphemeralNodesInQueries ExternalTablesExceptionsLimit FailoverToStandbyAfter FencedUDxMemoryLimitMB FilesPerProjectionLimit FlexTableDataTypeGuessMultiplier FlexTableRawSize ForceUDxFencedMode FsyncCatalogForLuck FsyncDataForLuck GBHashMemCapMB GlobalEEProfiling GlobalHeirUsername GlobalQueryProfiling GlobalSessionProfiling GroupGeneratorHashingEnabled HadoopConfDir HCatConnectionTimeout HCatParserName HCatSlowTransferLimit HCatSlowTransferTime HCatSourceName HCatWebserviceName HCatWebserviceVersion HistoryRetentionEpochs HistoryRetentionTime JavaBinaryForUDx JavaSideProcessMinHeapSizeMB KeepMinMaxOnAllColumns LGELagThreshold LoadMaxFinalROSCount LockTimeout LogHeartbeatInterval LowDiskSpaceWarningPct MaxAutoSegColumns MaxBundleableROSSizeKB MaxClientSessions MaxConstraintChecksPerQuery MaxDataCollectorFileSize MaxDesiredEEBlockSize MaxDVROSPerContainer MaxLogLineLength MaxMrgOutROSSizeMB MaxOptMemMB MaxOptMemMBInDBD MaxParsedQuerySizeMB MaxPartitionCount MaxQueryRetries MaxRecoverErrors MaxRecoverHistoricPasses MaxRefreshErrors MaxRefreshHistoricPasses MaxROSPerStratum MergeJoinInnerInitialMB MergeOutCache MergeOutInterval MinimumCatalogDiskMegabytes MinimumDataDiskMegabytes MinimumDataDiskTempMegabytes MinSortMergeJoinMB MoveOutInterval MoveOutMaxAgeEpochs MoveOutMaxAgeTime MoveOutSizePct NewEEGroupBySmallMemMB NewEEROSSubdivisionRows NewEEThreads NoRecoverShutdownWait OptVerticaOptions ParallelizeLocalSegmentLoad PatternMatchAllocator PatternMatchingMatchLimitRecursion PatternMatchingMaxPartition PatternMatchingMaxPartitionMatches PatternMatchingPerMatchWorkspaceSize PatternMatchingUseJit PatternMatchStackAllocator PinProcessors PinProcessorsOffset PreExcavatorReplicatedProjection PruneDataCollectorByTime PruneSystemTableColumns PurgeMergeoutPercent RangeWindowMaxMem ReapBeforeRecover RecoverByContainer RecoveryDirtyTxnWait ReflexiveMoveout RefreshByContainer RemoteInitiatorBufSize RemoveSnapshotInterval ReplayDeleteAlgorithmSwitchThreshold ResLowLimPctOfHighLim RestrictSystemTables ROSCacheBlocks ROSCacheLargeBlocks ROSPerStratum SaveDCEEProfileThresholdUS SecurityAlgorithm SegmentAutoProjection SegmentDataCollector SessionProfilingAgeOut SlowDeleteConsoleWarningLimit SlowDeleteSystemWarningLimit SmallROSSize SnapshotRetentionTime SnmpTrapDestinationsList SnmpTrapEvents SnmpTrapsEnabled SortCheckOption SortOrderReportLevel SortWorkerThreads SSLCA SSLCertificate SSLPrivateKey StandardConformingStrings StrictUDxParameterChecking SyslogEnabled SyslogEvents SyslogFacility SystemMonitorInterval SystemMonitorThreshold TerraceRoutingFactor TextIndexComputeDeletedTokens TextIndexMaxTokenLength TopKHeapMaxMem TransactionIsolationLevel TransactionMode TrustConstraintsAsUnique UDxFencedBlockTimeout UDxFencedCancelTimeout UDxFencedExternalProcedureTimeout UseModularHashForReseg UseOnlyResilientRedistribution UseRecoveringNodesInVirtualTableQueries UseSafeDecompression UseV50IntegerDivision UseZygoteForExternalProcedures WarnOnIncompleteStartupPacket WithClauseMaterialization

References

1: https://my.vertica.com/docs/7.2.x/HTML/index.htm#Authoring/AdministratorsGuide/ConfiguringTheDB/ConfiguringTheDatabase.htm
2: https://my.vertica.com/docs/7.2.x/HTML/Content/Authoring/SQLReferenceManual/SystemTables/MONITOR/CONFIGURATION_PARAMETERS.htm
3: https://my.vertica.com/docs/7.2.x/HTML/index.htm#Authoring/AdministratorsGuide/ConfiguringTheDB/SettingandClearingConfigParameters.htm

Implement Galera cluster provisioning

Mon, 14 Dec 2015 00:00:00

High availability of database instances is required in order to build users confidence when running production workloads in Trove. Apart from replication, clustering ensures the deployment is highly available by allowing data to be accessed from any node in the cluster. It enhances system uptime, prevents from data loss and makes the architecture more scalable for future growth. Currently Trove has support for MongoDB clusters and Percona clusters. Trove clustering for the MariaDB datastore will be based on Galera Cluster, a synchronous multi-master database cluster.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/mariadb-clustering

Problem Description

MariaDB datastore does not have support for clustering. Adding a cluster implementation will allow users to create, read and write scalable and highly available deployments.

Proposed Change

Galera Cluster will leverage the base cluster implementation that currently exists in Trove by creating pluggable and configurable strategies.

Requirements:

An instance of Galera Cluster consists of a series of nodes, preferably three or more.
Initial cluster creation will start with 3 or more nodes.
The cluster inspection will return IPs of all its nodes in order to allow read and write to any node at any time.
The name of each instance will be automatically generated.
Clusters shall allow the cluster nodes to be resizable for volume and flavors. No validation will occur if a user resizes volumes or flavors of an instance.

The following will be changed:

Base classes for Percona Cluster and Galera Cluster will be added to common/strategies/cluster/experimental/galera_common/{api,taskmanager,guestagent}.py
Percona Cluster specialization will live under common/strategies/cluster/experimental/pxc/{api,taskmanger,guestagent}.py This contains the API, the taskmanager strategy to handle the cluster provisioning for Percona and the guestagent strategy to handle the cluster provisioning, configuration and execution.
Galera Cluster specialization will be added to common/strategies/cluster/experimental/mariadb/{api,taskmanager,guestagent}.py This contains the API, the taskmanager strategy to handle the cluster provisioning for Galera and the guestagent strategy to handle the cluster provisioning, configuration and execution.
Unit and integration tests

Configuration

A new datastore configuration section will be added. This initial implementation will add no Galera Cluster specific configuration options.

Database

This implementation will leverage the existing database schema for cluster.

Public API

The API request for creating/deleting cluster will remain unchanged. However, there will be subtle differences in the response. Galera clustering, as Percona implementation, will return all IPs of its nodes. This will be used for the user who wants to scale read/write.

Create Cluster

Request:

POST /v1.0/<tenant_id>/clusters
{
  "cluster": {
    "name": "products",
    "datastore": {
      "type": "mariadb",
      "version": "10"
    },
    "instances": [
      {
        "flavorRef": "2",
        "volume": {
          "size": 100
        },
      },
      {
        "flavorRef": "2",
        "volume": {
          "size": 100
        },
      },
      {
        "flavorRef": "2",
        "volume": {
          "size": 100
        },
      }
    ],
  }
}

Response:

{
  "cluster": {
    "id": "c33385b2-6c2a-491e-b44e-bcbb4af24136",
    "task": {
      "id": 2,
      "name": "BUILDING",
      "description": "Building the initial cluster."
    },
    "name": "products",
    "created": "2015-12-14T20:19:23",
    "updated": "2015-12-14T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mariadb",
      "version": "10"
    },
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "status": "BUILD",
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
        "status": "BUILD",
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
        "status": "BUILD",
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
      }
    ],
  }
}

Show Cluster

Request:

GET /v1.0/<tenant_id>/clusters/c33385b2-6c2a-491e-b44e-bcbb4af24136

Response:

{
  "cluster": {
    "id": "c33385b2-6c2a-491e-b44e-bcbb4af24136",
    "task": {
      "id": 1,
      "name": "NONE",
      "description": "No tasks for the cluster."
    },
    "name": "products",
    "created": "2015-12-14T20:19:23",
    "updated": "2015-12-14T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mariadb",
      "version": "10"
    },
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "status": "ACTIVE",
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
        "ip": "10.0.0.1"
      }
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
        "status": "ACTIVE",
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
        "ip": "10.0.0.2"
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
        "status": "BUILD",
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "volume": {
          "size": 100
        },
        "ip": "10.0.0.3"
      }
    ],
  }
}

Show Instance

Request:

GET /v1.0/<tenant_id>/clusters/c33385b2-6c2a-491e-b44e-bcbb4af24136/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1

Response:

{
  "instance": {
    "status": "ACTIVE",
    "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
    "cluster_id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products-1",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mariadb",
      "version": "10"
    },
    "ip": ["10.0.0.1"],
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 100,
      "used": 0.17
    },
  }
}

List Clusters

Request:

GET /v1.0/<tenant_id>/clusters

Response:

{
  "clusters": [
    {
      "id": "c33385b2-6c2a-491e-b44e-bcbb4af24136",
      "task": {
        "id": 1,
        "name": "NONE",
        "description": "No tasks for the cluster."
      },
      "name": "products",
      "created": "2014-04-25T20:19:23",
      "updated": "2014-04-25T20:19:23",
      "links": [{...}],
      "ip": ["10.0.0.1", "10.0.0.2", "10.0.0.3"],
      "datastore": {
        "type": "mariadb",
        "version": "10"
      },
      "instances": [
        {
          "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
          "status": "ACTIVE",
          "flavor": {
            "id": "7",
            "links": [{...}]
          },
          "volume": {
            "size": 100
          },
          "ip": "10.0.0.1",
        }
        {
          "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
          "status": "ACTIVE",
          "flavor": {
            "id": "7",
            "links": [{...}]
          },
          "volume": {
            "size": 100
          },
          "ip": "10.0.0.2",
        },
        {
          "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
          "status": "ACTIVE",
          "flavor": {
            "id": "7",
            "links": [{...}]
          },
          "volume": {
            "size": 100
          },
          "ip": "10.0.0.3",
        }
      ]
    },
    ...
  ]
}

Delete Cluster

Request:

DELETE /v1.0/<tenant_id>/clusters/c33385b2-6c2a-491e-b44e-bcbb4af24136

Response:

HTTP 202 (Empty Body)

Public API Security

None

Python API

None

CLI (python-troveclient)

The same CLI implemented for Percona Cluster will be used to interact with Galera Cluster. The main features are accessed as follows:

Create Cluster

$ trove help cluster-create

usage: trove cluster-create <name> <datastore> <datastore_version>
                            [--instance <instance>]

Creates a new cluster.

Positional arguments:
  <name>                Name of the cluster.
  <datastore>           A datastore name or UUID.
  <datastore_version>   A datastore version name or UUID.

Optional arguments:

  --instance <flavor_id=flavor_id,volume=volume>
                        Create an instance for the cluster. Specify
                        multiple times to create multiple instances.

Request:

$ trove cluster-create products mariadb "10" \
  --instance flavor_id=7,volume=2 \
  --instance flavor_id=7,volume=2 \
  --instance flavor_id=7,volume=2

Response:

+-------------------+--------------------------------------+
| Property          | Value                                |
+-------------------+--------------------------------------+
| created           | 2015-12-14T01:46:51                  |
| datastore         | mariadb                              |
| datastore_version | 10                                   |
| id                | aa6ef0f5-dbef-48cd-8952-573ad881e717 |
| name              | products                             |
| task_description  | Building the initial cluster.        |
| task_name         | BUILDING                             |
| updated           | 2015-12-14T01:46:51                  |
+-------------------+--------------------------------------+

The cluster-create command will create a cluster with identical instances. Trove will choose a instance to start with the –wsrep-new-cluster option, then the remaining instances will be started sequentially. The cluster will be marked ACTIVE when all instances have started and the database is available for use.

Show Cluster

$ trove help cluster-show

usage: trove cluster-show <cluster>

Shows details of a cluster.

Positional arguments:
  <cluster>  ID or name of the cluster.

Request:

$ trove cluster-show aa6ef0f5-dbef-48cd-8952-573ad881e717

Response:

+-------------------+--------------------------------------+
| Property          | Value                                |
+-------------------+--------------------------------------+
| created           | 2015-12-14T01:46:51                  |
| datastore         | mariadb                              |
| datastore_version | 10                                   |
| id                | aa6ef0f5-dbef-48cd-8952-573ad881e717 |
| ip                | 10.0.0.2, 10.0.0.1, 10.0.0.3         |
| name              | products                             |
| task_description  | No tasks for the cluster.            |
| task_name         | NONE                                 |
| updated           | 2015-12-14T01:59:33                  |
+-------------------+--------------------------------------+

Show Cluster Instance

$ trove help cluster-instances

usage: trove cluster-instances <cluster>

Lists all instances of a cluster.

Positional arguments:
  <cluster>  ID or name of the cluster.

Request:

$ trove cluster-instances aa6ef0f5-dbef-48cd-8952-573ad881e717

Response:

+-------------------------------------+----------------+-----------+------+
| ID                                  | Name           | Flavor ID | Size |
+-------------------------------------+----------------+-----------+------+
| 45532fc4-661c-4030-8ca4-18f02a2b337 | products-1     | 7         |    2 |
| 7458a98d-6f89-4dfd-bb61-5cf1d65c121 | products-2     | 8         |    2 |
| 1557208f-5c23-4537-a9f2-52a9db38d3a | products-3     | 7         |    2 |
+-------------------------------------+----------------+-----------+------+

List Clusters

$ trove help cluster-list

usage: trove cluster-list [--limit <limit>] [--marker <ID>]

Lists all the clusters.

Optional arguments:
  --limit <limit>  Limit the number of results displayed.
  --marker <ID>    Begin displaying the results for IDs greater than the
                   specified marker. When used with --limit, set this to
                   the last ID displayed in the previous run.

Request:

$ trove cluster-list

Response:

+--------+----------+-----------+-----------+----------+-----------+
| ID     | Name     | Datastore | DsVersion | IP       | Task Name |
+--------+----------+-----------+-----------+----------+-----------+
| uuid-1 | products | mariadb   | 10        | ip1      | NONE      |
| uuid-2 | items    | percona   | 5.5       | ip2, ip3 | BUILDING  |
+--------+----------+-----------+-----------+----------+-----------+

Delete Cluster

$ trove help cluster-delete

usage: trove cluster-delete <cluster>

Deletes a cluster.

Positional arguments:
  <cluster>  ID of the cluster.

Request:

$ trove cluster-delete aa6ef0f5-dbef-48cd-8952-573ad881e717

Response:

(None)

Internal API

None

Guest Agent

None

Alternatives

None defined yet.

Dashboard Impact (UX)

The datastore must be enabled as a clustering datastore.

Implementation

Assignee(s)

Primary assignee:

vkmc
tellesnobrega

Milestones

Target Milestone for completion:

Mitaka-3

Work Items

Trove Integration change to add support for mariadb-galera-server
API strategy implementation
Task Manager strategy implementation
Guest Agent strategy implementation
Unit and integration tests

Upgrade Implications

To enable MariaDB cluster support, a different package needs to be installed on the guest instance rather than the default. This can be solved by updating the packages installed on the datastore version, although it is recommended that a new image with the desired package is built and loaded.

Dependencies

None

Testing

Unit tests will be added to cover non-trivial code paths.
Integration tests will be added to test end-to-end cluster features.

Documentation Impact

The response API will look different due to the nature of Galera clusters. Precisely, the view will contain all the IPs of the instances.

References

Besides the existing API, there will be new API to grow/shrink cluster handled in a follow up spec.

Trove does not monitor the status of the cluster instances. If an instance in the cluster becames detached from the PRIMARY instance, it will responsability of the user to detect the situation and take a corrective action. If the instance does not re-attach to the cluster when a Incremental State Transfer (IST) or State Snapshot Transfer (SST) is possible, the user would need to delete the instance and add a new node to the cluster. This will be later substituted by the grow/shrink cluster feature.

In the event of a Nova compute reboot, the nodes will automatically restart, and the cluster will automatically recover provided that the nodes can determine that the PRIMARY component can be properly recovered. In the case that the PRIMARY component cannot be recovered, operator intervention would be required to manually restart the cluster.

Appendix

None

CouchDB Database & User Functions

Wed, 25 Nov 2015 00:00:00

Even though CouchDB has been added as an experimental datastore in Trove, the functionality yet has some gaps to be filled. The motivation behind this spec is to implement some of those missing user and database functions for the CouchDB datastore in Trove.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/couchdb-database-user-functions

Problem Description

Currently, a CouchDB instance can be created using Trove but the functionality is very limited. No user or database actions can be performed via the Trove CLI. While, it is possible to perform such operations by accessing the CouchDB administration interface called Futon 1, it is not within the scope of Trove. Therefore for ease of use and consistency, it is best that these database and user functionalities for CouchDB be implemented as part of Trove.

Proposed Change

Implementation of the basic user and database functions will be performed for the CouchDB datastore in Trove using an HTTP-based REST API which CouchDB uses to communicate with the server. Curl commands will be sent over HTTP requests to the CouchDB server which will then execute actions.

The new operations that will be supported are as follows:

create/delete/show user
list users
grant/revoke/show access
root enable/show
create/delete database
list databases

A CouchDBAdmin class will be created in service.py which will have these methods in it and the functions in manager.py will be updated to call these new methods. Each method will use the core CouchDB API to make appropriate HTTP requests for that functionality on the server.

As of now an API extension for CouchDB does not exist so a new API extension will be created to implement datastore specifics while moderating user/ database operations. The validation in effect in the CouchDB specific models will correspond to CouchDB datastore requirements.

CouchDB Security

On a fresh CouchDB instance, the server allows any request to be made by anyone - it’s an Admin Party 2 . To enable the user functionalities, the guest agent’s CouchDB server will be made secure by creating a Trove user ‘os_admin’ by default. When a guest is first started, the os_admin user will be created and will be connected to the server using a password generated by the guest. This will be done using the following command- curl -X PUT http://localhost:5984/_config/admins/os_admin -d '"password"'

The username and generated password will be stored in a file in the home ~ directory on the instance (stream_codecs.JsonCodec will be used to handle it). All other users that are created are non-admin users so this preventing anybody from accidentally creating an admin user 3 . The only admin user that can be created is ‘root’ using the trove root-enable command by creating a user on the couchDB server with admin privileges.

Configuration

None

Database

None

Public API

The user will be able to run the implemented functions for CouchDB datastore via CLI.

Public API Security

None

Python API

None

CLI (python-troveclient)

The following Trove command line commands will be functional for CouchDB:

trove user-create
trove user-delete
trove user-grant-access
trove user-list
trove user-revoke-access
trove user-show
trove user-show-access
trove root-enable
trove root-show
trove database-create
trove database-delete
trove database-list

Internal API

None

Guest Agent

The CouchDB guest agent will be modified to support additional database and user functionality. In particular the following files will have added components:

trove/guestagent/datastore/experimental/couchdb/manager.py
trove/guestagent/datastore/experimental/couchdb/service.py
trove/guestagent/datastore/experimental/couchdb/system.py

Several new guest agent functions will be implemented using the CouchDB API. Below are sample API calls and associated details for each guest agent method.

create_user

Create non-admin user with name username and password password

curl -X PUT http://os_admin:password@localhost:5984/_users/org.couchdb
.user:username \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-d '{"name": "username", "password": "password", "roles": [], "type":
"user"}'

list_users

List all the users from the system database _users and all the databases from the system database _all_dbs. Then cross reference to list users and the databases that they have access to.

curl -s http://os_admin:password@localhost:5984/_users/_all_docs

curl -X http://os_admin:password@localhost:5984/_all_dbs

curl -X http://os_admin:password@localhost:5984/databasename/_security

delete_user

Deletes user username from the system database _users corresponding to rev number 1-3cd11775d7e3ba15a9f8c553cb3d47bd. The rev number for the document to be obtained using the second command listed below.

curl -X DELETE http://os_admin:password@localhost:5984/_users/org. couchdb.user:username?rev=1-3cd11775d7e3ba15a9f8c553cb3d47bd

curl -s http://os_admin:password@localhost:5984/_users/_all_docs

get_user

Shows the complete information associated with a specific user - username, databases user has access to and permissions(admin/member).

curl -X http://os_admin:password@localhost:5984/_all_dbs

curl -X http://os_admin:password@localhost:5984/databasename/_security

enable_root

Create user “root” and grant the role “admin”

curl -X PUT http://os_admin:password@localhost:5984/_config/admins/
root -d '"password"'

is_root_enabled

Checks if user root exists in the system database _config and has admin privileges

curl -s http://os_admin:password@localhost:5984/_config/_admins

delete_root

Deletes the root user from the CouchDB instance.

curl -X DELETE http://
os_admin:password@localhost:5984/_config/admins/root

grant_access

Modify the role of user username for database databasename to include username as a listed admin

curl -X PUT http://os_admin:password@localhost:5984
/databasename/_security \
> -d '{"admins":{"names":[], "roles":[]}, "members":{"names”:[
“username”],”roles":[]}}'

revoke_access

Similar to the implementation of grant access, modify the role of user ` username`` for database databasename to remove username as a listed member

curl -X PUT http://os_admin:password@localhost:5984
/databasename/_security \
> -d '{"admins":{"names":[], "roles":[]}, "members":{"names”:[],
”roles":[]}}'

list_access

Lists all the databases from the system database _all_dbs. Then cross reference to list all the databases that the specified user has has access to.

curl -X http://os_admin:password@localhost:5984/_all_dbs

curl -X http://os_admin:password@localhost:5984/databasename/_security

create_database

Creates a database with name database_name

curl -X PUT http://os_admin:password@localhost:5984/database_name

The database name must consist of one or more of the following characters and the name must begin with a lowercase letter 4 .

Lowercase characters (a-z)

Digits (0-9)

Any of the characters _, $, (, ), +, -, and /.

delete_database

Deletes the database with name database_name

curl -X DELETE http://os_admin:password@localhost:5984/database_name

list_databases

Lists all the databases from the system database _all_dbs

curl -X GET http://os_admin:password@localhost:5984/_all_dbs

Alternatives

None

Dashboard Impact (UX)

The dashboard will need to be updated for the users and databases tabs to be enabled for the CouchDB datastore. No new features will need to be added to the dashboard.

Implementation

Assignee(s)

Primary assignee:: imandhan (launchpad id) Ishita Mandhan <imandha@us.ibm.com>

Milestones

Mitaka

Work Items

Implementation of the functionality of user and database functions for CouchDB. Involves working on the manager, service and system files primarily. Tests will be written as required - both unit tests and int tests. The work will be split into 5 parts-

Enable authentication on server
Create/delete/get/list user
Enable/check root
grant/revoke/list access
create/delete/list database

Upgrade Implications

None

Dependencies

None

Testing

Unit tests and integration tests will be added as necessary to test new code added.

Documentation Impact

The CouchDB Trove documentation will need to be updated to indicate that user and database functions have been implemented.

References

1: CouchDB Futon: http://docs.couchdb.org/en/1.6.1/intro/futon.html
2: CouchDB Security: http://docs.couchdb.org/en/1.6.1/intro/security.html
3: CouchDB Authentication: http://docs.couchdb.org/en/1.6.1/config/auth.html
4: CouchDB Database Name Restrictions: http://couchdb-13.readthedocs.org/en/latest/api/database/

Appendix

None

Example Snippet Generator

Thu, 24 Sep 2015 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/trove/+spec/example-snippet-generator

Problem Description

Trove has ssome really nice documentation featuring snippets showing the what the JSON looks like for the standard REST calls and response. The only issue is these docs are not validated, meaning if the API changes in ways big or small its possible they will not match what a present day user of Trove will see.

Proposed Change

This blueprint proposes we fix this by automatically generating these examples and validating them in Trove.

The fix will involve actually making calls against the real Trove code and capturing the bodies of the requests and responses then writing them to the text files used by the docs. We can do this quickly by utilizing the same “fake mode” test doubles that are used by the tests already run in Tox. The beauty is all of the API code that determines what the request and responses look like will get run just the same as if the tests had executed against a fully stood up Trove environment, with the advantage that certain UUIDs can be altered to avoid them changing with every test run.

Configuration

None

Database

None

Public API

None

Public API Security

None

Internal API

None

Guest Agent

None

Data Model Impact

None.

REST API Impact

None, except that it will be tested even better than before.

Security Impact

None.

Notifications Impact

None.

Other End User Impact

None.

Performance Impact

The snippet generation will run in mere seconds as part of Tox. There will be no noticeable impact on the developers of Trove.

Other Deployer Impact

None.

Developer Impact

The generation will run fast enough to be invisible to most developers. However developers will be aware if they inadvertently change anything in the request or response payload and will have to argue for the changes to the API, even if they’re merely additions.

Community Impact

The generation of snippets will usher in a new golden era of devs and doc writers working together. Maybe sometimes devs will be doc writers, or doc writers will be devs. It isn’t a stretch to suggest it will be a utopia.

Alternatives

We could continue the implied agreement that devs are always constantly reading the docs and treating that as a contract, and that with each change they rabidly run integration tests but also manually run tests and inspect the output, making sure the snippets shown in the docs are accurate. Unfortunately since many of the current snippets are inaccurate I don’t think this will work.

Implementation

Assignee(s)

Primary assignee:: Tim Simpson

Milestones

Target Milestone for completion:: Kilo-1

Work Items

Implement the snippets generator.

Dependencies

None.

Testing

Tempest Tests

Functional Tests

API Tests

Documentation Impact

We will need to change the snippets at least initially as they have changed so much since when they were originally authored.

User Documentation

None.

Developer Documentation

We will need to document how this works in the Tox file.

References

Datastore maturity, experimental datastores

Thu, 24 Sep 2015 00:00:00

https://blueprints.launchpad.net/trove/+spec/experimental-datastores

Hitherto, Trove has not made a clear distinction of which datastores are considered to be “experimental” and which are considered to be “suitable for general use”. At the mid-cycle in Seattle (Kilo release) we discussed the possibility of introducing a clear definition of what an “experimental” datastore would be and how one could migrate a datastore from “experimental” to “suitable for general use”.

Problem description

When code for a datastore is submitted to Trove, there is no clear indication whether that datastore is in a state that would be sufficient for general use. For example, while a single instance datastore of MySQL is perfectly usable for some development and test use cases but a single instance datastore of (say) Cassandra is much less useful. A datastore that supports backup and restore is more useful than one that does not.

As a community we wish to be welcoming of changes but no all change sets are made equal. Some include extensive capabilities and test coverage while others are more basic. In an effort to make it clear what we, as a community, believe is the “readiness” of a datastore, we believe that a classification of datastores is a good improvement.

Proposed change

Introduce a classification of datastores

It is proposed that we create three groups of datastores as described in detail below. This classification will mean that any datastore could be considered to be either (a) “experimental”, (b) “technical preview”, or (c) “stable”.

In each release, beginning with the Kilo release, each datastore will be given a classification that will be one of the values listed above based strictly on its adherence to the requirements for each of those classifications.

The classifications will be inclusive; i.e. a datastore that meets the “technical preview” requirements must also meet all the requirements for “experimental”, and a datastore that meets the “stable” requirements must also meet all the requirements for “technical preview”.

Introduce a classification of strategies

In a similar vein, we will also assign a classification to each strategy. Some strategies may be more throughly vetted and tested (for example) than others. While (purely for purposes of illustration) mysql_binlog may be considered “stable”, one could have a different replication strategy that was considered “experimental”.

Configuration

At the very least, this change will impact the trove-guestagent.conf file as it lists the datastore registry by providing the path to the implementation (manager.py)

Any similar location either in a configuraiton file (or for that matter, in the code) that calls out a location of an implementation will have to change.

Database

This should have no database changes.

Public API

This change has no impact on the Public API.

Public API Security

Not applicable.

Internal API

No changes to the internal API.

Guest Agent

It does not change the behavior of the guest agent, merely the location of the code.

Initial Classifications

As part of this change, it is proposed that the following assignments be made.

Stable: MySQL Technical Preview: Cassandra and MongoDB Experimental: All others

Requirements

Experimental

A datastore will be considered for merging in the experimental stage if it includes the following items.
- implementation of a basic subset of the trove API including create and delete
- guest agent “elements” that will allow a user to create a guest agent
- a definition of supported operating systems
- basic unit tests that will verify the operation of the guest agent and test suite that is non-voting
A strategy will be considered experimental if an implementation is provided and includes basic unit tests to verify operation of the strategy. A passing, and non-voting test suite should also be provided.
Technical Preview

A datastore will be considered for “Technical Preview” if it meets the requirements of “Experimental” and further
- implements API’s required to implement the capabilities of the datastore as defined in the datastore compatibility matrix that is at [https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix]
Note that it is not required that the datastore implement all features (resize, backup, replication, clustering, …) to meet this classification.

It is also required that non-voting gate tests for all capabilities and a mechanism to build a guest image that will allow a user to exercise these capabilities is provided.

A strategy will not (normally) be considered for Technical Preview classification.
Stable

A datastore will be considered “Stable” if it meets the requirements of “Technical Preview” and a stable voting test suite is part of the gate.

A strategy will be considered “Stable” if it meets the requirements of “Experimental” and also has stable voting tests as part of the gate.

Details of Change

The plan is to edit all files required to reflect the change in location of some datastores and strategies.

Alternatives

The only other alternative is to have a webpage that lists this information without actually reorganizing the code.

Implementation

Assignee(s)

Primary assignee:: amrith slicknik

Milestones

kilo-3

Work Items

Determine which datastores are in each of the proposed classifications.
Determine which strategies are in each of the proposed classifications
Implement the change to relocate code into the appropriate directory structure
Implement changes in trove-integration to match this (redstack) if appropriate

Dependencies

None

Testing

Each datastore will have to be launched and verified for proper operation.

Documentation Impact

This will have a documentation impact and a bug will be opened for this.

References

This was discussed at the mid-cycle. The etherpad is [https://etherpad.openstack.org/p/trove-kilo-sprint-state-of-ci]

MongoDB Backup and Restore

Thu, 24 Sep 2015 00:00:00

Despite the previous efforts, Trove still does not provide backup or restore functionality for MongoDB. This blueprint aims to support backup and restore for single instances.

https://blueprints.launchpad.net/trove/+spec/backups-single-instance-mongodb

There is also a previous blueprint that incorporated single-instance backup 7.

Problem Description

Backup and restore capability is important for eventually moving the MongoDB datastore out of the “experimental” stage. MongoDB provides two utilities, mongodump and mongorestore, which work with BSON-based files: efficient, binary equivalents of the JSON data that MongoDB manages. These utilities will be used to provide a standard backup and restore strategy for the MongoDB datastore, equivalent to the MySQLDump strategy for MySQL-based datastores.

Proposed Change

An initial simple backup and restore strategy for MongoDB will be implemented using the default parameters to the mongodump and mongorestore utilities. For compatibility, it will avoid the new features provided in 3.0 (backups with oplogs) and avoid the options no longer supported in 3.0 (backups taken with direct filesystem access to the database).

Configuration

The default values for backup_strategy, backup_namespace and restore_namespace for MongoDB will change to point to the appropriate locations.

Database

None.

Public API

There are no changes to the existing API, but this blueprint enables the use of the backup-create, backup-delete and backup-list CLI commands for MongoDB datastores.

Public API Security

None.

Python API

None (empty section added after merging)

CLI (python-troveclient)

None (empty section added after merging)

Internal API

None.

Guest Agent

The MongoDB guest agent will be modified to support the new backup and restore strategies.

It is currently unclear whether the MongoDB backup and restore tools directly support streaming. There is an open JIRA ticket 5, and some unofficial third-party tools exist to support streaming 6. This seems to contradict the documentation 8 which suggests streaming is supported. If streaming can be implemented and supported it will be used, otherwise backup and restore will fall back to a two-stage process.

The backup procedure will use the mongodump command to connect directly to the running MongoDB server and dump the contents of the database onto local storage, or compressed and streamed on the fly, if supported. The backup will be streamed to the container configured in the backup_swift_container configuration option.

The restore procedure works in reverse, streaming and uncompressing the backup from object storage back onto the guest, then using the mongorestore command to restore the database, or streamed and uncompressed in-place, if the tools support it.

The _perform_restore and create_backup methods are implemented in: trove/guestagent/datastore/experimental/mongodb/manager.py

The backup and restore implementation will go in their respective strategy folders: trove/guestagent/strategies/backup/experimental/mongo_impl.py trove/guestagent/strategies/restore/experimental/mongo_impl.py

Alternatives

As of version 3.0, MongoDB supports consistent, point-in-time snapshots with the use of –oplog to the mongodump command for single-server deployments. Operations committed after the start of the mongodump command are logged to a separate file, and this file can be replayed with the –oplogReplay parameter to mongorestore. As previous versions of MongoDB, and sharded installations do not support this, it has been left as future work: a subclassed backup/restore strategy could be easily implemented for MongoDB 3.0 guests.

In addition, the MongoDB documentation recommends filesystem snapshots as the preferred way to do backups 1. This pushes the problem down to the storage layer, and so is currently not implemented as a backup/restore strategy for any datastores in Trove. A spec was previously proposed 2 3, and discussed on the mailing list 4 but has yet to be implemented.

Implementation

Assignee(s)

Primary assignee: atomic77

Building on previous efforts of:

Ramashri Umale <rumale@ebaysf.com>, Viswa Vutharkar <vpvutharkar@ebaysf.com>

Milestones

Target Milestone for completion:

liberty-1

Work Items

rebase of last patchset uploaded on Oct 14, 2014 to current upstream trove
change backup/restore to operate on a running MongoDB, as MongoDB 3.0 deprecates –journal and –dbpath for mongodump
review existing test cases for gaps, implement as necessary

Upgrade Implications

None.

Dependencies

None.

Testing

Test cases used for testing backup/restore for MySQL will be adapted to run against MongoDB.

Documentation Impact

The documentation should reflect that MongoDB backup/restore is supported for single instances.

References

1: http://docs.mongodb.org/manual/tutorial/backup-with-filesystem-snapshots/
2: https://blueprints.launchpad.net/trove/+spec/volume-snapshot
3: https://wiki.openstack.org/wiki/Trove/volume-data-snapshot-design
4: http://lists.openstack.org/pipermail/openstack-dev/2014-April/032673.html
5: https://jira.mongodb.org/browse/TOOLS-23
6: https://github.com/timisbusy/dumpstr
7: https://blueprints.launchpad.net/trove/+spec/single-instance-mongodb-ga
8: http://docs.mongodb.org/manual/reference/program/mongodump/#bin.mongodump

MariaDB Datastore Support

Mon, 10 Aug 2015 00:00:00

MariaDB 1 is a community-developed fork of MySQL licensed under GNU GPL.

It aims to maintain high compatibility with MySQL, ensuring a “drop-in” replacement capability with library binary equivalency and exact matching with MySQL APIs and commands.

https://blueprints.launchpad.net/trove/+spec/mariadb-datastore-support

Problem Description

MariaDB versioning follow the MySQL’s versioning scheme up to version 5.5. Thus, MariaDB 5.5 offers all of the MySQL 5.5 features. After the 5.5 version, MariaDB developers decided to start a branch numbered 10, as an attempt to make it clear that specific new features have been developed in MariaDB 10 that are not included in MySQL 5.6.

Currently, Trove has support for MySQL but not for MariaDB in particular. This could be easily fixed for MariaDB 5.5 by just adapting the main configuration file (my.cnf). But in the case of MariaDB 10, this will no-longer be so straightforward.

With this in mind, and the fact that several GNU/Linux distributions are adopting MariaDB as the default MySQL implementation, it seems important to add support for this datastore.

Proposed Change

To add support for this datastore, we will need the following:

Implement instance management for the datastore:

Launch new instance

Terminate instance

Reboot instance

Resize instance

Implement user and database management for the datastore:

CRUD users

CRUD databases

Add new elements in trove-integration project to enable the creation of Ubuntu/Fedora images with MariaDB

In this first approach, it makes sense to take profit of the MySQL Refactor spec 2 and inherit as much as possible from the MySQL implementation, providing a cleaner way to launch MariaDB 5.5 instances.

In a follow up spec, differences in implementation will be tackled (e.g. a GTID based replication strategy 3 for MariaDB 10 will be proposed) and we would be able to provide support for MariaDB newer versions.

Configuration

A new configuration group for MariaDB and the different configuration options specific to MariaDB have to be added to trove/common/cfg.py

Some of the examples for the configuration options are:

tcp_ports

udp_ports

backup_strategy

backup_incremental_strategy

replication_strategy

mount_point

volume_support

device_path

backup_namespace

restore_namespace

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

None

Guest Agent

This requires implementing the various datastore features for MariaDB like Launch, Reboot, Terminate and Resize.
This requires implementing the various CRUD features for users and databases for MariaDB.
This will include adding the following files specific to MariaDB under the guestagent/datastore module:

manager.py

service.py

These changes will not affect the behavior of the guestagent or its interaction with other components.

Alternatives

Do not support MariaDB.

Implementation

Assignee(s)

Primary assignee:

vkmc

Milestones

Liberty-3

Work Items

Update trove-integration to support MariaDB image creation
- Create MariaDB elements for Ubuntu/Fedora
Update trove/common/cfg.py with the configuration options for MariaDB
Add MariaDB configuration templates under templates/mariadb
Add support for instance management features in guestagent/experimental/mariadb/service.py [*]
Add support for user and databases management features in guestagent/experimental/mariadb/service.py [*]
Add unit and integration tests specific to MariaDB

[*] This items rely on the MySQL Manager Refactor spec 2 and in the first iteration it will only require inheriting from the MySQL existing implementation.

Upgrade Implications

None

Dependencies

MySQL Manager Refactor 2

Testing

Unit tests will be added for MariaDB guestagent
Integration tests will be added for end-to-end feature testing

Create/Delete MariaDB instances

Resize MariaDB instances

Documentation Impact

Docs will be updated to indicate that:

Which MariaDB version is supported

Trove capabilities for MariaDB datastore

Instructions to build guest MariaDB images using trove-integration elements

References

1: https://mariadb.org/
2(1,2,3): http://specs.openstack.org/openstack/trove-specs/specs/liberty/mysql-manager-refactor.html
3: https://mariadb.com/kb/en/mariadb/global-transaction-id/

MySQL Manager Refactor

Wed, 29 Jul 2015 00:00:00

There are a number of forks of MySQL that differ to varying degrees. This blueprint proposes the creation of a class structure for MySQL-derived datastores to avoid duplication of code for features and capabilities shared in common.

It is expected that the lessons learned from this effort will be applicable to future efforts to provide differentiation for systems such as MongoDB and PostgreSQL.

https://blueprints.launchpad.net/trove/+spec/mysql-manager-refactor

Problem Description

In recent years, there have been a number forks of MySQL with varying levels of divergence from MySQL Community Edition (CE). Commonly used variants of MySQL include Percona and MariaDB. Specialized versions specific to a variant also exist, such as Galera for MariaDB or MySQL Cluster (NDB). Despite the differences, there is a great deal shared among these variants. As such, wholly separate datastore implementations of each would result in duplicated code, leading to maintainability difficulties and potential confusion for operators expecting common functionality to be treated the same across systems.

Openstack Trove would benefit from the refactoring of the MySQL manager with more consistent support for MySQL-like systems, simplified code maintenance as the variants evolve and the ability to provide differentiating features. New variants could also be introduced with relative ease.

Proposed Change

The existing MySQL datastore code is found in trove/guestagent/datastore/mysql. This datastore already effectively serves as a sort of superclass, with both the Percona and MySQL datastores pointing to the same management code. MariaDB is supported if the underlying MariaDB instance is treated as if it were MySQL.

The crux of this blueprint is to move the majority of the existing MySQL manager code into a new set of abstract classes, with stub subclasses for MySQL, Percona and MariaDB datastores inheriting from them.

Maturity-Aware Reorganization

The base mysql code will reside in the current mysql datastore directory. This would result in

The creation of new base implementations for the manager and service modules in the trove/guestagent/datastore/mysql directory
The creation of explicit datastores for Percona and MariaDB with implementation classes that inherit from base MySQL.

The resulting file and directory structure would change from:

trove/guestagent/datastore/mysql/manager.py
trove/guestagent/datastore/mysql/service.py

to:

trove/guestagent/datastore/mysql/manager_base.py
trove/guestagent/datastore/mysql/service_base.py
trove/guestagent/datastore/mysql/manager.py
trove/guestagent/datastore/mysql/service.py
trove/guestagent/datastore/experimental/mariadb/manager.py
trove/guestagent/datastore/experimental/mariadb/service.py
trove/guestagent/datastore/experimental/percona/manager.py
trove/guestagent/datastore/experimental/percona/service.py

This approach has the benefit of less potential confusion about the maturity level of the base code. However, it is not as not as clean an organization: MySQL CE is treated as both a base system and an implementing datastore.

A pass through the MySQL manager code will be done to identify methods that should be made abstract. These methods would then be brought “down” into the subclasses.

This blueprint does not attempt to create optimized MariaDB or Percona datastores, but rather to lay the groundwork for their creation.

Service class dependency injection

The service.py module contains a number of classes, such as MySqlAppStatus, that are used in the manager.py module. These are currently tightly coupled: the mysql manager has an explicit import for each.

The make it possible to extend these classes arbitrarily for different datastores, and eliminate the tight-coupling, the old references will be refactored into class properties, which are to be injected by the concrete manager class.

Strategy consolidation

Currently not included in the scope of this refactor, but an important future consideration, are the associated strategies that can also have differences across variants. For example, it may make sense to bring some or all of the logic from the replication strategy into the datastore subtree to provide differentiation.

Configuration

Guest agent configuration options that point to fully qualified classnames, e.g:

datastore_registry_ext =
  mysql:trove.guestagent.datastore.mysql.manager.Manager,
  percona:trove.guestagent.datastore.mysql.manager.Manager

will need to point to the new class names, e.g:

datastore_registry_ext =
  mysql:trove.guestagent.datastore.mysql.manager.Manager,
  percona:trove.guestagent.datastore.percona.manager.Manager,
  mariadb:trove.guestagent.datastore.experimental.mariadb.manager.Manager

Database

Nothing expected, but confirm.

Python API

None.

CLI (python-troveclient)

None.

Public API

None.

Public API Security

None.

Internal API

None.

Guest Agent

Behaviour should remain the same, but location of the code would change.

Alternatives

Two alternatives are discussed in the main Proposed Change section.

Implementation

Assignee(s)

Primary assignee:

Launchpad/IRC: atomic77

Email: atomic@tesora.com

Milestones

Target Milestone for completion:

liberty-1

Work Items

Reorganize code
Create stub implementations of Percona and MariaDB datastores that inherit from base MySQL classes.
Review MySQL datastore implementation for initial candidates for abstract methods. Bring down and reimplement in each of the three datastore implementations.
Write additional integration tests

Upgrade Implications

As with any change to the layout of the source tree, care must be taken by the operator to ensure that the updating of the code on the guest agent coincides with the updating of configuration files. This would only be an issue for operators that eventually want to leverage the new optimized managers for Percona, MariaDB, etc. as the location of the MySQL CE manager would remain backwards-compatible.

Dependencies

None.

Testing

Additional tests should be added to ensure that subclassing is working correctly e.g. ensure that some Percona-specific code is not running against MySQL datastores, etc.

Documentation Impact

The documentation should be updated to inform operators of the new locations of datastore implementations that could be added to the guestagent configuration file.

References

A related blueprint is experimental-datastores [1] as this impacts the organization of datastore implementations into directories based on maturity level.

[1] https://blueprints.launchpad.net/trove/+spec/experimental-datastores

Cluster Scaling

Mon, 15 Jun 2015 00:00:00

Cluster support in trove currently has no direct support for growing or shrinking clusters. The current MongoDB cluster implementation does currently support adding a “shard” via the “add-shard” action, but it has been decided that this should be changed to use the new cluster-grow functionality outlined in this spec.

The cluster-grow and cluster-shrink functionality has been discussed at the previous mid-cycle and summit, and it was accepted as a preferable alternative to the add-shard functionality currently implemented by the MongoDB datastore. This specification is intended to develop a foundation for the shrink/grow functionality in Percona and Redis clustering, and to be backward compatible with the MongoDB clustering implementation.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/cluster-scaling

Problem Description

Trove cluster support needs to be extended with support for increasing or decreasing the size of a cluster. This support will be provided by adding new ReST APIs for cluster-grow and cluster-shrink with corresponding CLI and python-troveclient support.

The MongoDB cluster implementation will be enhanced to support cluster-grow in addition to the currently supported “add-shard” action.

Proposed Change

New ReST APIs

Two new ReST APIs will be added for this feature.

cluster-grow

Adds nodes to a cluster.

This new API adds 3 new attributes to the usual instance payload:

name: Name to assign to the instance
instance_type: Datastore specific type of the instance (eg., query/replica for Mongo, master/slave for Redis).
related_to: Defines a relationship from one member to another. The value is datastore specific, but will likely refer to the name of a previously defined instance in the same ReST payload.

Request:

POST /v1.0/<tenant_id>/clusters/<cluster-id>/action
{
    "grow": [
      {
        "name": "redis-clstr-member-5",
        "instance_type": "master",
        "flavorRef": "2",
        "volume": {
          "size": 2
        },
      },
      {
        "name": "redis-clstr-member-6",
        "instance_type": "slave",
        "related_to": "redis-clstr-member-5",
        "flavorRef": "2",
        "volume": {
          "size": 2
        },
      }
    ]
}

Response:

{
  "cluster": {
    "id": "edaac9ca-b5e1-4028-adb7-fa7653e11224",
    "task": {
      "id": 2,
      "name": "BUILDING",
      "description": "Building the initial cluster."
    },
    "name": "redis-clstr",
    "created": "2015-01-29T20:19:23",
    "updated": "2015-01-29T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "redis",
      "version": "3.0"
    },
    "ip": [],
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "name": "redis-clstr-member-5",
        "instance_type": "master",
        "status": "BUILD",
        "ip": [],
        "links": [{...}],
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        }
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
        "name": "redis-clstr-member-6",
        "instance_type": "slave",
        "related_to": "redis-clstr-member-5",
        "status": "BUILD",
        "ip": [],
        "links": [{...}],
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        }
      },
    ]
  }
}

HTTP Codes:

- Accepted.
- BadRequest. Server could not understand request.
- Forbidden. Local storage not specified in flavor ID: <ID>.
- Forbidden. A flavor is required for each instance in the cluster.
- Not Found. Flavor <id> not found.
- Not Found. Cluster <id> not found.

cluster-shrink

Deletes the specified instances from the cluster.

Request:

POST /v1.0/<tenant_id>/clusters/<cluster_id>/action
    "shrink": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
      }
    ]
}

Response:

N/A

HTTP codes:

- Accepted.
- Forbidden. Instance <instance_id> not in state to be removed.
- Instance <instance_id> not found in cluster <id>.

Configuration

n/a

Database

n/a

Public API

This change adds new CLI commands cluster-grow and cluster-shrink, detailed below. These CLI commands and their corresponding APIs should be consistent with existing CLI and API usage.

Public API Security

No security impacts are expected.

Python API

The following methods will be added to the Clusters class in the python-troveclient:

class Clusters:

  def grow(self, cluster, instances)
      """Grow the specified cluster.

      :param cluster: The cluster to grow
      :param instances: JSON describing instances to add
      """

  def shrink(self, cluster, instances)
      """Shrink the specified cluster.

      :param cluster: The cluster to shrink
      :param instances: JSON describing instances to remove
      """

CLI (python-troveclient)

cluster-grow

Basic Grow

Basic Grow would be suitable to databases such as Galera or Cassandra where all nodes are effectively peers - no special options are required.

$ trove cluster-grow mycluster --instance flavor=7
<example output to follow>

--instance parameter may be specified multiple times.

Generated Request:

POST /v1.0/<tenant_id>/clusters/<cluster-id>/action
{
    "grow": [
      {
        "flavorRef": "7",
      }
    ]
}

Add Master and Slave

Master and Slave grow would be suitable for databases such as Redis which support master/slave replication within a cluster.

$ trove cluster-grow mycluster \
  --instance name=newnode1,type=master,flavor=7 \
  --instance name=newnode2,type=slave,related_to=newnode1,flavor=7

Generated Request:

POST /v1.0/<tenant_id>/clusters/<cluster-id>/action
{
    "grow": [
      {
        "name": "newnode1",
        "instance_type": "master",
        "flavorRef": "7"
      },
      {
        "name": "newnode2",
        "instance_type": "slave",
        "related_to": "newnode1",
        "flavorRef": "7 "
      }
    ]
}

To add a standby node to a node named “node1” in a database such as Vertica or Redis, a simpler form of the above could be used.

$ trove cluster-grow mycluster \
  --instance type=standby,related_to=node1,flavor=7

Generated Request:

POST /v1.0/<tenant_id>/clusters/<cluster-id>/action
{
    "grow": [
      {
        "instance_type": "standby",
        "related_to": "node1",
        "flavorRef": "7 "
      }
    ]
}

Add Replica Set

This form of Grow Cluster would be used for databases such as MongoDB which implement growth by shards, in this case called “replica sets”. The example below creates a new query server plus two replica sets of 3 nodes each.

$ trove cluster-grow mycluster \
  --instance name=rs2q,type=query,flavor=7 \
  --instance name=rs2a,type=replica,flavor=7 \
  --instance name=rs2b,type=replica,related_to=rs2a,flavor=7 \
  --instance name=rs2c,type=replica,related_to=rs2b
  --instance name=rs3a,type=replica,flavor=7 \
  --instance name=rs3b,type=replica,related_to=rs3a,flavor=7 \
  --instance name=rs3c,type=replica,related_to=rs3b

Generated Request:

POST /v1.0/<tenant_id>/clusters/<cluster-id>/action
{
    "grow": [
      {
        "name": "rs2q",
        "instance_type": "query",
        "flavorRef": "7"
      },
      {
        "name": "rs2a",
        "instance_type": "replica",
        "flavorRef": "7"
      },
      {
        "name": "rs2b",
        "instance_type": "replica",
        "related_to": "rs2a",
        "flavorRef": "7 "
      },
      {
        "name": "rs2c",
        "instance_type": "replica",
        "related_to": "rs2b",
        "flavorRef": "7 "
      },
      {
        "name": "rs3a",
        "instance_type": "replica",
        "flavorRef": "7"
      },
      {
        "name": "rs3b",
        "instance_type": "replica",
        "related_to": "rs3a",
        "flavorRef": "7 "
      },
      {
        "name": "rs3c",
        "instance_type": "replica",
        "related_to": "rs3b",
        "flavorRef": "7 "
      }
    ]
}

cluster-shrink

$ trove cluster-shrink mycluster nodename-1 nodename-2
<example output to follow>

Internal API

Appropriate methods will be added to the base Cluster Strategy.

Guest Agent

Appropriate methods will be added to the base Cluster Strategy.

The MongoDB Implementation will be adapted to support both the existing add-shard functionality, plus the new cluster-grow functionality.

Alternatives

n/a

Implementation

Assignee(s)

Primary assignee:: Matthew Van Dijk Morgan Jones

Milestones

Target Milestone for completion:: Liberty-3

Work Items

Implement base cluster-shrink/cluster-grow functionality
Update MongoDB Clustering implementation to use new APIs

Upgrade Implications

The MongoDB clustering implementation will continue to support the existing “add-shard” functionality.

Dependencies

n/a

Testing

int-tests for clustering functionality is still under consideration.

Documentation Impact

The new cluster-grow and cluster-shrink functionality will need to be documented.

The MongoDB Datastore documentation will need to be updated to reflect Mongo’s implementation of both the old and new functionality.

References

n/a

Redis Clustering

Mon, 15 Jun 2015 00:00:00

https://blueprints.launchpad.net/trove/+spec/redis-cluster

Redis is a NoSQL database designed for high performance caching and persistence of key/value data. This document outlines a proposal for implementing clustering for Redis.

Problem Description

Implement clustering for Redis.

Proposed Change

The Redis clustering feature will strive to implement the minimum functionality required for a user to implement a Redis cluster in a Trove environment. While Trove will provide functionality to support operations not possible directly through the Redis API, functionality will be left to the user to perform via the Redis API wherever possible. This decision is based on the belief that it would be extremely difficult to provide all the required functionality through a web interface, and that Redis users will be familiar with the Redis command set.

Configuration

The following configuration values will be implemented in the Redis configuration group:

cfg.BoolOpt('cluster_support', default=True,
            help='Enable clusters to be created and managed.'),
cfg.StrOpt('api_strategy',
           default='trove.common.strategies.cluster.experimental.'
           'redis.api.RedisAPIStrategy',
           help='Class that implements datastore-specific API logic.'),
cfg.StrOpt('taskmanager_strategy',
           default='trove.common.strategies.cluster.experimental.redis.'
           'taskmanager.RedisTaskManagerStrategy',
           help='Class that implements datastore-specific task manager '
                'logic.'),
cfg.StrOpt('guestagent_strategy',
           default='trove.common.strategies.cluster.experimental.'
           'redis.guestagent.RedisGuestAgentStrategy',
           help='Class that implements datastore-specific Guest Agent API '
                'logic.'),

Database

No changes.

REST API

Create Cluster

The cluster-create command will allow the user to create a cluster with the specified number of master nodes. The data slots will be evenly divided between the created nodes.

Request:

POST /v1.0/<tenant_id>/clusters
{
  "cluster": {
    "name": "redis-clstr",
    "datastore": {
      "type": "redis",
      "version": "3.0"
    },
    "instances": [
      {
        "flavorRef": "2",
        "volume": {
          "size": 2
        }
      },
      {
        "flavorRef": "2",
        "volume": {
          "size": 2
        }
      },
      {
        "flavorRef": "2",
        "volume": {
          "size": 2
        }
      },
      {
        "flavorRef": "2",
        "volume": {
          "size": 2
        }
      },
    ]
  }
}

Response:

{
  "cluster": {
    "id": "edaac9ca-b5e1-4028-adb7-fa7653e11224",
    "task": {
      "id": 2,
      "name": "BUILDING",
      "description": "Building the initial cluster."
    },
    "name": "redis-clstr",
    "created": "2015-01-29T20:19:23",
    "updated": "2015-01-29T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "redis",
      "version": "3.0"
    },
    "ip": [],
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "name": "redis-clstr-member-1",
        "status": "BUILD",
        "ip": [],
        "links": [{...}],
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        }
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
        "name": "redis-clstr-member-2",
        "status": "BUILD",
        "ip": [],
        "links": [{...}],
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        }
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
        "name": "redis-clstr-member-3",
        "status": "BUILD",
        "ip": [],
        "links": [{...}],
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        }
      },
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "name": "redis-clstr-member-4",
        "status": "BUILD",
        "ip": [],
        "links": [{...}],
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        }
      },
    ]
  }
}

HTTP Codes:

- Accepted.
- BadRequest. Server could not understand request.
- Forbidden. Local storage not specified in flavor ID: <ID>.
- Forbidden. A flavor is required for each instance in the cluster.
- Not Found. Flavor not found.

Grow Cluster

Adds nodes to a cluster. The added nodes will be master nodes empty of data and will have no slots assigned to them.

Request:

POST /v1.0/<tenant_id>/clusters/edaac9ca-b5e1-4028-adb7-fa7653e11224/action
{
    "grow": [
      {
        "flavorRef": "2",
        "volume": {
          "size": 2
        }
      },
      {
        "flavorRef": "2",
        "volume": {
          "size": 2
        }
      }
    ]
}

Response:

{
  "cluster": {
    "id": "edaac9ca-b5e1-4028-adb7-fa7653e11224",
    "task": {
      "id": 2,
      "name": "BUILDING",
      "description": "Building the initial cluster."
    },
    "name": "redis-clstr",
    "created": "2015-01-29T20:19:23",
    "updated": "2015-01-29T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "redis",
      "version": "3.0"
    },
    "ip": [],
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "name": "redis-clstr-member-5",
        "status": "BUILD",
        "ip": [],
        "links": [{...}],
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        }
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
        "name": "redis-clstr-member-6",
        "status": "BUILD",
        "ip": [],
        "links": [{...}],
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        }
      },
    ]
  }
}

HTTP Codes:

- Accepted.
- BadRequest. Server could not understand request.
- Forbidden. Local storage not specified in flavor ID: <ID>.
- Forbidden. A flavor is required for each instance in the cluster.
- Not Found. Flavor not found.

Shrink Cluster

Removes the specified nodes from the cluster. It is expected that all data slots have been removed from the node - the shrink operation will fail otherwise.

Request:

POST /v1.0/<tenant_id>/clusters/<cluster_id>/action
    "shrink": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
      }
    ]
}

Response:

N/A

HTTP codes:

- Accepted.
- Forbidden.  One or more nodes have data slots assigned.
- Not found.  Instance <id> does not exist.

Show Cluster

Request:

GET /v1.0/<tenant_id>/clusters/edaac9ca-b5e1-4028-adb7-fa7653e11224

Response:

{
  "cluster": {
    "id": "edaac9ca-b5e1-4028-adb7-fa7653e11224",
    "task": {
      "id": 1,
      "name": "NONE",
      "description": "No tasks for the cluster."
    },
    "name": "redis-clstr",
    "created": "2015-01-29T20:19:23",
    "updated": "2015-01-29T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "redis",
      "version": "3.0"
    },
    "ip": ["10.0.0.1", "10.0.0.2", "10.0.0.3", "10.0.0.4",],
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "name": "redis-clstr-member-1",
        "status": "ACTIVE",
        "ip": ["10.0.0.1"],
        "links": [{...}],
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        },
      }
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
        "name": "redis-clstr-member-2",
        "status": "ACTIVE",
        "links": [{...}],
        "flavor": {
        "ip": ["10.0.0.2"],
          "id": "7",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        },
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
        "name": "redis-clstr-member-3",
        "status": "BUILD",
        "ip": ["10.0.0.3"],
        "links": [{...}],
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        },
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
        "name": "redis-clstr-member-4",
        "status": "BUILD",
        "ip": ["10.0.0.4"],
        "links": [{...}],
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        },
      }
    ]
  }
}

HTTP Codes:

200 - OK.
404 - Not Found. Cluster not found.

Show Instance

Request:

GET /v1.0/<tenant_id>/clusters/edaac9ca-b5e1-4028-adb7-fa7653e11224/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1

Response:

{
  "instance": {
    "status": "ACTIVE",
    "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
    "cluster_id": "edaac9ca-b5e1-4028-adb7-fa7653e11224",
    "name": "redis-clstr-member-1",
    "created": "2015-01-29T20:19:23",
    "updated": "2015-01-29T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "redis",
      "version": "3.0"
    },
    "ip": ["10.0.0.1"],
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 2,
      "used": 0.17
    }
  }
}

HTTP Codes:

- OK.
- Not Found. Cluster not found.
- Not Found. Instance not found.

List Clusters

Request:

GET /v1.0/<tenant_id>/clusters

Response:

{
  "clusters": [
    {
      "id": "edaac9ca-b5e1-4028-adb7-fa7653e11224",
      "task": {
        "id": 1,
        "name": "NONE",
        "description": "No tasks for the cluster."
      },
      "name": "redis-clstr",
      "created": "2015-01-29T20:19:23",
      "updated": "2015-01-29T20:19:23",
      "links": [{...}],
      "datastore": {
        "type": "redis",
        "version": "3.0"
      },
      "instances": [
        {
          "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
          "name": "redis-clstr-member-1",
          "links": [{...}],
        }
        {
          "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
          "name": "redis-clstr-member-2",
          "links": [{...}],
        },
        {
          "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
          "name": "redis-clstr-member-3",
          "links": [{...}],
        },
        {
          "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
          "name": "redis-clstr-member-4",
          "links": [{...}],
        }
      ]
    },
    ...
  ]
}

HTTP Codes:

200 - OK.

Delete Cluster

Request:

DELETE /v1.0/<tenant_id>/clusters/<cluster_id>

Response:

N/A

HTTP codes:

202 - Accepted.
404 - Not found

Public API

No public API changes.

Public API Security

n/a

Python API

No Python API changes.

CLI (python-troveclient)

No CLI changes.

Internal API

No changes are envisioned to the guestagent api, beyond implementing the existing API methods.

Guest Agent

The following methods will be implemented in the RedisGuestAgentAPI:

def get_node_ip(self):
    LOG.debug("Retrieve ip info from node.")
    return self._call("get_node_ip",
                      guest_api.AGENT_HIGH_TIMEOUT, self.version_cap)

def get_node_id_for_removal(self):
    LOG.debug("Validating cluster node removal.")
    return self._call("get_node_id_for_removal",
                      guest_api.AGENT_HIGH_TIMEOUT, self.version_cap)

def remove_nodes(self, node_ids):
    LOG.debug("Removing nodes from cluster.")
    return self._call("remove_nodes", guest_api.AGENT_HIGH_TIMEOUT,
                      self.version_cap, node_ids=node_ids)

def cluster_meet(self, ip, port):
    LOG.debug("Joining node to cluster.")
    return self._call("cluster_meet", guest_api.AGENT_HIGH_TIMEOUT,
                      self.version_cap, ip=ip, port=port)

def cluster_addslots(self, first_slot, last_slot):
    LOG.debug("Adding slots %s-%s to cluster.", first_slot, last_slot)
    return self._call("cluster_addslots",
                      guest_api.AGENT_HIGH_TIMEOUT, self.version_cap,
                      first_slot=first_slot, last_slot=last_slot)

def cluster_complete(self):
    LOG.debug("Notifying cluster install completion.")
    return self._call("cluster_complete", guest_api.AGENT_HIGH_TIMEOUT,
                      self.version_cap)

Alternatives

n/a

Implementation

Assignee(s)

Primary assignee:: vgnbkr

Milestones

Target Milestone for completion:: Liberty-3

Work Items

implement clustering for Redis

implement new Taskmanager cluster strategy for Redis implement python API and shell for shrink/grow (this may be already done through cluster-scaling bp implementation) add grow/shrink to cluster strategy (in absence of scaling implementation) implement guest agent support for joining/leaving a cluster implement unit tests as appropriate implement int-test if the mechanism for doing so has been worked out by that time

Upgrade Implications

As this is a new implementation, no upgrade implications are envisioned.

Dependencies

This functionality depends on the cluster scaling functionality outlined in https://blueprints.launchpad.net/trove/+spec/cluster-scaling

Testing

Implementation of int-tests for clustering is still being worked out for MongoDB and Cassandra. It is expected that Redis will implement/run similar int-tests.

Documentation Impact

Datastore documentation for Redis will need to be updated to reflect clustering support.

References

http://redis.io/topics/cluster-tutorial http://redis.io/topics/cluster-spec

Root Disable

Tue, 09 Jun 2015 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/trove/+spec/root-disable

Since there is the ability to enable a root user on an instance the ability to disable the root user should be provided as well.

Problem Description

Currently, the root user can be enabled on an instance. It is expected that the root user can also be disabled.

Disabling the root user should not change the ability of the command root-show to determine if the root user has ever been enabled.

Proposed Change

This change will add a new root-disable command.

The command will remove the root user from the specified instance. No changes will be made in the root_enable_history table from the execution of the command. This ensures the root-show command will continue to operate as expected.

Configuration

None

Database

None

Public API

REST API:

DELETE /instances/{id}/root

Public API Security

None

Python API

Python API in class Root:

def delete(self, instance):
“””Implements root-disable API.

Removes the root user for the specified db instance.

param instance

The instance from which the root user is removed from

“””

CLI (python-troveclient)

CLI:

trove root-disable <instance>

Internal API

None

Guest Agent

The appropriate root disable method will added only for MySQL. All other datastores will also need to have the appropriate not implemented error.

Alternatives

Another option is root-disable will simply call root-enable and not return the password back to the user. This alternative also has no impact on the root-show.

However, an operator may believe the root user to be completely removed from the database after the root-disable call is made. Leaving the root user may not be what the operator is expecting.

Discussion Summary

Some concerns were raised during the discussion of this blueprint. The full details can be found here https://review.openstack.org/#/c/189837/.

In summary the concerns revolved around the fact that after the user performs a root enable the following issues can occur.

A security hole exists where a user can create an alternate root user, drop the existing root user and do a backup then restore leaving no trace that the root user has been enabled for the restored database and an “unknown” alternate root user is left in the restored database.
The enabled root user can change the management user access leaving the instance in a state that cannot be managed via Trove.

It was determined that adding the root-disable command does not cause or make the concerns raised above to be any worse than already exists.

Implementation

Assignee(s)

Primary assignee:: dloi

Milestones

liberty-1

Work Items

Implement CLI, Python API and REST API calls
In the guest add disable_root method for MySQL database and not implemented stubs for all other datastores
Unit and integration tests

Upgrade Implications

None

Dependencies

None

Testing

Add to existing root enable tests to test disabling the root user.

Documentation Impact

New root-disable command needs to be added to the API documentation.

References

None

Redis Replication

Mon, 08 Jun 2015 00:00:00

Replication functionality needs to be added to the Redis datastore.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/redis-replication

Problem Description

At present, only single instances of Redis can be created. While useful, having multiple slaves that replicate off of a designated master is also desirable. This functionality will be addressed in this spec.

Proposed Change

Redis replication is a very simple to use (and configure) master-slave replication. It allows Redis slaves to be exact copies of master servers. 1

Replication

Redis replication has the following features:

Asynchronous
Multiple slaves
Slave-of-slave
Non-blocking initialization of slaves

To improve performance, persistence can be turned off on the master node. This however can lead to a loss of data if the system reboots and Redis starts automatically. For this reason, the master node will be required to have persistence enabled.

Creating a Redis replication network is handled by the Redis SLAVEOF command. A new instance (or set of instances) will be created and the SLAVEOF command executed on each one. Having Trove create a backup and restore it is not necessary, as Redis has this capability built into the SLAVEOF command. This means that the Redis replication strategy will need to bypass the creation of a backup to add to the snapshot info, and the taskmanager will need to be modified to handle this case.

Note: Redis replication could be enabled using the current backup/restore
implementation, however once the slave restarts (or starts for the first
time), it will automatically do a full resync, thus rendering the backup
obsolete.  [1]_

Enough disk space must be available on the master node to allow Redis to persist its data to storage.

Note: Starting in version 2.8.18, Redis has the (experimental) ability to
stream the backup directly to the slave.  Since this behaviour is still
considered experimental (in version 3.0), a specific version of Redis will
not be required - beyond being >= 2.8 - as the feature could be removed in
a future release.  If it exists, however, it can be used by Redis to
increase performance on systems with slow disks.  A configuration parameter
will be provided to allow operators to turn this feature on.

The Redis configuration file on each slave will have the corresponding values set so that subsequent starting of the database will maintain the slave status. As part of the slave configuration, all slaves will also be set to read-only. As with the MySQL implementation, slave-of-slave will not be allowed. The feature could be augmented to include this in the future.

The steps to create a replication network is as follows:

Create the necessary configuration file. This will have the following settings:
- slaveof <master_ip> <master_port>
- slave-read-only
- repl-diskless-sync-delay (if more than one slave is specified)
Create ‘n’ new slave instances with the correct configuration file

Detach Replica

The current API for detach-replica will need to be implemented. No additions to the API are anticipated.

Failover

The current APIs for failover (both eject-replica-source and promote-to-replica-source) will need to be implemented. When ejecting the current replica source, a slave needs to be chosen as the new one. This will be done by overriding the _most_current_replica() method and having it query each slave and choose the one with the smallest value for ‘master_last_io_seconds_ago.’ This, presumably, will be the one with the most current data.

Configuration

The default values for the following config options will need to be updated:

replication_strategy

Database

None

Public API

None

Public API Security

None

Python API

Existing Python bindings are sufficient, and no changes are anticipated.

CLI (python-troveclient)

Once these changes are implemented, the following Trove CLI commands will now be fully functional with Redis:

create –replica_of <id> –replica_count <n>

eject-replica-source

promote-to-replica-source

detach-replica

Internal API

None

Guest Agent

The following files will need to be added to the guest agent, where the corresponding implementation will reside:

guestagent/strategies/replication/experimental/redis_sync.py

The following existing files will be updated:

guestagent/datastore/experimental/redis/manager.py
guestagent/datastore/experimental/redis/service.py
guestagent/datastore/experimental/redis/system.py

No backwards compatibility issues are anticipated.

Alternatives

No alternative solutions are proposed at this time.

Implementation

Assignee(s)

Primary assignee:: peterstac

Milestones

Target Milestone for completion:: Liberty-2

Work Items

Create replication strategy for Redis.
Implement API calls for detach_replica, promote_to_replica_source and eject_replica_source.

Upgrade Implications

None

Dependencies

None

Testing

No new tests are deemed to be required (beyond the requisite unit tests). The int-tests group for Redis will be modified to run replication-related commands during integration test runs.

Documentation Impact

Datastore specific documentation should be modified to indicate that replication is now supported by Redis, along with the corresponding detach/failover commands.

References

1: Redis Replication: http://redis.io/topics/replication

Guestagent Configuration Manager

Thu, 04 Jun 2015 00:00:00

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/guestagent-configuration-manager

Problem Description

Each datastore currently has to implement a strategy for maintaining its configuration files including functionality for parsing, updating properties, saving and managing overrides. With increasing number of supported datastores this approach leads to a significant duplication in the production and testing codebase.

Proposed Change

Facilitate code reuse by implementing a manager class that could be used by all guestagents to manage their configuration files and overrides.

The patch set will consist of four main features.

Codecs for serialization and deserialization of configuration files.

These would be responsible for parsing a Python structure (dict) into an appropriate serialized form (see the simple example below).

An INI-style file (serialized form):
[section_1]
key1 = value1
key2 = value2
...

[section_2]
key1 = value1
key2 = value2
...
Dict representation of the above contents (deserialized form):
{'section_1': {'key1': 'value1', 'key2': 'value2', ...},
 'section_2': {'key1': 'value1', 'key2': 'value2', ...}
 ...
}
Codecs for common configuration formats currently present in Trove will be implemented.
These include:
INI-style format: see the example above

YAML format: see http://pyyaml.org/wiki/PyYAMLDocumentation for examples of Python representation.
Properties format:
Serialized form:
key1 k1arg1 k1arg2 ... k1argN
key2 k2arg1 k2arg2 ... k2argN
key3 k3arg1 k3arg2 ...
key3 k3arg3 k3arg4 ...
Dict representation of the above contents (deserialized form):
{'key1': [k1arg1, k1arg2 ... k1argN],
 'key2': [k2arg1, k2arg2 ... k2argN],
 'key3': [[k3arg1, k3arg2, ...], [k3arg3, k3arg4, ...]]}
Functions for reading and writing using a given codec.

A ‘write’ function saves a serialized form of a given dict into a file. A ‘read’ function applied on the file produces the same dict structure that was written in the previous step. The write function will write a file with superuser privileges when provided an optional ‘as_root’ keyword. This will be implemented by writing a temporary file first moving it to place using the existing operating system calls.

The configuration manager.

ConfigurationManager will be responsible for management of datastore configuration files. Its base functionality will include reading and writing configuration files and updating or retrieving current values. It will be also responsible for validating user inputs and requests. When supplied an override strategy (below) it allows the user to manage configuration overrides as well.

The configuration manager will be responsible for enforcing the limit on the number of applied overrides (there can currently be only one override applied to an instance at any given time).

Common override strategies.

An override strategy object responsible for management of overrides in the configuration manager (when provided).

It will implement functions to:

apply updated values on the current revision of the configuration file.

remove the last applied overrides and effectively restoring the previous version of the configuration file.

The strategies provided with this patch set will be:

Rolling Override Strategy:

A strategy suitable for applications that do not support includes in their configuration files. It applies updates to the configuration file while maintaining a backup of the previous version that can be restored when the overrides get removed.

Apply procedure:

Save a backup copy of the current configuration file to a known configurable location.

Load and parse the current configuration file into a dict (using an appropriate codec).

Update the dict with the overridden values.

Overwrite the configuration file with the serialized form of the dict (using the same codec).

Remove procedure:

Move the appropriate backup revision over the current configuration file.

Import Override Strategy:

A strategy useful for datastores that support imports in their configuration files (like MySQL). The overrides are stored in a known configurable directory which is then imported by the base configuration file which itself remains intact.

Apply procedure:

Store a serialized form (using an appropriate codec) of the overrides dict in an imported location.

Remove procedure:

Remove the appropriate file from the imported location.

Configuration

None

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Internal API

None

Guest Agent

The affected files would include:

guestagent/common/operating_system.py: I/O functions

common/stream_codecs.py: Implementations of codecs.

common/configurations.py: Reuse the codecs in configuration parsers.

guestagent/common/configuration.py: Implementations of the configuration manager and override strategies.

Alternatives

None

Implementation

Assignee(s)

Petr Malik <pmalik@tesora.com>

Milestones

Liberty-1

Work Items

Implement codecs for serialization and deserialization of common configuration files.
Implement functions for reading and writing files using a given codec.
Implement the configuration manager.
Implement common override strategies.
Add unit tests for the above functionality.

Upgrade Implications

None

Dependencies

The current implementation works around limitations of the ConfigParser in Python 2.6. OpenStack no longer supports this version of Python and Trove gate tests run against Python 2.7 We can therefore remove the compatibility requirement and make full use of the ConfigParser in Python 2.7 in the existing code base.

Testing

Unit tests will be added to cover non-trivial code paths.

Documentation Impact

None

References

None

Datastore Registration API

Wed, 03 Jun 2015 00:00:00

Blueprint:

https://blueprints.launchpad.net/trove/+spec/datastore-registration-api

The proposal is to introduce a new Management API that facilitates a the datastore registration on an existing Trove Deployment

Problem Description

Today, the only means of adding/registering a datastore/datastore-version on an existing deployment is to use some sequence of trove-manage (datastore_update, datastore_version_update) to add a datastore followed by adding the datastore-version with its image.

Another problem with this process is one needs to have access to the trove-api server from where one can execute trove-manage, alteratively one needs direct access to database if the want to by-pass trove-manage and add the entries by using an insert/update command to tables, both of these methods requires to relax the security of trove-api/database server for direct access to operator/deployer.

One more problem that lies with registering datastores is we need to use multiple commands to register a new datastore with an image-id.

Use Cases

As a deployer, I want to be able to add new datastores through the Trove management API, without having to log in to the Trove control plane and manually running a trove-manage command.

Proposed Change

Following functionalities would be added as part of the change::

Add/register a new datastore version.
Get a list of datastore-versions.
Get information about an existing datastore-version.
Update image/manager/packages/active/default for an existing datastore version.
Delete an existing datastore version.

While registering a datastore version, we would create a datastore with the requested datastore_name, if it does not exists.

These functionalities would only be available to an admin user who would access the APIs via an admin tenant.

Configuration

No Configuration changes.

Database

No Database changes.

Public API

No public API changes.

Public API Security

None

Python API

None

CLI (python-troveclient)

None

Management API

The request/response for new APIs are as follows. These APIs would be invokable only through an admin tenant.

Add/Register Datastore Version

Request:

POST /v1.0/<tenant_id>/mgmt/datastore-versions
{
  "version": {
    "datastore": "mysql",
    "name": "5.6",
    "manager": "mysql",
    "image": "154b350d-4d86-4214-9067-9c54b230c0da",
    "packages": ["mysql-server-5.6"],
    "active": true,
    "default": true,
    }
}

Response:

{
}

HTTP Codes:

- Accepted.
- Bad Request. Datastore Version Already Exists.
- Not Found. Image not found.

Get Datastore Versions

Request:

GET /v1.0/<tenant_id>/mgmt/datastore-versions

Response:

{
    "versions": [
      {
        "datastore_id": "b80c2b43-cd87-4d5d-9f32-a4996bd57cb1",
        "datastore_name": "mysql",
        "id": "b8a23fa1-1faf-441a-a6b7-83a19c30f347",
        "name": "5.6",
        "manager": "mysql",
        "image": "154b350d-4d86-4214-9067-9c54b230c0da",
        "packages": ["mysql-server-5.6"],
        "active": true,
        "default": true,
      },
      {
        "datastore_id": "127bc577-8054-4b32-9ed3-2d6b01773810",
        "datastore_name": "vertica",
        "id": "21c8805a-a800-4bca-a192-3a5a2519044d",
        "name": "7.1",
        "manager": "vertica",
        "image": "6230baf1-dffe-40fa-a1fb-47d9ff346503",
        "packages": ["vertica-7.1"],
        "active": true,
        "default": true,
      }]
}

HTTP Codes:

200 - OK.

Get information about a Datastore Version

Request:

GET /v1.0/<tenant_id>/mgmt/datastore-versions/<datastore_version_id>

Example:

GET /v1.0/<tenant_id>/mgmt/datastore-versions/b8a23fa1-1faf-441a-a6b7-83a19c30f347

Response:

{
    "version":
      {
        "datastore_id": "b80c2b43-cd87-4d5d-9f32-a4996bd57cb1",
        "datastore_name": "mysql",
        "id": "b8a23fa1-1faf-441a-a6b7-83a19c30f347",
        "name": "5.6",
        "manager": "mysql",
        "image": "154b350d-4d86-4214-9067-9c54b230c0da",
        "packages": ["mysql-server-5.6"],
        "active": true,
        "default": true,
      }
}

HTTP Codes:

200 - OK.
404 - Not Found. Datastore Version not found.

Update Image/Manager/Packages/Active/Default for a Datastore Version

Request:

PATCH /v1.0/<tenant_id>/mgmt/datastore-versions/<datastore_version_id>

Payload for this request can be those attributes which one wants to update.

Example:

PATCH /v1.0/<tenant_id>/mgmt/datastore-versions/b8a23fa1-1faf-441a-a6b7-83a19c30f347
{
    "image": "e33f8e2f-1148-461c-a7ea-f8228e7c5f4a",
}

Response:

{
}

HTTP Codes:

- Accepted.
- Not Found. Datastore Version not found.
- Not Found. Image not found.

Delete an existing Datastore Version

Request:

DELETE /v1.0/<tenant_id>/mgmt/datastore-versions/<datastore_versionid>

Example:

DELETE /v1.0/<tenant_id>/mgmt/datastore-versions/b8a23fa1-1faf-441a-a6b7-83a19c30f347

Response:

{
}

HTTP codes:

- Accepted.
- Not Found. Datastore Version not found.
- Conflict. Instance(s) exists for the datastore version.

Internal API

No internal API changes.

Guest Agent

No Guest Agent changes.

Alternatives

The current alternative is use trove-manage or use database directly, but both these methods need relaxation on access-policies of api/db servers which has its own security implications.

Implementation

Assignee(s)

Sushil Kumar (skm.net@gmail.com)

Milestones

Liberty-2

Work Items

Implement API routes
Implement management API
Implement unit-tests

Upgrade Implications

None.

Dependencies

There is an ongoing work to associate flavors with datastore versions(https://review.openstack.org/#/c/109824).

It has to be noted once we have these flavor-mappings in place, there would be two available options while deleting a datastore-version:

Don’t delete datastore-version if it has any flavor-mappings.
Delete the flavor-mappings along with the datastore-versions.

Testing

Unit tests will be added to cover non-trivial code paths.

Documentation Impact

Management API’s documentation would be updated with new API.

References

None.

Cluster root enable

Thu, 09 Apr 2015 00:00:00

This spec is intended to add root enable support for clusters.

Launchpad blueprint:

https://blueprints.launchpad.net/trove/+spec/vertica-cluster-user-features

Problem Description

Trove’s cluster API currently does not support root enable.

Proposed Change

Add support for root-enable for clusters, with optional user-provided passwords. User supplied passwords are necessary to create a good user experience in Horizon where enabling a root user should allow user input rather than requiring the user to wait for a password to popup on a response dialog.

Configuration

None

Database

None

Public API

This change will add a new /clusters/<cluster_id>/root resource which is similar to the /root resource on instances with the addition of an optional password.

Request:

{
  "password": "secretsecret"
}

POST /v1.0/[account-id]/clusters/<cluster_id>/root

Response:

{
  "username": "root",
  "password": "secretsecret"
}

GET /v1.0/<tenant_id>/clusters/<cluster_id>/root
DELETE /v1.0/<tenant_id>/clusters/<cluster_id>/root

Public API Security

The security of this API will be handled in the same manner as existing implementations.

Python API

None

CLI (python-troveclient)

New cluster-root-enable and cluster-root-show commands will be added

Changes will effect:

troveclient/v1/root.py troveclient/v1/shell.py troveclient/v1/clusters.py

Internal API

Root enablement support should be added for clusters. Changes include:

trove/extensions/routes/mysql.py trove/extensions/mysql/service.py trove/extensions/mysql/models.py trove/guestagent/api.py

Guest Agent

The guestagent API will get a new enable_root(self, password) function which allows optionally provided passwords to be supplied to a guest agent. This function will be in addition to the existing function, allowing for backwards compatibility with existing images.

Additionally, each guest agent datastore impl will need to implement the enable_root(password) function.

Alternatives

None

Implementation

Assignee(s)

Milestones

Liberty

Work Items

API changes
Guest agent changes
CLI changes
Internal API changes

Upgrade Implications

None

Dependencies

None

Testing

New unit and scenario tests will be added to assert that root enablement works as expected at the cluster level

Documentation Impact

The API changes to the clusters resource and the addition of the new cluster root resource need to be documented.

References

None

Add instance name as parameter to various CLI commands

Tue, 24 Feb 2015 00:00:00

Blueprint:

https://blueprints.launchpad.net/trove/+spec/add-instance-name-to-cli

The proposal is to allow instance-name to be specified wherever instance-ID is currently used throughout the CLI.

Problem Description

Currently, only the “trove show” command will take instance-ID or instance-name as its parameter. There are many more commands in the CLI that require an instance reference as a parameter but require that the instance-ID be used. It would be helpful to customers to be able to use instance-ID or instance-name interchangeably throughtout the CLI.

Proposed Change

Allow instance-id or instance-name to be passed in for the following commands:

backup-create
backup-list-instance
configuration-attach
configuration-default
configuration-detach
database-create
database-delete
database-list
detach-replica
delete
metadata-create
metadata-delete
metadata-edit
metadata-list
metadata-show
metadata-update
resize-flavor
resize-instance
resize-volume
restart
root-enable
root-show
update
user-create
user-delete
user-grant-access
user-list
user-revoke-access
user-show
user-show-access
user-update-attributes

For example, trove delete currently looks like this:

usage: trove delete <instance>

Deletes an instance.

Positional arguments:
    <instance>  ID of the instance.

The proposal is to make it look like this:

usage: trove delete <instance>

Deletes an instance.

Positional arguments:
    <instance>  ID or name of the instance.

This will be a CLI change only.

Currently the trove show command will display an error if there is more than one instance with the name provided. The error indicates that, in this case, the instance ID must be used. This behavior will be preserved and used across all the commands mentioned above.

Configuration

None

Database

None

Public API

None

Public API Security

None

Internal API

None

Guest Agent

None

Alternatives

None

Implementation

Assignee(s)

Primary assignees:

0-doug (dougshelley66)
peterstac

Milestones

Target Milestone for completion:: Kilo-2

Work Items

Go through python-troveclient/troveclient/v1/shell.py and call _find_instance(cs, args.instance) ahead of the “real” work in each “do_<>” method.
Alter the help text in shell.py to indicate the use of name or ID.
Alter the existing unit tests as appropriate.

Dependencies

None

Testing

If we had Tempest coverage for the CLI it would be adjusted to exercise passing instance name to all the altered commands. Also, there is an existing LP bug [1] that indicates we should write tests for the CLI.

Both of these are considered out of scope for this BP.

[1] https://bugs.launchpad.net/python-troveclient/+bug/1314793

Documentation Impact

The help text for the aforementioned commands will be altered to now include instance name as a valid parameter. I believe that the CLI documentation is generated from the code.

References

None

Associate instance flavors with datastores

Tue, 24 Feb 2015 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/trove/+spec/associate-flavors-datastores

Motivation: Each datastore type has its own hardware requirements (minimum and maximum). Currently, trove lacks the ability to enforce this requirement. This change enables trove with the ability to associate the virtual hardware templates (flavors) with datastore types.

Problem Description

Trove supports multiple datastore types. However, each datastore type has its own hardware requirements, which presently, cannot be enforced in trove.

For example: MySQL - Minimum System Requirements: - 2 or more CPU cores - 2 or more GB of RAM - Disk I/O subsystem applicable for a write-intensive database

Find more information about other datastore specific hardware requirements in the References section.

Trove uses virtual hardware templates called ‘flavors’ in OpenStack. In order for trove to be able to enforce the datastore specific hardware requirements, there needs to be a way in which datastore types can be associated with their minimum flavor (hardware) requirements. This way, the user/administrator can at least be notified while provisioning the datastore with the inappropriate flavor, that does not meet the minimum hardware requirements.

Proposed Change

The trove-manage utility will provide the ability to add and delete the datastore version-flavor associations.
There will be an additional API call in order to get a list of flavors for the specified datastore version id.

Configuration

None

Database

No impact on existing tables.

One new entity will be created in the trove database: datastore_version_metadata. This will store any additional metadata related to a datastore version including its relation with flavors: key=flavor and the value=flavor_id.

The blueprint specifies the table attributes in detail.

Public API

Does not impact any other API which the user has access to.

Public API Security

None

Internal API

None

Guest Agent

None

Alternatives

None

Implementation

Assignee(s)

Launchpad: riddhi89

IRC: Riddhi

Email: ridhi.j.shah@gmail.com

Milestones

Kilo-1

Work Items

Add the new database schema for datastore version metadata.
trove-manage util extension to associate a flavor list with a datastore version id
trove-manage util extension to delete a flavor associated with a datastore version id.
REST API call to list flavors for a datastore version id (/{tenant_id}/flavors/{datastore_version_id})
Unit and integration tests

Dependencies

None

Testing

Unit tests for individual components.
Integration tests for end-to-end testing.

Documentation Impact

New API call to list flavors given a datastore version id.
trove-manage util extensions to add/delete datastore version - flavor associations.

References

Further hardware requirements for different datastore types: https://wiki.openstack.org/wiki/TroveFlavorsPerDatastore

Datastore Visibility

Tue, 24 Feb 2015 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/trove/+spec/datastore-visibility

Motivation: Since Trove supports multiple datastore types, there is a need for Trove to have a greater control over the customer visibility of these datastore types. This change enables Trove to control the visibility of the various datastore types.

Problem Description

There might be some datastore types, which the deployers require to be active but not visible to customers in the production environment. Example use case: Say we want to have an active datastore A in production and not expose it to customers yet.

Proposed Change

This change suggests adding a visibility attribute to the datastore versions. This enables the datastore to still be active, but not visible to the users. The visibility flag will ensure it is visible on the datastore list call only to admins.

Visibility attribute to the datastore version. It can be: public/private/unlisted/deprecated.
Adding a datastore version members table to add tenants for private datastores.
If visibility is public:
- All users can view it in the list.
- All users can make a GET call on the datastore version.
- All admin can view it in the list.
- All admin can make a GET call on the datastore version.
If visibility is private:
- Members only can view it in the list.
- Members only can make a GET call on the datastore version.
- All admin can view it in the list.
- All admin can make a GET call on the datastore version.
- All admin can add and remove tenants as members of a datastore version.
If visibility is unlisted:
- All users can make a GET call on the datastore version.
- All admin can view it in the list.
- All admin can make a GET call on the datastore version.
If visibility is deprecated:
- All admin can view it in the list.
- All admin can make a GET call on the datastore version.
- Only admins can create a ‘deprecated’ instance.

Configuration

None

Database

Database migration of adding a column ‘visibility’ to the datastore_versions table.

Table datastore_versions:

Field          | Type        | Null | Key     | Default | Extra|
---------------------------------------------------------------|
id             | varchar(36) | NO   | PRIMARY | NULL    |      |
datastore_id   | varchar(36) | YES  | MUL     | NULL    |      |
name           | varchar(255)| YES  |         | NULL    |      |
image_id       | varchar(36) | NO   |         | NULL    |      |
packages       | varchar(511)| YES  |         | NULL    |      |
active         | tinyint(1)  | NO   |         | NULL    |      |
manager        | varchar(255)| YES  |         | NULL    |      |
visibility     | varchar(255)| NO   |         | public  |      |

New table datastore_version_members which consists of columns - id, datastore_version_id,tenant_id.

Table datastore_version_members:

Field               | Type        | Null | Key         | Default | Extra|
------------------------------------------------------------------------|
id                  | varchar(36) | NO   | PRIMARY     | NULL    |      |
datastore_version_id| varchar(36) | YES  | Foreign [1] | NULL    |      |
tenant_id           | varchar(36) | NO   |             | NULL    |      |

[1] datastore_version_id is a Foreign-Key on datastore_version.id

Public API

Public API Security

None

Internal API

None

Guest Agent

None

Alternatives

None

Implementation

Assignee(s)

Primary:

Launchpad: riddhi89
IRC: Riddhi
Email: ridhi.j.shah@gmail.com

Co-Authored by:

Theron Voran
Email: theron.voran@rackspace.com

Milestones

Kilo-1 Kilo-2

Work Items

Already in review process - References [1].

Implementation

It is in the process of review - References [1].

Dependencies

None

Testing

Unit tests, fake tests and real mode tests.

Documentation Impact

Since API calls have been added/modified, their respective samples would need to be incorporated in the API docs.

References

https://review.openstack.org/#/c/110197/

Guest RPC Ping Pong

Tue, 24 Feb 2015 00:00:00

Blueprint:

https://blueprints.launchpad.net/trove/+spec/guest-rpc-ping-pong

The proposal is to introduce a new Management API route that facilitates a lightweight ping/pong mechanism for determining whether RPC communication is working as expected.

Problem Description

Today, the only means of ascertaining whether direct or indirect RPC connectivity between the control-plane and the guest is functioning, via an API, is by using the ‘show’ operation in the Management API.

Although mgmt-show indirectly answers the question of “is the guest reachable?”, it is a relatively heavyweight call. Beyond consulting the filesystem for rudimentary information, mgmt-show sends ‘show’ requests to Nova and Cinder.

Use Cases

As a deployer, I want to be able to determine whether RPC connectivity between the control-plane and a specific guest is broken.

Proposed Change

Introduce:

GET HOST:8779/v1.0/<tenant_id>/mgmt/instances/<instance_id>/rpc_ping

Behavior

The request:

GET HOST:8779/v1.0/<tenant_id>/mgmt/instances/<instance_id>/rpc_ping

will result in the following if it’s successful:

HTTP/1.1 204 No Content
Content-Length: 0
Content-Type: application/json
Date: Thu, 02 Oct 2014 23:32:26 GMT

on the other hand, if trove-guest is shutdown, or the guest is generally not responding, the following will be returned:

HTTP/1.1 400 Bad Request
Content-Length: 237
Content-Type: application/json; charset=UTF-8
Date: Thu, 02 Oct 2014 23:32:59 GMT

{"badRequest": {"message": "An error occurred communicating
 with the guest: Timeout while waiting on RPC response -
 topic: \"guestagent.1e4099f1-7f8d-4564-bd83-0b9f66aa35b9\",
 RPC method: \"rpc_ping\" info: \"<unknown>\".", "code": 400}}

Configuration

No Configuration changes.

Database

No Database changes.

Public API

No public API changes.

Public API Security

No public API Security related changes.

Management API

Introduces:

GET HOST:8779/v1.0/<tenant_id>/mgmt/instances/<instance_id>/rpc_ping

Internal API

No internal API changes.

Guest Agent

No Guest Agent changes.

Alternatives

The current alternative is mgmt-show, but it’s too heavyweight of a call, and can fail to return meaningful information if the underlying compute and/or volume are unhealthy.

Implementation

Assignee(s)

Primary assignee:: Auston McReynolds (amcrn)

Milestones

Kilo-1

Work Items

See https://review.openstack.org/#/c/125819/

Dependencies

No dependencies.

Testing

Standard.

Documentation Impact

If the Management API is documented (which I don’t believe it is), then the addition of the ‘rpc_ping’ route is relevant.

References

None.

Enhance Mgmt-Show To Support Deleted Instances

Tue, 24 Feb 2015 00:00:00

Blueprint:

https://blueprints.launchpad.net/trove/+spec/mgmt-show-deleted

Today, GET /v1.0/<tenant_id>/mgmt/instances/<instance_id> does not work for Trove instances that are marked as deleted.

The proposal is to introduce an optional query parameter that indicates that the Trove instance information should be returned irrespective of whether the Trove instance is marked as deleted or not.

Problem Description

Without an API to retrieve information about a deleted instance, the deployer is forced to consult the Trove infrastructure database directly.

The mgmt-list operation supports returning deleted instances if ?deleted=true is provided, so it only makes sense that additional information about a deleted instance be accessible via mgmt-show.

Use Cases

As a deployer, I want to be able to retrieve information about a deleted Trove instance.

Proposed Change

Support the ?deleted=true/false query parameter in GET /v1.0/<tenant_id>/mgmt/instances/<instance_id>

Changes In Behavior:

If deleted=true, if the Trove instance UUID is present in the infrastructure database, the instance information is returned and the request is successful.
If deleted=false, or the deleted query parameter is omitted, the request will only succeed if the trove instance is not marked as deleted.
If the compute_instance_id exists in the trove.instances row, return it in the mgmt-show response at instance.server.id, even if the Nova API consultation fails.
If the volume_id exists in the trove.instances row, return it in the mgmt-show response at instance.volume.id, even if the Cinder API consultation fails.

Configuration

No Configuration changes.

Database

No Database changes.

Public API

No public API changes.

Public API Security

No public API Security related changes.

Management API

GET /v1.0/<tenant_id>/mgmt/instances/<instance_id> will be enhanced to support the ‘deleted’ query parameter.

For reference, the response of a mgmt-show: http://git.io/2RwWWA

Today, if the compute_instance_id is not recognized by the Nova API, the mgmt-show response will not include instance.server{}. Along the same lines, if the volume_id is not recognized by the Cinder API, the mgmt-show response will not include instance.volume{}.

This blueprint changes that behavior by always returning instance.server.id and instance.volume.id if they exist in the trove.instances table.

Internal API

No internal API changes.

Guest Agent

No Guest Agent changes.

Alternatives

No relevant alternatives given the prior art on using ‘deleted=true/false’ in other Trove routes.

Implementation

Assignee(s)

Primary assignee:: Auston McReynolds (amcrn)

Milestones

Kilo-1

Work Items

See https://review.openstack.org/#/c/128446/

Dependencies

No dependencies.

Testing

Standard.

Documentation Impact

If the Management API is documented (which I don’t believe it is), then the addition of the ‘deleted’ query parameter is relevant.

References

None.

Use native OS::* Heat resources for internal orchestration

Tue, 24 Feb 2015 00:00:00

https://blueprints.launchpad.net/trove/+spec/native-os-heat-resources

Problem Description

A far-fetching goal in Trove is to use OpenStack Orchestration service (Heat) for all the internal orchestration tasks. Current Trove usage of Heat involves exclusively AWS-compatible template syntax and Heat resources.

As of Icehouse OpenStack release Heat community declared new, native HOT template format to be stable and ready for wide usage. Moreover, most of the innovation will happen in native OS::* resources and HOT template format to fully leverage available OpenStack functionality, as they do not have to keep compatibility with AWS CloudFormation service.

Proposed Change

Align default Heat template with latest changes in Heat. That involves:

usage of HOT template format and internal functions;
usage of native OS::* resources wherever possible.

Configuration

All the changes are in the default.heat.template file.

Database

None

Public API

None

Public API Security

No public API Security related changes.

Internal API

None

Guest Agent

None

Alternatives

Do not change anything and continue to use AWS-compatible resources, missing many of the new features Heat can offer.

Implementation

Assignee(s)

Primary assignee:: pshchelo

Milestones

Kilo-2

Work Items

rewrite default.heat.template to use as much of native Heat capabilities and OS resources as possible (implementation is on review https://review.openstack.org/#/c/112035/)

Dependencies

None

Testing

Unfortunately testing Heat-based orchestration is not yet enabled on Trove gates. Enabling it might be a topic for another blueprint.

Documentation Impact

None

References

None

Trove Replication V2

Tue, 24 Feb 2015 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/trove/bp/replication-v2

The Juno release of Trove laid the foundation of Trove Replication support. The V1 version of replication focused on providing read-only slave replication in MySQL 5.5. For the V2 replication release for Kilo, replication will be extended to provide support for manual failover in MySQL replication leveraging the latest replication features of MySQL 5.6.

Problem description

For the Kilo release of OpenStack, trove replication support will be extended to support manual failover when a replication master fails. Specifically, this means that a user can instruct Trove to demote a replication master and promote a slave to be the new master. For V2, manual promotion means that the user will be required to execute an action to cause failover - a component to detect failure and cause the failover to occur will not be within the scope of V2.

Proposed change

Supported Features:

manual failover
master/slaves in different availability zones
automatic slave generation to replace slaves promoted to master
automatically generated slave will be created in the same az as the slave that was promoted to master
public ips assigned to deleted/demoted master will be transferred to new master
public ips of promoted slave will be transferred to new slave
GTIDs will be used to facilitate master promotion (Note: this limits feature set to MySQL 5.6 and later)
if a master site is reachable, a chosen slave may be promoted to master and the old master will be demoted to a slave. This operation will be done in such a way as to prevent the loss of data. This operation would be useful for resizing a master without downtime.
a master site may be deleted, in which case Trove will pick a slave to be promoted to master (see MASTER_PROMOTION_STRATEGY below) and a new slave will be generated to replace the promoted slave. If the master site in not reachable, it will be forcefully removed from Trove/Nova; this is how an unreachable master would be “failed over”.
new master selection process on delete has following MASTER_PROMOTION_STRATEGY (CONF) switch: MOST_RECENT: the slave with the most recent updates is chosen as new master, PROXIMATE_AZ: slave IN MASTER’s AZ with most recent updates is chosen as new master, PROXIMATE_REGION: slave IN MASTER’s REGION with most recent update is chosen as new master. PROXIMATE_REGION will be the default (though for now equivalent to MOST_RECENT) and may be the only implemented option for V2.
replication from existing backup and incremental snapshot will be implemented
replica_count option will be added to create-instance to allow N slaves to be spun up from a given snapshot. All replicas from the given snapshot will have the same “create-instance” options.

Features Not Supported:

automatic failover
region support
writable slaves
features related to the promotion of slaves to masters will not be supported by MySQL versions prior to 5.6
replication_strategy per datastore - this could be implemented in Kilo via an independent blueprint
GTID based replication for MariaDB (binlog replication will not be tested for MariaDB, but should be compatible with MySQL)
host affinity/anti-affinity
dealing with “error transactions” created when updates are executed directly on slaves in conflict with changes on the master. Performing updates directly on slaves is not supported by Trove and slave sites will be put into “read only” mode.

Replication V2 Components

The V2 Replication feature will consist of several components:

Implement a new replication strategy to support GTID Based Replication in addition to Bin Log replication.
Manual failover from replication master
Replication configuration using incremental snapshots based on existing backups.
Creation of multiple slaves from master in single call

Upgrade from Binlog Replication to GTID Based Replication

MySQL 5.6 introduced a new type of replication which is based on Global Transaction IDs (GTID). By assigning a GTID to each transaction, MySQL is able to simplify transaction coordination between masters and slaves, allowing for simpler and more reliable failover to a new master.

This feature requires that the trove-integration project upgrade to use Ubuntu 14.04 and MySQL 5.6.

A new Replication Strategy named “MysqlGTIDReplicationStrategy” will be created to support the new GTID based replication with MySQL 5.6 and later, and the existing Replication Strategy named “MysqlBinlogReplication” will continue to be supported for MySQL 5.5 but without support for the new features listed in this document.

Manual Failover from Replication Master

It will be possible for a user to cause a slave to become the new master for replication by executing a trove command. For the V2 release of replication, no facility for detecting a master failure condition will be provided.

To assist the user in minimizing data loss, there will be two different ways for the user to cause a slave to be promoted to master. If the user wishes to promote a slave to replace a master which is healthy and reachable, they will execute a new “promote_to_replica_source” function against a slave to promote it in place of the existing master; this function will coordinate with the master site to ensure that no data is lost. If a master site is unreachable, the user will use the “eject_replica_source” function to remove that instance from the replication set and the replication strategy will choose the slave with the most recent updates to promote to master; this operation may result in the loss of any transactions that were committed at the master site but not replicated to any of the slaves. Trove will not allow a reachable master site to be deleted as that would unnecessarily result in lost data.

There will be no accomodation made to allow users or operators to “fix” slaves which have gotten out of sync with the master site. Instead, every effort will be made to configure replication so that the slave will not fall out of sync with the master. The following MySQL options will be set to ensure safe replication:

Master Options

Binary logs will be configured for MIXED mode logging. This will allow statement based replication where it is safe to do so, and row based replication will be used where necessary.
The enforce_gtid_consistency option will be used to prevent statements which will conflict with the use of GTID replication.
When the Percona database is being used, the Percona enforce_storage_engine option will be used to restrict replication to the InnoDB storage engine. This is to prevent the use of MyISAM tables which could be corrupted during a crash recovery.

Slave Options

Slave will execute in READ_ONLY mode to avoid transaction conflicts between master and slave. By default, users are not given root access to the database; if they choose to enable root access, they are assumed to be sufficiently advanced as to not execute operations on a slave which will disturb replication.
The slaves’ relay log will be stored in a table in the database to provide transactional consistency between the statements executed against the database and the recording of the slave’s position in executing the relay log.
Relay log recovery will be turned on to cause relay log recovery during mysql startup. relay_log_purge will be enabled in support for relay_log_recovery.

Promotion of Slave to Master

The user may select a slave to be promoted to be the new master of a replication set. This operation would consist of the following steps:

Contact each slave, abort operation if any not reachable
Make the old master read-only
Detach old master’s public IP
Detach master candidate’s public IP
Record latest GTID of master
For each slave (including master candidate)
- Wait for slave to receive/apply master’s latest GTID
Set master candidate as replication master site
For each remaining slave
- Make instance slave of new master
Make old master be slave of new master
Assign master candidate’s IP to old master (which is now slave)
Make new master writable
Assign old master’s public IP to new master

Promote to Master API

To replace a healthy master site, the promote_to_replica_source API call will be added to the client and taskmanager APIs.

Ejection of Master Site

If a replication master site is out of service, the user may choose to “eject” the instance from the replication set. Ejecting an unreachable instance which is a master for replication would result in one of its slaves being chosen to be promoted to be the new master site, and a new slave generated to fill out the replication set. The ejected master will be available for examination, but will no longer participate in replication. This operation would consist of the following steps:

Abort operation if the master site can be contacted
Contact each slave, abort operation if any not reachable
Detach master’s public IP
Record master’s Region/Zone
Select master candidate (see Master Candidate Selection)
Switch the master candidate from slave to master
For each remaining slave
- Connect slave to new master instance
Mark new master as writable
Attach master’s public IP to new master
Create new slave in same Region/Zone as old master
Assign master candidate’s public IP to new slave

Master Candidate Selection

When selecting a slave to be promoted to master to replace an unreachable master site, the algorithm for choosing the master candidate will be determined by the value of the MASTER_PROMOTION_STRATEGY configuration option of the Taskmanager Config (not datastore specific). The possible values for this option are outlined below:

Strategy	Description
MOST_RECENT	The slave with the highest GTID is chosen as the master candidate
PROXIMATE_AZ	The slave with the highest GTID in the same Availability Zone as the old master is chosen
PROXIMATE_REGION	The slave with the highest GTID in the same Region as the old master is chosen

The PROXIMATE_REGION setting will be the default as this will ensure that the new master site will be in the same region as the old master; for the Kilo release, this will be equivalent to the MOST_RECENT option (and may be implemented as such) as Region support is not implemented in Trove.

Incremental Snapshots

To improve the performance of slave creation, the default action will be to take the most recent backup (full or incremental) and create an incremental backup to be used for the replication snapshot. If no previous backup can be found, a full backup will be created to include in the replication snapshot. Should the “backup” option be specified in addition to the “replica_of” option, an incremental backup will be performed from the indicated backup.

Multiple Slave Creation

A replica_count option will be added to support the creation of multiple slaves from a single replication snapshot.

a replica_count option will be added to the trove create command
a replica_count parameter will be added to the create_instance taskmanager ReST API
the taskmanager FreshInstanceTasks.create_instance method will iteratively create the specified number of slaves from a single replication snapshot (the implementor is free to implement slave creation in parallel if time permits, and should investigate doing so, but it is not a requirement for V2)

Configuration

MysqlGTIDReplicationStrategy value added to ReplicationStategy option for MySQL configuration.

New configuration option master_promotion_strategy added to MySQL configuration with values as above.

Database

No database impacts are envisioned.

Public API

Promote to Replica Source

A new action will be added to the Trove REST API to allow a replica to be promoted to be the master of its replication set:

POST http://127.0.0.1:8779/v1.0/<tenant id>/instance/<instance id>/action
{
    *"promote_to_replica_source": null*
}

RESP: [200]
  {
      'date': '<date>',
      'content-length': '<RESP BODY len>',
      'content-type': 'application/json'
  }
RESP BODY:
  {
      "instance": {
          *"status": "PROMOTE",*
          "updated": "2014-11-25T21:25:11",
          "name": "m",
          "links": [
              {
                  "href": "https:\/\/10.40.10.178:8779\/v1.0\/...\/instances\/...",
                  "rel": "self"
              },
              {
                  "href": "https:\/\/10.40.10.178:8779\/instances\/...",
                  "rel": "bookmark"
              }
          ],
          "created": "2014-11-25T21:25:06",
          "ip": [
              "10.0.0.2"
          ],
          "replicas": [
              {
                  "id": "8e5710df-ef39-4201-a059-764d9091f079",
                  "links": [
                      {
                          "href": "https:\/\/10.40.10.178:8779\/v1.0\/...\/instances\/...",
                          "rel": "self"
                      },
                      {
                          "href": "https:\/\/10.40.10.178:8779\/instances\/...",
                          "rel": "bookmark"
                      }
                  ]
              }
          ],
          "id": "fff6d8c5-9d05-4a00-ab58-d8954ec945a3",
          "volume": {
              "used": 0.13,
              "size": 1
          },
          "flavor": {
              "id": "7",
              "links": [
                  {
                      "href": "https:\/\/10.40.10.178:8779\/v1.0\/...\/flavors\/7",
                      "rel": "self"
                  },
                  {
                      "href": "https:\/\/10.40.10.178:8779\/flavors\/7",
                      "rel": "bookmark"
                  }
              ]
          },
          "datastore": {
              "version": "5.5",
              "type": "mysql"
          }
      }
  }

A new CLI command will be added to invoke the promote_to_replica_source API:

trove promote-to-replica-source <replica id>

Eject Replica Source

A new action will be added to the Trove REST API to allow a replica source to be ejected from a replication set:

POST http://127.0.0.1:8779/v1.0/<tenant id>/instance/<instance id>/action
{
    *"eject_replica_source": null*
}

RESP: [200]
  {
      'date': '<date>',
      'content-length': '<RESP BODY len>',
      'content-type': 'application/json'
  }
RESP BODY:
  {
      "instance": {
          *"status": "EJECT",*
          "updated": "2014-11-25T21:25:11",
          "name": "m",
          "links": [
              {
                  "href": "https:\/\/10.40.10.178:8779\/v1.0\/...\/instances\/...",
                  "rel": "self"
              },
              {
                  "href": "https:\/\/10.40.10.178:8779\/instances\/...",
                  "rel": "bookmark"
              }
          ],
          "created": "2014-11-25T21:25:06",
          "ip": [
              "10.0.0.2"
          ],
          "replicas": [
              {
                  "id": "8e5710df-ef39-4201-a059-764d9091f079",
                  "links": [
                      {
                          "href": "https:\/\/10.40.10.178:8779\/v1.0\/...\/instances\/...",
                          "rel": "self"
                      },
                      {
                          "href": "https:\/\/10.40.10.178:8779\/instances\/...",
                          "rel": "bookmark"
                      }
                  ]
              }
          ],
          "id": "fff6d8c5-9d05-4a00-ab58-d8954ec945a3",
          "volume": {
              "used": 0.13,
              "size": 1
          },
          "flavor": {
              "id": "7",
              "links": [
                  {
                      "href": "https:\/\/10.40.10.178:8779\/v1.0\/...\/flavors\/7",
                      "rel": "self"
                  },
                  {
                      "href": "https:\/\/10.40.10.178:8779\/flavors\/7",
                      "rel": "bookmark"
                  }
              ]
          },
          "datastore": {
              "version": "5.5",
              "type": "mysql"
          }
      }
  }

A new CLI command will be added to invoke the eject_replica_source API:

trove eject-replica-source <replica source id>

Trove Create Replica Count

The Trove REST API for the create instance operation will be augmented with a new field replica_count to specify the number of replicas to be created from the indicated instance:

POST http://127.0.0.1:8779/v1.0/<tenant id>/instances
{
    "instance": {
        "volume": {"size": 1},
        "flavorRef": "7",
        "name": "s",
        "replica_of": "<master id>",
        *"replica_count": "<n>"*
    }
}

RESP *unchanged*

An option will be added to the “trove create” CLI command to specify the new replica count option:

trove create <name> <flavor id> --replica_count=<count> ...

Internal API

promote_to_replica_source method added to taskmanager API. eject_replica_source method added to taskmanager API.

Guest Agent

The implementation of this feature set will result in many additions to the MySQL guest agent. There should be minimal impact to pre-existing code, and there is not expected to be any impact on backward compatibility of the APIs.

Alternatives

None.

Implementation

Assignee(s)

Primary assignee:: vgnbkr
Secondary assignee:: peterstac

Milestones

Target Milestone for completion:: Kilo-2

Work Items

Work Item	Assignee	Scheduled Release
GTID Support	Morgan	Kilo-3
Failover	Morgan	Kilo-3
Slave Count	Peter	Kilo-3
Incremental Snapshots	Peter	Kilo-3

Dependencies

n/a

Testing

The existing int-tests are believed to be sufficient for testing the GTID replication changes, as there are no functionality changes, just implementation changes.

New Int-Tests:

Promote to Master Positive

Create a new replication set of two sites. Attach floating ip addresses to each instance. Execute the promote_to_replica_source API call and verify that the master/slave relationships are correctly changed, and that the floating ip addresses maintain their affinity to master and slave.

Promote to Master Negative

Create a new replication set of two sites. Execute “service mysql stop” on the master site. Verify that promote_to_replica_source cannot be executed against the slave site.

Delete Master Positive

Create a new replication set of two sites. Attach floating ip addresses to each instance. Execute “service mysql stop” on the master to simulate the master site crashing. Execute the delete API call against the master site. Ensure that the slave has been promoted to master, a new slave has been added, and that the floating ip addresses have been moved appropriately.

Replica Count

No int-test will be done for this feature due to the resource requirements

Incremental Snapshots

No int-test will be done for this feature as there is no way to verify that the restore was actually done from an incremental backup rather than a full backup

Documentation Impact

User Guide

add section explaining manual failover, both via promote-to-replica-source and via deletion of a failed master
section on replication should be updated to document replica_count option to “trove create”

CLI Reference

add promote-to-replica-source command
add eject-replica-source command
update create command with replica_count

References

https://etherpad.openstack.org/p/kilo-summit-trove-replication-v2

Make Rsync for Guest Optional

Tue, 24 Feb 2015 00:00:00

Blueprint:

https://blueprints.launchpad.net/trove-integration/+spec/rsync-optional

Today, the instance rsyncs the guestagent code and trove-guestagent.conf via http://git.io/qI9ivw (or http://git.io/p88Njw)

The proposal is to introduce an alternative that does not require guest-to-controller SSH connectivity: bake the guestagent code and trove-guestagent.conf into the image.

Problem Description

In production, permitting SSH connectivity between guests and the control-plane is a security no-no. Although trove-integration is considered to be only a sample reference implementation, we owe it to deployers to provide insight into how to properly secure Trove.

Use Cases

As a deployer, I want to avoid ssh connectivity between guests and the control-plane.

Proposed Change

Add additional elements in trove-integration to stage the guestagent code and trove-guestagent.conf during the extra-data.d hook, and subsequently install them in the install.d hook, vs. relying on upstart/systemd to rsync.

See https://review.openstack.org/#/c/119488/

This is not turned on by default, and therefore is backwards compatible.

Configuration

To make use of this functionality, it requires setting GUEST_LOCAL_TROVE_DIR and GUEST_LOCAL_TROVE_CONF. The aforementioned values are used in the newly introduced diskimage-builder elements.

Database

No database changes.

Public API

No public API changes.

Public API Security

No public API Security related changes.

Internal API

No internal API changes.

Guest Agent

No Guest Agent changes.

Alternatives

No alternatives.

Implementation

Assignee(s)

Primary assignee:: Auston McReynolds (amcrn)

Milestones

Kilo-1

Work Items

See https://review.openstack.org/#/c/119488/

Dependencies

No dependencies.

Testing

diskimage-builder element additions/changes are not tested via traditional means at the moment.

Documentation Impact

No documentation impact.

References

None.

Switch to OSLO Messaging

Tue, 24 Feb 2015 00:00:00

https://wiki.openstack.org/wiki/Trove-rpc-versioning

This blueprint is to document the project to switch trove to use OSLO Messaging.

Problem Description

Get Trove on the same page w/ other projects + oslo. This will require refactoring the rpc layer

Prevent backward incompatibility between Trove components

The rpc client can send an older version of a message/call to a newer message handler without issue.
The rpc client can send a newer version of a message/call to an older message handler without killing it (need to confirm this).
The rpc client can check to see if it can send a newer version of a message/call.
Reduce the need for downtime during deployments.

** Effectively allows for a mix of versions between the control plane: and guest agents (in most cases). There may still be updates that will require down time.

Proposed Change

Remove rpc common components from oslo incubator?
Use oslo messaging library

** The minimum version will be picked from global-requirements. At the: time of this writing, it’s >= 1.4.0

Keep track of a “version history” in comments in the code
Update trove calls to the openstack.common.rpc client to include a version cap param. (This is already supported in the client)

** Trove API –> Task Manager ** Trove API –> Guest Agent ** Guest Agent –> Conductor ** Task Manager –> Conductor ** Document the use cases and examples of how to add/modify API calls

Configuration

oslo.messaging has been designed to be backwards compatible config-wise. All the config compatibilities will be kept until all the projects have been migrated to oslo.messaging.

Database

Public API

Public API Security

Internal API

Guest Agent

Alternatives

None

Implementation

Assignee(s)

Primary assignee:: sgotliv

Milestones

Kilo-1

Work Items

see above.

Dependencies

None

Testing

Additional tests will be proposed if required, existing tests amended.

Documentation Impact

Yes

References

See https://review.openstack.org/#/c/94484/

Cassandra Backup & Restore

Tue, 24 Feb 2015 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/trove/+spec/cassandra-backup-restore

Problem Description

The Cassandra datastore currently does not support any backup/restore strategy.

Proposed Change

The patch set will implement full backup/restore of a single instance using the Nodetool 1 utility for Cassandra 2.1 3.

Configuration

The following Cassandra configuration options will be updated:

backup/restore namespaces
backup_strategy

Database

None

Public API

None

Public API Security

None

Python API

None

CLI (python-troveclient)

The following Trove CLI commands (upon completion) will be fully functional with Cassandra:

backup-create

backup-delete

backup-list

create –backup

Internal API

None

Guest Agent

We are implementing full backup using node snapshots following the procedure outlined in the Nodetool manual 2. Nodetool can take a snapshot of one or more keyspace(s). A snapshot can be restored by moving all *.db files from a snapshot directory to the respective keyspace overwriting any existing files.

When a snapshot is taken Cassandra starts saving all changes into new data files keeping the old ones at the same state as when the snapshot was taken. The data storage must have enough capacity to accommodate the backlog of all changes throughout the duration of the backup operation until the snapshots get cleaned up.

Backups are streamed to and from a remote storage as (TAR) archives. We now outline the general procedure for creating and restoring such an archive.

Unique backup IDs will be used for snapshot names, to avoid collisions between concurrent backups.

The Backup Procedure:

Make sure the database is up an running.
Clear any existing snapshots (nodetool clearsnapshot) with the same name as the created one.
Take a snapshot of all keyspaces (nodetool snapshot).
Collect all *.db files from the snapshot directories.
Package the snapshot files into a single TAR archive (compressing and/or encrypting as required) while streaming the output to Swift storage under the database_backups container.

Transform the paths such that the backup can be restored simply by extracting the archive right to an existing data directory. This is to ensure we can always restore an old backup even if the standard guest agent data directory changes.
Clear the created snapshots as in (1).

The Restore Procedure:

Stop the database if running and clear any files generated in the system keyspace.
Create a new data directory.
Read backup from storage unpacking it to the data directory.
Update ownership of the restored files to the Cassandra user.

Additional Considerations:

Instances are created as single-node clusters. A restored instance should therefore belong to its own cluster as well. The original cluster name property has to be reset to match the new unique ID of the restored instance. This is to ensure that the restored instance is a part of a new single-node cluster rather than forming a one with the original node or other instances restored from the same backup. Cluster name is stored in the database and is required to match the configuration value. Cassandra fails to start otherwise.

A general ‘cluster_name’ reset procedure is:

Update the name in the system keyspace table.
Update the name in the configuration file.
Restart the service.

The ‘superuser’ (“root”) password stored in the system keyspace needs to be reset before we can start up with restored data.

A general password reset procedure is:

Disable user authentication and remote access.
Restart the service.
Update the password in the ‘system_auth.credentials’ table.
Re-enable authentication and make the host reachable.
Restart the service.

Alternatives

None

Implementation

Assignee(s)

Petr Malik <pmalik@tesora.com>

Milestones

Liberty-1

Work Items

Implement functionality needed for resetting cluster name and superuser password.
Implement backup/restore API calls.

Upgrade Implications

None

Dependencies

The patch set will be building on functionality implemented in blueprints: cassandra-database-user-functions 4 and cassandra-configuration-groups 5

Testing

Unittests will be added to validate implemented functions and non-trivial codepaths. We do not implement functional tests as a part of this patch set.

Documentation Impact

The datastore documentation should be updated to reflect the enabled features. Also note the new configuration options - backup/restore namespaces and backup_strategy for Cassandra datastore.

References

1: Documentation on Nodetool utility for Cassandra 2.1: http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsNodetool_r.html
2: Manual on Backup and Restore for Cassandra 2.1: http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_backup_restore_c.html
3: Documentation on Cassandra 2.1: http://docs.datastax.com/en/cassandra/2.1/cassandra/gettingStartedCassandraIntro.html
4: Database and User Functions for Cassandra: https://blueprints.launchpad.net/trove/+spec/cassandra-database-user-functions
5: Configuration Groups for Cassandra: https://blueprints.launchpad.net/trove/+spec/cassandra-configuration-groups

MongoDB Configuration Groups

Tue, 24 Feb 2015 00:00:00

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/mongodb-configuration-groups

Problem Description

The MongoDB guestagent currently does not support configuration groups.

Proposed Change

The patch set will implement configuration groups for MongoDB 2.6 and above.

MongoDB 2.6 and above stores its configuration 1 in a YAML ‘mongod.conf’ file. The database service has to be restarted for any changes to the configuration file to take effect. All configuration changes will therefore be requiring database restart and ‘apply_overrides’ will be implemented as no-op.

Overrides will be implemented by replacing the current file with an updated one. The old file will be backed up in the same directory (as *.old) and restored on configuration reset.

The platform-default configuration file will be used as a base for our configuration template. Guest agent interfaces exposing the configuration properties will be made available to other modules such as backup and replication.

Most configuration properties will be available via configuration groups. Some, however, do not make sense in the Trove context.

These would include:

irrelevant options (like automatic snapshots, since the Trove user cannot retrieve them)

guestagent specific (e.g. file paths, passwords)

items that Trove needs to control (replication/clustering properties)

See Available Configuration Properties for the full list of supported options.

The user should be able to specify configurations properties as standard Python YAML objects - key-value pairs and dicts.

Available Configuration Properties

Keys not included in the lists are kept at their default values and are not configurable via Trove. See 1 for more details and default values.

The properties configurable by the user via the Trove API:

systemLog.verbosity

systemLog.component.accessControl.verbosity

systemLog.component.command.verbosity

systemLog.component.control.verbosity

systemLog.component.geo.verbosity

systemLog.component.index.verbosity

systemLog.component.network.verbosity

systemLog.component.query.verbosity

systemLog.component.replication.verbosity

systemLog.component.sharding.verbosity

systemLog.component.storage.verbosity

systemLog.component.storage.journal.verbosity

systemLog.component.write.verbosity

systemLog.quiet

systemLog.traceAllExceptions

systemLog.logAppend

systemLog.logRotate

systemLog.timeStampFormat

net.maxIncomingConnections

net.wireObjectCheck

net.unixDomainSocket.enabled

net.ipv6

net.http.enabled

net.http.JSONPEnabled

net.http.RESTInterfaceEnabled

security.sasl.hostName

security.sasl.serviceName

security.sasl.saslauthdSocketPath

security.javascriptEnabled

operationProfiling.slowOpThresholdMs

operationProfiling.mode

storage.indexBuildRetry

storage.journal.enabled

storage.directoryPerDB

storage.syncPeriodSecs

storage.engine

storage.mmapv1.nsSize

storage.mmapv1.quota.enforced

storage.mmapv1.quota.maxFilesPerDB

storage.mmapv1.smallFiles

storage.mmapv1.journal.debugFlags

storage.mmapv1.journal.commitIntervalMs

storage.wiredTiger.engineConfig.cacheSizeGB

storage.wiredTiger.engineConfig.statisticsLogDelaySecs

storage.wiredTiger.engineConfig.journalCompressor

storage.wiredTiger.engineConfig.directoryForIndexes

storage.wiredTiger.collectionConfig.blockCompressor

storage.wiredTiger.indexConfig.prefixCompression

replication.oplogSizeMB

replication.secondaryIndexPrefetch

sharding.clusterRole

auditLog.format

auditLog.filter

snmp.subagent

snmp.master

replication.localPingThresholdMs

sharding.autoSplit

sharding.configDB

sharding.chunkSize

setParameter

Non-configurable properties with updated default values:

systemLog.path (controlled-by guestagent)

systemLog.destination file

processManagement.pidFilePath (controlled-by guestagent)

processManagement.fork True

security.keyFile (controlled-by replication)

security.clusterAuthMode (keyFile)

security.authorization True

storage.dbPath (controlled-by guestagent)

replication.replSetName (controlled-by replication)

sharding.archiveMovedChunks False

auditLog.destination file

auditLog.path (controlled-by guestagent)

Configuration

None

Database

None

Public API

None

Public API Security

None

Python API

None (empty section added after merging)

CLI (python-troveclient)

None (empty section added after merging)

Internal API

CLI (python-troveclient)

This work will enable the following client commands:

configuration-attach

configuration-create

configuration-default

configuration-delete

configuration-detach

configuration-instances

configuration-list

configuration-parameter-list

configuration-parameter-show

configuration-patch

configuration-show

configuration-update

Guest Agent

Update facilities for handling of YAML config files in the operating_system module.
Implement API to update_overrides in the manager and service modules.
The current configuration template will be updated to the default version for the target platform with changes noted in Available Configuration Properties.

The following existing files will be updated:

guestagent/datastore/experimental/mongodb/manager.py
guestagent/datastore/experimental/mongodb/service.py
templates/mongodb/config.template

Alternatives

None

Implementation

Assignee(s)

Petr Malik <pmalik@tesora.com>

Milestones

Liberty

Work Items

Implement functionality to handle (read/write/update) MongoDB YAML configuration files.

Implement configuration-related manager API calls.

def update_overrides(self, context, overrides, remove=False)
def apply_overrides(self, context, overrides) [no-op]

Upgrade Implications

None

Dependencies

None

Testing

Unit tests will be added to validate implemented functions and non-trivial codepaths. Relevant integration tests will be added.

Documentation Impact

The datastore documentation should be updated to reflect the enabled features.

References

1(1,2): Documentation on MongoDB configuration: http://docs.mongodb.org/manual/reference/configuration-options/

MongoDB database management

Tue, 24 Feb 2015 00:00:00

https://blueprints.launchpad.net/trove/+spec/mongodb-database

Enable MongoDB database management functionality.

Problem Description

The MongoDB datastore does not support database management features. Allowing the user to create, list, and delete databases through the API is essential.

Proposed Change

Three standard Trove commands will be enabled for MongoDB:

database-create
database-list
database-delete

The changes will be confined to the guestagent code. The taskmanager, API, and conductor do not require any code changes. The code changes will be to implement a new class service.MongoDBAdmin and the corresponding methods as members of the class. The methods of manager.Manager will be updated to call the admin functions.

Calls to MongoDB will be done in Python via the PyMongo library, which is required to be pre-installed on the guest.

Configuration

No changes will be made to any configuration files.

Database

No new items will be added here.

Public API

No API changes.

Public API Security

No API Security changes.

Python API

None (empty section added after merging)

CLI (python-troveclient)

None (empty section added after merging)

Internal API

No Internal API changes.

Guest Agent

Modified files:

trove/guestagent/db/models.py - add a MongoDBSchema class.
trove/guestagent/datastore/experimental/mongodb/manager.py - enable functions.
trove/guestagent/datastore/experimental/mongodb/service.py - add functions.

The Guest Agent will be changed to support the following manager functions:

create_database - MongoDB does not have a method to explicitly create a database. The database to use is specified via ‘use <dbname>’, but this does not create the database. Databases are created when a ‘document’ is first inserted into them. To ensure a database is created, a dummy document will be inserted but then deleted.
list_databases - Run ‘pymongo.MongoClient.database_names()’ and return the resulting list.
delete_database - Drop the database with ‘pymongo.MongoClient.drop_database(“<dbname>”)’. Users associated with the database will have to be removed manually.

Alternatives

create_database could not create a dummy object in the database.

Implementation

Assignee(s)

Matthew Van Dijk

Milestones

liberty-1

Work Items

The changes will be implemented in a single commit. The scope is small and the functionality is linked.

Upgrade Implications

There will be no upgrade implications.

Dependencies

There are no dependencies on other work in progress.

Testing

Unit tests will be added to validate non-trivial code paths. Integration tests may be added as needed.

Documentation Impact

The MongoDB datastore documentation can be updated to reflect the enabled features.

References

There are no external references in this document.

MongoDB users management

Tue, 24 Feb 2015 00:00:00

https://blueprints.launchpad.net/trove/+spec/mongodb-users

Enable MongoDB users management functionality.

Problem Description

The MongoDB guest agent does not support users management features. Allowing the user to create, list, and delete users, enable/check root access, and grant/revoke user access through the API is essential.

MongoDB users are unique to each database. A user is identified by both its name and database. Therefore the Trove actions requiring user names need the value to be in the following string format: “<database>.<username>”. MongoDB does not allow the following characters in the database name: /. “$

Proposed Change

Calls to MongoDB will be done in Python via the PyMongo library, which is required to be pre-installed on the guest.

Features

A number of Trove commands will be enabled for the MongoDB datastore. Also, MongoDB’s authentication protocol will be enabled by default, making the database secure.

Functions

Several standard Trove commands will be enabled for MongoDB. Below are the corresponding CLI commands with all allowed arguments:

user-create <instance> <database>.<name> <password>
user-list <instance>
user-delete <instance> <database>.<name>
user-show <instance> <database>.<name>
root-enable <instance>
root-show <instance>
user-grant-access <instance> <database>.<name> <access_databases>
user-revoke-access <instance> <database>.<name> <access_database>
user-show-access <instance> <database>.<name>

The code changes will be to implement a new class service.MongoAdmin and the corresponding methods as members of the class. The methods of manager.Manager will be updated to call the admin functions.

Security

Currently the MongoDB datastore creates an insecure database. To make the above functionality useful the guest agent’s MongoDB server will be made secure. This will be done by enabling authentication and creating a Trove user ‘os_admin’ on the server by default. The Trove user will have user and database administrator privileges, but not read/write. When the guest is first started the guest agent will generate a password and connect (because of the localhost exception) to the server. It will then run the following (below is Mongo shell code, but it will be implemented using PyMongo):

use admin
db.createUser(
    {
        user: "os_admin",
        pwd: "<generated_password>",
        roles: [
            { role: "userAdminAnyDatabase", db: "admin" },
            { role: "dbAdminAnyDatabase", db: "admin" },
            { role: "cluserAdmin", db: "admin" }
        ]
    }
)

Authentication will be enabled by default. To disable it a configuration group can be used with the option {‘auth’: ‘false’}.

Note: enabling authentication on MongoDB effects clustering as shards are required to use a shared key file to connect to each other. Therefore a new cryptographically secure key will be generated during cluster-create to be used by the shards. This key value will be pushed to each shard, where it will be stored for connection use. The key will not be stored on the controller. The controller will ask an existing cluster guest for the key when adding new shards.

Configuration

The mongodb configuration file will have ‘auth’ set to ‘on’ to enable authentication on the server.

A new MongoDB RC script will be created at ~/.mongorc.js. This file will store the username and generated password, allowing the local MongoDB client to automatically connect using those credentials.

Database

No new items will be added here.

Public API

No API changes.

Public API Security

No API Security changes.

Python API

None (empty section added after merging)

CLI (python-troveclient)

None (empty section added after merging)

Internal API

No Internal API changes.

Guest Agent

Modified files:

trove/guestagent/db/models.py - add a MongoDBUser class.
trove/guestagent/datastore/experimental/mongodb/manager.py - enable functions.
trove/guestagent/datastore/experimental/mongodb/service.py - add functions.
trove/guestagent/datastore/experimental/mongodb/system.py - store system constants.

The Guest Agent will be changed to support the following manager functions:

create user - using ‘createUser()’
list users - query the admin database’s system.users collection
delete user - using ‘dropUser()’
show user - using ‘getUser()’
enable root - create user “root” and grant the role “root”
check if root is enabled - check if user “root” exists
grant user access to a database - using ‘updateUser()’
revoke user access to a database - using ‘updateUser()’
show user access to a database - using ‘getUser()’

Alternatives

Grant os_admin “root” role for full access.

Implementation

Assignee(s)

Matthew Van Dijk

Milestones

liberty-1

Work Items

The work will be split into four deliverables:

Enable authentication on server
create/list/show/delete users
enable/check root
grant/show/revoke access

Upgrade Implications

There will be no upgrade implications.

Dependencies

There are no dependencies on other work in progress.

Testing

Unit tests will be added to validate non-trivial codepaths. Integration tests may be added if necessary.

Documentation Impact

Documentation will be required to explain that authentication is enabled on MongoDB guests. The MongoDB datastore documentation can be updated to reflect the enabled features.

References

MongoDB manual

Redis Backup And Restore

Tue, 24 Feb 2015 00:00:00

Backup and Restore functionality needs to be added to the Redis datastore.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/redis-backup-restore

Problem Description

Trove instances created with the Redis datastore do not currently have a way to create a backup or to restore from one. This functionality needs to be added.

Proposed Change

Redis currently has two persistence strategies: RDB * and AOF † (both can be used concurrently). The backup strategy is the same for both, however restoring is a little different if AOF is enabled (whether or not RDB is also enabled).

Note: All necessary changes are anticipated to be made in the Trove guestagent: code.

*: Redis DataBase
†: Append-Only File

Backup

The redis-cli utility will be used to create the backup. This process is the same regardless of the persistence strategy configured within Redis. The process will be as follows 1:

Determine the directory where the backup files are written. This directory is stored in the redis configuration file (typically /etc/redis/redis.conf) under the dir keyword. At present this location is hard-coded in the redis config.template file, however it could be exposed in the future through configuration groups.
Find the time of the last Redis persistence run, and retain the information temporarily. This is accomplished by executing the LASTSAVE command through the redis-cli, and is necessary to determine when the next run completes (as explained below).
Start a backup using the redis-cli utility. This is done by executing the BGSAVE command. 2 Note: If Redis persistence is turned off (it is on by default in the config.template) then enough disk space must be available to write the backup file.
Wait until the backup completes. This is determined by polling using the LASTSAVE command as above, and waiting until the timestamp changes. 3
Compress/encrypt the backup
Stream the compressed/encrypted output to storage in Swift under the database_backups container. If persistence is turned off, the backup file can then be deleted.

Restore

Restoring a Redis server from backup depends on the persistence method that is running on the server. 4

To restore a Redis server from a backup 1:

Determine if AOF mode is enabled

If AOF is disabled:

Determine the location where backup files are located
Make sure the redis server is not running (stop if necessary)
Remove the existing dump.rdb file
Retrieve the backup from storage
Put the backup into the proper location
Change the ownership to redis:redis
Start the redis server

If AOF is enabled:

Determine the location where backup files are located
Make sure the redis server is not running (stop if necessary)
Remove existing dump.rdb and appendonly.aof
Retrieve the backup from storage
Put the backup into the proper location
Change the ownership to redis:redis
Disable AOF in the Redis configuration
Start the redis server
Create a new AOF file
Stop the Redis server
Turn on AOF in the Redis configuration
Start the Redis server

Configuration

The default values for the following config options will need to be updated:

backup_namespace
restore_namespace
backup_strategy

Database

None

Public API

None

Public API Security

None

Python API

Existing Python bindings are sufficient, and no changes are anticipated.

CLI (python-troveclient)

Once these changes are implemented, the following Trove CLI commands will now be fully functional with Redis:

backup-create

backup-delete

backup-list

backup-list-instance

backup-show

create –backup

Internal API

None

Guest Agent

The following files will need to be added to the guest agent, where the corresponding implementation will reside:

guestagent/strategies/backup/experimental/redis_impl.py
guestagent/strategies/restore/experimental/redis_impl.py

The following existing files will be updated:

guestagent/datastore/experimental/redis/manager.py

No backwards compatibility issues are anticipated.

Alternatives

None

Implementation

Assignee(s)

Primary assignee:: peterstac

Milestones

Target Milestone for completion:: Liberty-1

Work Items

The API calls for backup and restore need to be implemented.

Upgrade Implications

None

Dependencies

None

Testing

No new tests are deemed to be required (beyond the requisite unit tests). The int-tests group for Redis will be modified to run backup-related commands during integration test runs. It would be good if a 3rd party Redis CI could be set up to test Redis integration functionality, but at present this is not available.

Documentation Impact

Datastore specific documentation should be modified to indicate that backup and restore of a Redis Trove instance is now supported.

References

1(1,2): Backup and Restore procedure: http://zdk.blinkenshell.org/redis-backup-and-restore
2: Create Redis Backup: http://redis.io/commands/bgsave
3: How to tell when a Backup is finished: http://redis.io/commands/lastsave
4: Redis Persistence: http://redis.io/topics/persistence

Redis Configuration Groups

Tue, 24 Feb 2015 00:00:00

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/redis-configuration-groups

Problem Description

The Redis guestagent currently does not support configuration groups.

Proposed Change

The patch set will implement configuration groups for Redis 3.0 or above.

Redis stores its configuration 1 in ‘redis.conf’ file which can optionally import other files. If there are multiple configuration directives for a single property the last one is used. Configuration directives have key-value format: “keyword argument1 argument2 … argumentN”

Redis configuration will be reconfigured on the fly without stopping and restarting the service using the special CONFIG command 1.

Note: Not all directives are supported by the CONFIG commands and that modifying the configuration on the fly has no effects on the ‘redis.conf’ file.

The base configuration file for Redis 3.0 2 will be used as the default configuration template. Guest agent interfaces exposing the configuration properties will be made available to other modules such as backup and replication.

Most configuration properties will be available via configuration groups. Some, however, do not make sense in the Trove context.

These include:

irrelevant options (like automatic snapshots, since the Trove user cannot retrieve them)

guestagent specific (e.g. file paths, passwords)

items that Trove needs to control (replication/clustering properties)

See Available Configuration Properties for the full list of supported options.

The CONFIG directives will be renamed to a name known only to the guestagent for its internal use. This is to prevent a user from bypassing Trove configuration groups by changing properties in the client. Guestagent managed properties such as locations of database files and logs could be potentially used to hijack the NOVA instance.

Configuration overrides will be implemented using the in-file imports supported by the Redis configuration file format. User defined overrides if any, will be saved in ‘redis.conf.override’ and will be imported at the end of the default ‘redis.conf’ file.

Available Configuration Properties

Keys not included in the lists are kept at their default values and are not configurable via Trove. See 2 for more details and default values.

The properties configurable by the user via the Trove API:

tcp-backlog

timeout

tcp-keepalive

loglevel

databases

save

stop-writes-on-bgsave-error

rdbcompression

rdbchecksum

slave-serve-stale-data

slave-read-only

repl-diskless-sync

repl-diskless-sync-delay

repl-ping-slave-period

repl-timeout

repl-disable-tcp-nodelay

repl-backlog-size

repl-backlog-ttl

slave-priority

min-slaves-to-write

min-slaves-max-lag

requirepass

maxclients

maxmemory

maxmemory-policy

maxmemory-samples

appendonly

appendfsync

no-appendfsync-on-rewrite

auto-aof-rewrite-percentage

auto-aof-rewrite-min-size

aof-load-truncated

lua-time-limit

cluster-node-timeout

cluster-slave-validity-factor

cluster-migration-barrier

cluster-require-full-coverage

slowlog-log-slower-than

slowlog-max-len

latency-monitor-threshold

notify-keyspace-events

hash-max-ziplist-entries

hash-max-ziplist-value

list-max-ziplist-entries

list-max-ziplist-value

set-max-intset-entries

zset-max-ziplist-entries

zset-max-ziplist-value

hll-sparse-max-bytes

activerehashing

client-output-buffer-limit normal

client-output-buffer-limit slave

client-output-buffer-limit pubsub

hz

aof-rewrite-incremental-fsync

Non-configurable properties with updated default values:

daemonize yes

pidfile (controlled-by guestagent)

logfile (controlled-by guestagent)

dir (controlled-by guestagent)

slaveof (controlled-by replication)

masterauth (controlled-by guestagent)

cluster-enabled (controlled-by clustering)

cluster-config-file (controlled-by guestagent)

Configuration

None

Database

None

Public API

None

Public API Security

None

Python API

None (empty section added after merging)

CLI (python-troveclient)

None (empty section added after merging)

Internal API

CLI (python-troveclient)

This work will enable the following client commands:

configuration-attach

configuration-create

configuration-default

configuration-delete

configuration-detach

configuration-instances

configuration-list

configuration-parameter-list

configuration-parameter-show

configuration-patch

configuration-show

configuration-update

Guest Agent

Update facilities for handling of Redis config files in the operating_system module.
Implement update_overrides and apply_overrides in the manager and service modules.
The current configuration template will be updated to 2 with changes noted in Available Configuration Properties.

The following existing files will be updated:

guestagent/datastore/experimental/redis/manager.py
guestagent/datastore/experimental/redis/service.py
templates/redis/config.template

Alternatives

None

Implementation

Assignee(s)

Petr Malik <pmalik@tesora.com>

Milestones

Liberty

Work Items

Implement functionality to handle (read/write/update) Redis configuration files.

Implement configuration-related manager API calls:

def update_overrides(self, context, overrides, remove=False)
def apply_overrides(self, context, overrides)

Upgrade Implications

None

Dependencies

None

Testing

Unit tests will be added to validate implemented functions and non-trivial codepaths. Relevant integration tests will be added.

Documentation Impact

The datastore documentation should be updated to reflect the enabled features.

References

1(1,2): Documentation on Redis configuration: http://redis.io/topics/config
2(1,2,3): Self-documented configuration file for Redis 3.0: https://raw.githubusercontent.com/antirez/redis/3.0/redis.conf

Implement Vertica cluster provisioning

Wed, 18 Feb 2015 00:00:00

https://blueprints.launchpad.net/trove/+spec/implement-vertica-cluster

Problem description

Provide HP Vertica CE clusters through the Trove Clustering API.

Proposed change

Trove provides a well-designed framework for implementing clustering for different datastores. The Vertica datastore requires additional effort to accomplish clustering.

HP Vertica CE requires a minimum of 3 nodes to achieve fault tolerance.

Development plan:

API strategy, needed for handling clustering requests for Vertica.

Taskmanager strategy, needed to handle cluster provisioning for Vertica.

Guestagent strategy, needed to handle RPC API calls from Taskmanager to guest instances.

Unit and integration tests.

The cluster will return IPs of its nodes. This is done so that the cluster can be used for multi write and read.

The name of each instance will be automatically generated; it will be the <cluster_name>-member-<instance_num>.

There will be no grow/shrink functions to the cluster, since the CE edition provides for a maximum of 3-nodes and a 2-node HP Vertica Cluster is not fault-tolerable.

How to deploy HP Vertica Cluster

To bind multiple instances into a Vertica cluster, run the command “install_vertica” with each node IP. Next, run admintools to create a database on all included nodes in the cluster.

Step-by-Step guide for cluster provisioning:

Spin up N instances.

Once N instances are up and running, collect their IPs.

Enable password-less ssh between member instances for root and dbadmin users, by copying public key to authorized_keys.

Classify one of the nodes as first node(first-node is only logical to trove, Vertica has no preference as such.)

From first node, run install_vertica with all IPs from this node to stich all nodes into vertica cluster.

From first node, run admintools to create a database including all nodes in the cluster.

Configuration

Vertica datastore needs new configuration options to enable clustering:

cluster_support - describes the availability of clustering feature.
Type: Boolean

api_strategy - fully qualified class name of API strategy implementation.
Type: String

taskmanager_strategy - fully qualified class name of a Taskmanager implementation.
Type: String

guestagent_strategy - fully qualified class name of a Guestagent implementation.
Type: String

Database

This implementation will leverage the existing database schema for cluster.

Public API

The API request for creating/deleting cluster will remain unchanged. However, there will be subtle differences in the response. Vertica clustering will return IPs of all of its nodes. This will be used for user who want to scale read/write.

Create Cluster

Request:

POST /v1.0/<tenant_id>/clusters
{
  "cluster": {
    "name": "vertica-clstr",
    "datastore": {
      "type": "vertica",
      "version": "7.1"
    },
    "instances": [
      {
        "flavorRef": "2",
        "volume": {
          "size": 2
        }
      },
      {
        "flavorRef": "2",
        "volume": {
          "size": 2
        }
      },
      {
        "flavorRef": "2",
        "volume": {
          "size": 2
        }
      }
    ]
  }
}

Response:

{
  "cluster": {
    "id": "edaac9ca-b5e1-4028-adb7-fa7653e11224",
    "task": {
      "id": 2,
      "name": "BUILDING",
      "description": "Building the initial cluster."
    },
    "name": "vertica-clstr",
    "created": "2015-01-29T20:19:23",
    "updated": "2015-01-29T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "vertica",
      "version": "7.1"
    },
    "ip": [],
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "name": "vertica-clstr-member-1",
        "status": "BUILD",
        "ip": [],
        "links": [{...}],
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        }
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
        "name": "vertica-clstr-member-2",
        "status": "BUILD",
        "ip": [],
        "links": [{...}],
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        }
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
        "name": "vertica-clstr-member-3",
        "status": "BUILD",
        "ip": [],
        "links": [{...}],
        "flavor": {
          "id": "2",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        }
      }
    ]
  }
}

HTTP Codes:

- Accepted.
- BadRequest. Local storage not specified in flavor ID: <ID>.
- BadRequest. The number of instances for your cluster must be 3.
- BadRequest. A volume size is required for each instance in the cluster.
- BadRequest. A flavor is required for each instance in the cluster.
- Not Found. Flavor not found.

Show Cluster

Request:

GET /v1.0/<tenant_id>/clusters/edaac9ca-b5e1-4028-adb7-fa7653e11224

Response:

{
  "cluster": {
    "id": "edaac9ca-b5e1-4028-adb7-fa7653e11224",
    "task": {
      "id": 1,
      "name": "NONE",
      "description": "No tasks for the cluster."
    },
    "name": "vertica-clstr",
    "created": "2015-01-29T20:19:23",
    "updated": "2015-01-29T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "vertica",
      "version": "7.1"
    },
    "ip": ["10.0.0.1", "10.0.0.2", "10.0.0.3",],
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "name": "vertica-clstr-member-1",
        "status": "ACTIVE",
        "ip": ["10.0.0.1"],
        "links": [{...}],
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        },
      }
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
        "name": "vertica-clstr-member-2",
        "status": "ACTIVE",
        "links": [{...}],
        "flavor": {
        "ip": ["10.0.0.2"],
          "id": "7",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        },
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
        "name": "vertica-clstr-member-3",
        "status": "BUILD",
        "ip": ["10.0.0.3"],
        "links": [{...}],
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "volume": {
          "size": 2
        },
      }
    ]
  }
}

HTTP Codes:

200 - OK.
404 - Not Found. Cluster not found.

Show Instance

Request:

GET /v1.0/<tenant_id>/clusters/edaac9ca-b5e1-4028-adb7-fa7653e11224/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1

Response:

{
  "instance": {
    "status": "ACTIVE",
    "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
    "cluster_id": "edaac9ca-b5e1-4028-adb7-fa7653e11224",
    "name": "vertica-clstr-member-1",
    "created": "2015-01-29T20:19:23",
    "updated": "2015-01-29T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "vertica",
      "version": "7.1"
    },
    "ip": ["10.0.0.1"],
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 2,
      "used": 0.17
    }
  }
}

HTTP Codes:

- OK.
- Not Found. Cluster not found.
- Not Found. Instance not found.

List Clusters

Request:

GET /v1.0/<tenant_id>/clusters

Response:

{
  "clusters": [
    {
      "id": "edaac9ca-b5e1-4028-adb7-fa7653e11224",
      "task": {
        "id": 1,
        "name": "NONE",
        "description": "No tasks for the cluster."
      },
      "name": "vertica-clstr",
      "created": "2015-01-29T20:19:23",
      "updated": "2015-01-29T20:19:23",
      "links": [{...}],
      "datastore": {
        "type": "vertica",
        "version": "7.1"
      },
      "instances": [
        {
          "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
          "name": "vertica-clstr-member-1",
          "links": [{...}],
        }
        {
          "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2",
          "name": "vertica-clstr-member-2",
          "links": [{...}],
        },
        {
          "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b",
          "name": "vertica-clstr-member-3",
          "links": [{...}],
        }
      ]
    },
    ...
  ]
}

HTTP Codes:

200 - OK.

Delete Cluster

HTTP Method:

DELETE

Route:

/v1.0/<tenant_id>/clusters/<cluster_id>

Request:

{

}

Response:

{

}

HTTP codes:

202 - Accepted.

Cluster Actions

No cluster actions.

Internal API

From API service to Taskmanager service:

create_cluster:

checks if all instances are in BUILDING state.

designates one of the instances as first instance.

calls guest “get_keys” to receive public key from all member instances.

calls guest “authorize_keys” to register set of public keys to all member instances.

calls guest “install_cluster” to bind all nodes into a cluster with a database.

calls guest “cluster_complete” to all nodes to complete the activity.

checks if all instances are in ACTIVE state.

Guest Agent

To accomplish clustering for Vertica datastore more RPC APIs would be needed as per new guestagent-strategy.

Implementation

Assignee(s)

Primary assignee:

Milestones

“Kilo-3”

Work Items

Implement API strategy.
Implement Taskmanager strategy.
Implement Guestagent strategy.
Unit and integration tests

Dependencies

Single instance Vertica datastore to be merged first into Trove.

Testing

There will be unit tests that test every components in the strategies. There will be integration tests which will test cluster features.

Documentation Impact

The docs would need updates for:

Trove Capabilities for Vertica Clusters.

Vertica cluster creation methodolgy.

Modified responses of existing due to nature of HP Vertica clustering.

References

[1] http://my.vertica.com/docs/7.1.x/HTML/index.htm

Add Vertica datastore for Trove

Thu, 29 Jan 2015 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/trove/+spec/vertica-db-support

Problem Description

The aim of this blueprint is to enable Trove to support a new datastore type - HP Vertica 7.1 CE on Ubuntu.

Proposed Change

To add support for this new datastore, we need to implement the following:

Add a new diskimage-builder element for Vertica
Implement the various datastore features like:
```
- Launch
- Reboot
- Terminate
- Resize
```

Configuration

A new configuration group for Vertica and the different configuration options specific to Vertica in trove/common/cfg.py.

Some of the examples for the configuration options are:

- tcp_ports
- udp_ports
- backup_strategy
- backup_incremental_strategy
- replication_strategy
- backup_namespace
- restore_namespace
- root_on_create
- mount_point
- volume_support
- device_path

Database

None

Public API

None

Internal API

None

Guest Agent

This requires implementing the various datastore feature for Vertica like Launch, Reboot, Terminate, Resize.
This will include adding the following files specific to Vertica under the guestagent/datastore module:
```
- manager.py
- service.py
- system.py
```

These changes will not affect the behavior of the guestagent or its interaction with other components.

Disk-Image-Builder Elements

Disk-Image-Building would be a 2-step process, basically:

- Download the Vertica package.
- Execute the image-builder

CE is the free edition of HP Vertica database available at www.vertica.com
One needs to go through a free registration process, which enables a user to download the Vertica package.
Image-building elements would expect user to copy $VERTICA_PACKAGE_FILE at $VERTICA_SOURCE.

It would be then the job of DIB elements to:

- Copy $VERTICA_PACKAGE_FILE from $VERTICA_SOURCE and host in the guest-image.
- Install the essential requisite packages.
- Install the Vertica package to guest-image.
- Create the Vertica dba user named as "dbadmin", with desired group and path & time-zone settings setup to profile.
- Create a base location to host database files.

Implementation

Assignee(s)

Milestones

“Kilo-3”

Dependencies

None

Testing

Vertica datastore would be tested using a 3rd party CI hosted by HP.
New unit tests would be added for the Vertica guestagent.
Integration tests needs to be added for end-to-end feature testing:
```
- create/delete instance
- resize instance
```

Documentation Impact

Documentation would need the update on:

- HP Vertica being added as new datastore.
- Capabilities of Trove for HP Vertica datastore.
- How to build the guest-image required for hosting HP Vertica guest.

References

[1] https://my.vertica.com/docs/7.1.x/HTML/index.htm

[2] https://my.vertica.com/download-community-edition/#EE

[3] https://my.vertica.com/hp-vertica-community-edition-software-license-agreement/

Contributing to: trove-specs

Fri, 05 Dec 2014 00:00:00

If you would like to contribute to the development of OpenStack, you must follow the steps in this page:

http://docs.openstack.org/infra/manual/developers.html

Once those steps have been completed, changes to OpenStack should be submitted for review via the Gerrit tool, following the workflow documented at:

http://docs.openstack.org/infra/manual/developers.html#development-workflow

Pull requests submitted through GitHub will be ignored.

Bugs should be filed on Launchpad, not GitHub:

https://bugs.launchpad.net/trove

Move the Trove Guest Agent to its own module

Mon, 24 Nov 2014 00:00:00

https://blueprints.launchpad.net/trove/+spec/moving-trove-guestagent

Currently the guestagent code is part of the trove package. This blueprint’s goal is to refactor the guestagent and common functions into new top level packages, so that the guestagent can be deployed onto instances independently.

Note: Splitting the guestagent into its own repo was a requirement in the original specification. However, it was determined at the Kilo Mid Cycle that doing so would introduce dependencies between trove components that is difficult to maintain.

For example, commits for AMQP / API message changes between trove core and guestagent would need to be manually coordinated if trove and the guestagent were in separate repos.

Since the overhead of such coordination outweighs the benefit of splitting the guestagent into its own repo, we’ll proceed with the splitting at the directory level, but defer the repo splitting until further discussions.

Problem description

This blueprint will bring upon the following improvements:

Simplify installation of the guestagent on instances.
Reduce the amount of code that is deployed on instances.
Lower the memory footprint of the guestagent by not importing unnecessary code from Trove core.

Proposed change

Configuration

A new top level oslo.cfg for the troveguest will be added. (This should be a small subset of the values in trove.common.cfg)
The values in trove.common.cfg that are specifically for the guest will be removed.
It is possible that some configuration values will need to be added.

Database

None

Public API

None

Internal API

None

Guest Agent

1) The guestagent code will be moved to a top level module in the existing trove repository called ‘troveguest’.

2) The import statements will be changed from ‘trove.guestagent’ to ‘troveguest’

3) Imports for common functions will be updated. See the “Common Code” section below for details.

4) The code that currently rsyncs the code over to instances in development will be changed to just include the ‘troveguest’ and common function modules. (It is possible that another delivery method will be used, however that should probably be done in its own blueprint)

Common Code

The guestagent currently uses a large amount of common code in trove.common (/opt/stack/trove/trove/common) and trove.openstack.common (/opt/stack/trove/trove/openstack/common).

Code under the “trove.common” module will be moved up to a “common” module:

/opt/stack/trove/trove/common -> /opt/stack/trove/common

All import statements will be changed from ‘trove.common’to ‘common’

Code under the “trove.openstack.common” module will be moved up to a “common.openstack” module:

/opt/stack/trove/trove/openstack/common -> /opt/stack/trove/common/openstack

All import statements will be changed from ‘trove.openstack.common’to ‘common.openstack’

Splitting the common modules out of trove allows the guestagent and the common module to be deployed onto an instance independently. We’ll no longer need to deploy the whole trove code tree onto the guest instance just because the guestagent needs to use some common functions.

Alternatives

None

Implementation

Assignee(s)

Primary assignee:

Robert Myers (robertmyers)
Simon Chang (schang)

Milestones

Target Milestone for completion:: Kilo

Work Items

Stage 1

https://review.openstack.org/#/c/119425/

Add a new package to the [files] section of setup.cfg:

[files]
packages =
    trove
    troveguest

Move the trove/guestagent module up one level, and name it “troveguest”. All references of “trove.guestagent” will be changed to “troveguest”.

The new module layout will look like the following:

trove/
    ...
    doc/
    etc/
    tools/
    trove/
    troveguest/   < --- new module
    setup.py
    setup.cfg
    ...

Stage 2

Move the “trove.common” module up one level to “common”, and the “trove.openstack.common” into “common.openstack”. See the “Common Code” section for detail. Rename imports.

The new module layout will look like the following:

trove/
    ...
    common/   < --- moved from /opt/stack/trove/trove/common
    common/openstack   < --- /opt/stack/trove/trove/openstack/common
    doc/
    etc/
    tools/
    trove/
    troveguest/
    setup.py
    setup.cfg
    ...

Dependencies

None

Testing

The guestagent tests need to be split out from the trove.tests modules, then we need to make sure the tests are discovered properly.

Possibly modify tox.ini:

[testenv:cover]
basepython = python2.7
commands =
    {envpython} run_tests.py --group=does_not_exist
    coverage erase
    python setup.py testr --coverage
    coverage run -a run_tests.py
    coverage run -a troveguest/run_tests.py
    coverage html
    coverage report

Documentation Impact

Any docs or config file content that reference the old trove.guestagent and common module paths will need to be updated. For example:

The trove-guestagent section of this wiki: https://wiki.openstack.org/wiki/Trove
This sample config file: <trove_dir>/etc/trove/trove-guestagent.conf.sample

References

Kilo mid cycle discussion notes: https://etherpad.openstack.org/p/trove-kilo-sprint-blueprints-bugs

Apache CouchDB plugin for Trove

Tue, 11 Nov 2014 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/trove/+spec/couchdb-plugin-trove

Problem Description

The aim of this blueprint is to enable Trove to support a new datastore type - Apache CouchDB v1.6.1 on Ubuntu and Fedora, in addition to the other NoSQL databases supported by Trove.

Proposed Change

To add support for this new datastore, we need to implement the following: - Add a new diskimage-builder element for Apache CouchDB on Ubuntu and Fedora - Implement the various datastore features like:

- Launch
- Reboot
- Terminate
- Backup
- Restore
- Resize
- Replication

Configuration

A new configuration group for CouchDB and the different configuration options specific to CouchDB in /trove/common/cfg.py.

Some of the examples for the configuration options are:

- tcp_ports
- udp_ports
- backup_strategy
- mount_point
- usage_timeout
- volume_support
- device_path

Database

None

Public API

None

Internal API

None

Guest Agent

This requires implementing the various datastore feature for Apache CouchDB like Launch, Reboot, Terminate,Backup, Restore, Resize and Replication. This will include adding the following files specific to Apache CouchDB under the guestagent/datastore module:

- manager.py
- service.py
- system.py

In addition to this, there will also be a class under guestagent/strategies/backup to implement the backup and restore features and another class under guestagent/strategies/replication to implement the replication feature. CouchDB uses the replication interface to do backups hence we will only be implementing full backups for this release.

These changes wont affect the behavior of the guestagent or its interaction with other components.

Implementation

Assignee(s)

mariamj@us.ibm.com (Primary Assignee)
Susan Malaika (CouchDB Contact)

Milestones

Kilo

Dependencies

None

Testing

Add new unit tests for the CouchDB guestAgent

Add integration tests for end-to-end feature testing:

- create/delete instance
- create/restore backups
- replication

Documentation Impact

None

References

[1] http://guide.couchdb.org/draft [2] http://guide.couchdb.org/draft/api.html [3] http://guide.couchdb.org/draft/replication.html

DB2 Express-C guest agent for Trove

Tue, 11 Nov 2014 00:00:00

http://creativecommons.org/licenses/by/3.0/legalcode ..

Launchpad blueprint:

https://blueprints.launchpad.net/trove/+spec/db2-plugin-for-trove

Problem Description

The aim of this blueprint is to enable Trove to support a new datastore type - DB2, in addition to the other SQL databases supported by Trove. For the first release, we will be using DB2 Express-C v10.5.4(on Ubuntu) which is the free version of DB2 available for enterprises.

The following features will be implemented: Launch, Reboot, Terminate, Resize and Users.

Proposed Change

To add support for this new datastore, we need to implement the following:

Add a new diskimage-builder element for DB2 Express-C on Ubuntu
Implement the various datastore features like:
- Launch
- Reboot
- Terminate
- Resize
- Users

Configuration

A new configuration group for DB2 and the different configuration options specific to it will be defined in /trove/common/cfg.py.

Some of the examples for the configuration options are:

tcp_ports
udp_ports
backup_strategy
backup_incremental_strategy
mount_point
volume_support
device_path
backup_namespace
restore_namespace
cluster_support
replication_strategy

Database

None

Public API

None

Internal API

None

Guest Agent

This requires implementing the various datastore features(API) for DB2 like Launch, Reboot, Terminate, Backup, Restore, Resize and Users. This will include adding the following new files specific to DB2 under the guestagent module:

manager.py
service.py
system.py

These changes wont affect the behavior of the guestagent or its interaction with other components.

Disk-Image-Builder Elements

DB2 Express-C is the free version of IBM DB2 database available for download. Inorder to download the packages for DB2 Express-C, user needs to go through a free registration process.

Unlike other datastores supported by Trove, DB2 Express-C cannot be downloaded from a public repository. Hence the recommendation for creating DIB elements for DB2 Express-C is to have users go through the registration process and download the DB2 Express-C packages from the link provided in Reference Section [1]. The downloaded packages can be made available to Trove by storing it in a private repository or on the local filesystem. Create an extra-data.d element to then copy the package to the image. Use the environment variable, DATASTORE_PKG_LOCATION to specify the location of the package.

Implementation

Assignee(s)

mariamj@us.ibm.com (Primary Assignee)
Susan Malaika (DB2 Contact)

Milestones

Kilo

Dependencies

None

Testing

Add new unit tests for the DB2 Express-C guestAgent
Add integration tests for end-to-end feature testing:
- create/delete instance
- create/delete/list databases
- create/delete/list users

Documentation Impact

The DB2 Express-C packages can be downloaded from the link provided in Reference Section [1]. Click on the link “DB2 Express-C for Linux 64-bit”. New users can either get an IBM ID or click on the “Proceed without an IBM ID”. Users will have to register first inorder to download the packages. After downloading the packages, users can make it accessible to Trove for building guest images by:

storing it in a private repository and defining the variable ‘DATASTORE_PKG_LOCATION’ to be the base url of the private repository

storing it on the local filesystem in a directory accessible to the DB2 DIB element and setting the ‘DATASTORE_PKG_LOCATION’ to point to this directory.

For example, DATASTORE_PKG_LOCATION can be set as follows:

export DATASTORE_PKG_LOCATION=’/home/stack/db2’

export DATASTORE_PKG_LOCATION=’http://www.foo.com/pkgs/db2’

References

[1] DB2 Download Link: http://www-01.ibm.com/software/data/db2/express-c/download.html

trove-specs

Wed, 18 Jun 2014 00:00:00

OpenStack Database Service (Trove) Specifications

Free software: Apache license
Documentation: http://docs.openstack.org/developer/trove-specs

Features

TODO