Resources Management

Monitoring Drives

To monitor drives in your Object Storage system open the GUI > Resources > Drives

image23a

Viewing Drives Properties (VPSA Object Storage Admin)

The Drives details (properties and metering), are shown in the South Panel tabs:

Properties

Each Drive includes the following properties:

Property

Description

ID

An internally assigned unique ID

Name

Automatically assigned name.

Capacity

The Drive capacity in GiB

Storage Node

The Storage Node that contains the selected Drive

Virtual Controller

The virtual controller that owns the selected drives and performs IO operations on it

Storage Policy

The Storage Policy where the selected Drive belongs

Fault Domain

The Zadara cloud Fault Domain this Drive resides belongs to

Protection Zone

The Zadara cloud protection zone this drive is physically located at

Type

Drive type: SATA, SAS, SSD

UUID

The unique identifier of the drive

Status

  • Normal – All drives are in sync

  • Failed – The drive does not function

  • Absent – The drive does not exist

Added

Date & time when the drive was added

Modified

Date & time when the drive was last modified


Disk Metering

The Metering Charts provide live metering of the IO workload associated with the selected Drive.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, 1 hour, 1 day, 1 week. The Auto button lets you see continuously-update live metering info.

The following charts are displayed:

Chart

Description

IOPs

The number of read and write commands issued to the selected Drive per second

Bandwidth (MB/s)

Total throughput (in MB) of read and write commands issued to the selected Drive per second

Latency (ms)

Average response time of all read and write commands issued to the selected Drive per selected interval

image23


Backend Metering

The Metering Charts provide live metering of the IO workload on the selected Drive.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

Throughput (OP/s)

The number of operations (PUT/GET/DELETE) that were sent to the selected Drive per second

Bandwidth (MB/s)

Total throughput (in MB) of read and write commands that were sent to the selected Drive per second

Latency (ms)

Average response time of all operations (PUT/GET/DELETE) that were sent to the selected Drive per selected interval

image24


Monitoring Virtual Controllers

Virtual Controllers are Virtual Machines (VM) on the Zadara cloud that serves client operations on the Object Storage. For a full list of the VC responsibilities refer to Virtual Controller. Virtual Controllers are automatically created and added/removed to the Object Storage configuration, depending on the number of the allocated drives. There are minimum of two VCs in each VPSA Object Storage deployment: vc-0, vc-1. These VCs have role of Proxy+Storage. To improve performance you can add Proxy only VCs from the Zadara Provisioning Portal as described at Adding Proxy Virtual Controllers (VPSA Object Storage Admin).

Viewing VCs Properties (VPSA Object Storage Admin)

Properties

Each Virtual Controller has the following properties:

Property

Description

ID

An internally assigned unique ID

Storage Role

Proxy+Storage / Proxy-Only

Management Role

  • Ring Master – Runs the Object Storage Rings

  • Ring Slave – Standby to run the Object Storage Rings

  • VC – Regular VPSA Object Storage VC

Status

  • Created – VC is running normally

  • Failed – VC is not running

  • Passivating – VC is shutting down

  • Deleting – in the process of being removed from the cluster

Storage Node

The Storage Node hosting selected VC

Fault Domain

The Zadara cloud Fault Domain this VC resides belongs to

Protection Zone

The Zadara cloud protection zone this VC is allocated to

Frontend IP

The IPv4 or IPv6 address allocated to the VC

Backend IP

The VC IP address on the backend network that connects to the Drives

Load Balancer Group

(Optional) In case an ELB is enabled, represent the LB Group of the VC

Added

Date & time when the VC was added

Modified

Date & time when the VC was last modified

Drives

List the drives assigned to the selected Storage Policy.

System Usage

This chart shows the CPU utilization of the selected VC.

Backend Metering

The Metering Charts provide live metering of the IO workload at the backend of the selected VC.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

Throughput (OP/s)

The number of operations (PUT/GET/DELETE) issued to objects and handled by the selected VC per second

Bandwidth (MB/s)

Total throughput (in MB) of read and write commands issued by the selected VC per second

Latency (ms)

Average response time of all operations (PUT/GET/DELETE) issued to objects and handled by the selected VC per selected interval

image25


Account Service Metering

The Metering Charts provide live metering of the IO workload on the accounts database at the backend of the selected VC.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

Throughput (OP/s)

The number of operations (PUT/GET/DELETE) issued to the accounts database and handled by the selected VC per second

Bandwidth (MB/s)

Total throughput (in MB) of read and write commands issued by the selected VC to the accounts DB per second

Latency (ms)

Average response time of all operations (PUT/GET/DELETE) issued to the accounts database and handled by the selected VC per selected interval

image26


Container Service Metering

The Metering Charts provide live metering of the IO workload on the containers database at the backend of the selected VC .

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

Throughput (OP/s)

The number of operations (PUT/GET/DELETE) issued to containers and handled by the selected VC per second

Bandwidth (MB/s)

Total throughput (in MB) of operations (PUT/GET/DELETE) issued by the selected VC to containers per second

Latency (ms)

Average response time of all operations (PUT/GET/DELETE) issued to containers and handled by the selected VC per selected interval


Frontend Metering

The Metering Charts provide live metering of the IO workload at the frontend of the selected VC.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

Throughput (OP/s)

The number of operations (PUT/GET/DELETE) issued to objects and handled by the proxy of the selected VC per second

Bandwidth (MB/s)

Total throughput (in MB) of read and write commands issued to proxy of the selected VC per second

Latency (ms)

Average response time of all operations (PUT/GET/DELETE) issued to objects and handled by proxy of the selected VC per selected interval

image27


Load Balancer Groups (Optional)

Zadara Elastic Load Balancer can be enabled for the VPSA Object Storage in order to provide improved performance and load allocation across the VPSA Object Storage VCs. Once enabled, a dedicated VC will be added to the VPSA Object Storage as the Load Balancer Master. The Load Balancer Master VC will aggregate a bulk of up to 12 VCs from the same protection zone under the same VRID index.

From this view, a VPSA Object Storage administrator can review the utilization and distribution of each Load Balancer Group.

Property

Description

ID

An internally assigned unique ID

VRID

The VRRP VRID assigned to the Elastic Load Balancer Group

VC Master

The Load Balancer Group Active (Master) VC

IPv4/IPv6

The Load Balancer Group Active (Master) VC IP address

Protection Zone

The Zadara cloud protection zone

Added

Date & time when the LBG was created

Modified

Date & time when the LBG was last modified

Managing Storage Policies

Storage Policies provide a way for object storage providers to differentiate service levels, features and behaviors of a Object Storage deployment.

Policies can be thought of as a group of drives, with a redundancy level policy assigned to it.

Before placing object data into the VPSA Object Storage, users create a container which holds the listing of all objects stored under the container’s namespace. Users can select the Storage Policy that will be used when storing data objects under a container’s namespace when they create the container. All objects stored in a container will be placed according the configuration of the Storage Policy which was set when the container was created. If no policy was specified at the container creation time, the default policy is used.

To ensure availability of the VPSA object Storage data, the drives assigned to a Storage Policy are evenly distributed between Object Storage Fault Domains. The cloud administrator defines the Fault Domain of each Storage Node. The system makes sure to allocated drives across zones based on the Storage Policy type

Storage Policies allow some level of segmenting in terms of quality of service, within a single system for various purposes. Storage Policies allow objects to be stored based on the following criteria:

  • Quality of Service: By using different disk drives for different policies, tiers of storage performance can be created. For example, an SSD-only policy can be created used to implement a low-latency/high performance tier.

  • Number of Replicas: offer different protection levels in the same VPSA Object Storage cluster

    • 2 Way replication offers protection for one FD failure, at the cost of 50% storage utilization.

    • Erasure Coding (2+1, 6+3) offers protection for 1 FD failures, at the cost of 67% storage utilization.

    • Erasure Coding (6+2, 9+3) offers protection for 1 FD failures, at the cost of 75% storage utilization.

    • 2 Way replication (Multizone-HA offers protection for one FD failure, on each zone or a complete protection zone failure at the cost of 50% storage utilization per protection zone.

    • Erasure Coding (4+2, 2+1 - Multizone-HA) offers protection for 1 FD failures, on each zone or a complete protection zone failure at the cost of 67% storage utilization per protection zone.

    • Erasure Coding (6+2, 9+3 - Multizone-HA) offers protection for 1 FD failures, on each zone or a complete protection zone failure at the cost of 75% storage utilization per protection zone.

    The following Storage Policies are supported:

    Object Storage Data Protection Policies

    Policy Type

    Redundancy

    Minimal Configuration

    2 Way

    x 2

    2 VCs on 2 SNs, each in a different Fault Domain

    Erasure Coding 2+1

    x 1.5

    3 VCs on 3 SNs, each in a different Fault Domain

    Erasure Coding 4+2

    x 1.5

    3 VCs on 3 SNs, each in a different Fault Domain

    Erasure Coding 6+2

    x 1.33

    4 VCs on 4 SNs, each in a different Fault Domain

    Erasure Coding 9+3

    x 1.33

    4 VCs on 4 SNs, each in a different Fault Domain

    Erasure Coding 2+1 4+2 (Multizone-HA)

    x 1.5

    3 VCs on 3 SNs, each in a different Fault Domain (in each region)

    Erasure Coding 6+2 9+3 (Multizone-HA)

    x 1.33

    4 VCs on 4 SNs, each in a different Fault Domain (in each region)

Each drive in the system is assigned to one Storage Policy.

VPSA Object Storage is created with a default data Storage Policy for objects and another system Storage Policy for metadata. The VPSA Object Storage administrator (zios_admin) can later on create additional storage policies, expand existing policies and set a specific policy as the Default Policy.

For Multizone HA VPSA Object Storage, the protection policy is created symmetrically on both zones, in each availability zone, the same protection policy will be created in order to sustain a complete availability zone failure.

Policies are assigned when a container is created. Once a container has been assigned for a policy, it cannot be changed (unless it is deleted/recreated).


Creating Policy (VPSA Object Storage Admin)

When a VPSA Object Storage system is created, 2 policies are created by default:

  • MetadataPolicy: Used to store the Accounts and Containers metadata, usually contains 2 SSD drives, with 2-way replication

  • 2-way/EC-protection-policy: Used to store the users’ objects, usually contain all the drives that were assigned to the Object Storage at creation time, with 2-way replication or EC protection, according to the initial selection on the provisioning portal.

Storage Policies can be added form the provisioning Portal by the VPSA Object Storage admin. To create additional Policies, Go to the Provisioning Portal, select the VPSA Object Storage of the new Storage Policy, and then click the Add Storage Policy button in the right panel. Follow the instruction here: Adding Storage Policy (VPSA Object Storage Admin)


Viewing Policies Properties (VPSA Object Storage Admin)

You can view the following properties and metering information in the Policies Details South Panel tabs:

Properties

Each Policy includes the following properties:

Property

Description

ID

An internally assigned unique ID

Name

The name of Policy

Type

Object or Account/Container

State

Not Configured / Configuring / Initialized

Description

A user defined policy description

GB per Month Price

Price of used capacity for charge back purposes

Health Status

Normal / Degraded / Critical

Health Percentage

Indicates the percentage of the stored data that is accessible.

Rebalance Status

Normal / Rebalancing / Failed

Rebalance Prec.

Indicates the progress of the rebalance process

Cross-Region Rep-mode

Synchronous (for VPSA Multizone HA)

Capacity

Total usable capacity of the Storage Policy

Used Capacity

Amount of written data in the Storage Policy

Containers

Total amount of containers created within the policy

Objects

Total amount of objects created within the policy

Default

Yes/No

Redundancy Level

2-way/EC

Ring Version

Ring Database version

Rebalanced

The date and time when the Policy was last rebalanced. (Rebalance usually happens when the HW configuration changes, Adding/removing drives, failed components, etc.)

Added

The date and time when the Policy was added

Modified

The date and time when the Policy was last modified

Drives

List the drives assigned with the selected Storage Policy.

Capacity Metering

The Metering Charts provide live metering of the capacity usage associated with the selected Policy.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

Used Capacity

Total storage capacity consumed in the selected Policy

Containers

Total numbers of containers that store their objects in the selected policy

Objects

Total numbers of objects stored in the selected policy

image29

Backend Metering

The Metering Charts provide live metering of the IO workload associated with the selected Policy.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

Throughput (OP/s)

The number of operations (PUT/GET/DELETE) issued to the Drives of the selected Policy per second

Bandwidth (MB/s)

Total throughput (in MB) of read and write commands issued to the Drives of selected Policy per second

Av. Drive Latency

Average response time of all operations (PUT/GET/DELETE) issued to objects in the selected Policy per selected interval

image30

Frontend Metering

The Metering Charts provide live metering of the IO workload associated with the traffic coming to the selected Policy.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

Throughput (OP/s)

The number of operations (PUT/GET/DELETE) issued to objects in the selected Policy per second

Bandwidth (MB\s)

Total throughput (in MB) of read and write commands issued to the selected Policy per second

Avg. Latency (ms)

Average response time of all operations (PUT/GET/DELETE) issued to objects in the selected Policy per selected interval

Capacity Alerts

Alert Threshold – Send Alert when it is estimated that the Policy will be at full capacity within the given time period (in minutes)

Alert Interval - Calculate the above threshold based on the capacity usage growth in the previous given time period (in minutes)


Set default Policy (VPSA Object Storage Admin)

The default Policy is the Policy used by default for newly created containers, if no other policy is explicitly specified. To change the default Policy, select the Policy you want to make the default and click “Set as Default”.

Note

As long as there is only one Policy for user data (this is the situation when a new VPSA Object Storage system is created), that only Policy is the default, and there no way to change it.


Adding Drives to Policy (VPSA Object Storage Admin)

Drives are added to an Object Storage policy via the Provisioning Portal. To add drives into a policy, go to the Provisioning Portal, select the VPSA Object Storage of interest and click Add Storage. Follow the instruction here: Adding Drives to an Existing Storage Policy (VPSA Object Storage Admin).

Note

Drive-related operations in a storage policy will require rebalance that might take several hours until completion.


Removing Drives from Policy (VPSA Object Storage Admin)

If the there is a need to reduce the total available capacity of a given policy, or to remove some failed drives that were detached from the policy, you may remove drives from the policy and return them to the cloud for a different use. To remove drives from a Policy go to GUI > Policies, select the policy of interest, and click Remove Drives.

image34

The dialog that will open will list all the drives types and quantities that currently belong to the policy. Set the number of drives you want to remove from each type, and click Remove.

Note

Drive-related operations in a storage policy will require rebalance that might take several hours until completion.