Resources Management

Scope: Object Storage Administrator

Monitoring Drives

To monitor drives in your Object Storage system, open Resources > Drives.

Viewing Drives Properties

The Drives details (properties and metering), are shown in the South Panel tabs:

Properties

Each Drive includes the following properties:

Property

Description

ID

An internally assigned unique ID

Name

Automatically assigned name.

Capacity

The Drive capacity in GiB

Storage Node

The Storage Node that contains the selected Drive

Virtual Controller

The virtual controller that owns the selected drives and performs IO operations on it

Storage Policy

The Storage Policy where the selected Drive belongs

Fault Domain

The Zadara cloud Fault Domain this Drive resides belongs to

Protection Zone

The Zadara cloud protection zone this drive is physically located at

Type

Drive type: SATA, SAS, SSD

UUID

The unique identifier of the drive

Status

  • Normal – All drives are in sync

  • Failed – The drive does not function

  • Absent – The drive does not exist

Added

Date & time when the drive was added

Modified

Date & time when the drive was last modified

Disk Metering

The Metering Charts provide live metering of the IO workload associated with the selected Drive.

The charts display the last captured performance samples. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, 1 hour, 1 day, 1 week.

The Auto button allows to switch to continuously-update live metering info.

The following charts are displayed:

Chart

Description

IOPs

The number of read and write commands issued to the selected Drive per second

Bandwidth (MB/s)

Total throughput (in MB) of read and write commands issued to the selected Drive per second

Latency (ms)

Average response time of all read and write commands issued to the selected Drive per selected interval

Backend Metering

The following charts are displayed:

Chart

Description

Throughput (OP/s)

The number of operations (PUT/GET/DELETE) that were sent to the selected Drive per second

Bandwidth (MB/s)

Total throughput (in MB) of read and write commands that were sent to the selected Drive per second

Latency (ms)

Average response time of all operations (PUT/GET/DELETE) that were sent to the selected Drive per selected interval


Monitoring Virtual Controllers

Virtual Controllers are Virtual Machines running on the Zadara zStorage cloud that serve client operations on the Object Storage. For a full list of the VC responsibilities refer to Virtual Controller. Virtual Controllers are automatically created and added/removed to the Object Storage configuration, depending on the number of the allocated drives. In case the workload have changed and additional performance are required - the object storage owner can add Proxy only VCs from the Zadara Provisioning Portal as described in Adding Proxy Virtual Controllers.

Viewing VCs Properties

Properties

Each Virtual Controller has the following properties:

Property

Description

ID

An internally assigned unique ID

Storage Role

Proxy+Storage / Proxy-Only

Management Role

  • Ring Master – Runs the Object Storage Rings

  • Ring Slave – Standby to run the Object Storage Rings

  • VC – Regular Object Storage VC

Status

  • Created – VC is running normally

  • Failed – VC is not running

  • Passivating – VC is shutting down

  • Deleting – in the process of being removed from the cluster

Storage Node

The Storage Node hosting selected VC

Fault Domain

The Zadara cloud Fault Domain this VC resides belongs to

Frontend IP

The IPv4 or IPv6 address allocated to the VC

Backend IP

The VC IP address on the backend network that connects to the Drives

Added

Date & time when the VC was added

Modified

Date & time when the VC was last modified

Drives

List the drives assigned to the selected Storage Policy.

Virtual Networks

A set of available IP addresses within a specific network segment. Virtual networks are allocated for a specific cloud tenant and within a specific avilable cloud VLAN.

System Usage

This chart shows the CPU utilization of the selected VC.

Backend Metering

The Metering Charts provide live metering of the IO workload at the backend of the selected VC.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

IOPS

The number of operations (PUT/GET/DELETE) issued to objects and handled by the selected VC per second

Bandwidth (MB/s)

Total throughput (in MB) of read and write commands issued by the selected VC per second

Latency (ms)

Average response time of all operations (PUT/GET/DELETE) issued to objects and handled by the selected VC per selected interval


Account Metadata Metering

The Metering Charts provide live metering of the IO workload on the accounts database at the backend of the selected VC.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

IOPS

The number of operations (PUT/GET/DELETE) issued to the accounts database and handled by the selected VC per second

Latency (ms)

Average response time of all operations (PUT/GET/DELETE) issued to the accounts database and handled by the selected VC per selected interval


Container Metadata Metering

The Metering Charts provide live metering of the IO workload on the containers database at the backend of the selected VC .

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

IOPS

The number of operations (PUT/GET/DELETE) issued to containers and handled by the selected VC per second

Latency (ms)

Average response time of all operations (PUT/GET/DELETE) issued to containers and handled by the selected VC per selected interval


Object Metadata Metering

The Metering Charts provide live metering of the IO workload on the object database at the backend of the selected VC .

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

IOPS

The number of operations (PUT/GET/DELETE) issued to objects and handled by the selected VC per second

Latency (ms)

Average response time of all operations (PUT/GET/DELETE) issued to objects and handled by the selected VC per selected interval


Frontend Metering

The Metering Charts provide live metering of the IO workload at the frontend of the selected VC.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

Throughput (OP/s)

The number of operations (PUT/GET/DELETE) issued to objects and handled by the proxy of the selected VC per second

Bandwidth (MB/s)

Total throughput (in MB) of read and write commands issued to proxy of the selected VC per second

Latency (ms)

Average response time of all operations (PUT/GET/DELETE) issued to objects and handled by proxy of the selected VC per selected interval

Managing Storage Policies

The Storage Policy provide a way for object storage providers to differentiate service levels, features and behaviors of an Object Storage deployment.

Policies can be thought of as a group of drives, with a redundancy level policy assigned to it.

Before placing object data into the Object Storage, users create a container which holds the listing of all objects stored under the container’s namespace. Users can select the Storage Policy that will be used when storing data objects under a container’s namespace when they create the container. All objects stored in a container will be placed according the configuration of the Storage Policy which was set upon the Object Storage creation.

To ensure availability of the Object Storage data, the drives assigned to a Storage Policy are evenly distributed between Object Storage Fault Domains. The cloud administrator defines the Fault Domain of each Storage Node. The system allocates drives across zones, based on the Storage Policy type.

Storage Policies allow objects to be stored based on the following criteria:

  • Quality of Service: By using different disk drives for different policies, tiers of storage performance can be created. For example, an SSD-only policy can be created and used to implement a low-latency/high performance tier.

  • Number of Replicas:

    • 2 Way replication offers protection for one FD failure, at the cost of 50% storage utilization.

    • Erasure Coding (4+2) offers protection for one FD failure, at the cost of 67% storage utilization.

    The following Storage Policies are supported:

    Object Storage Data Protection Policies

    Policy Type

    Redundancy

    Minimal Configuration

    2 Way

    x 2

    2 Storage VCs on 2 SNs

    Erasure Coding 4+2

    x 1.5

    3 Storage VCs on 3 SNs

Each drive in the system is assigned to one Storage Policy.

Object Storage is created with a default data Storage Policy for objects, and another system Storage Policy for metadata. The Object Storage administrator (zios_admin) can later expand/shrink the storage policy based on their needs.

Storage Policies

When the Object Storage system is created, 2 policies are created by default:

  • MetadataPolicy: Used to store the Accounts and Containers’ metadata

  • 2-way/EC-protection-policy: Used to store the users’ objects, usually contains all the drives that were assigned to the Object Storage at creation time, with 2-way replication or EC protection, according to the initial selection on the provisioning portal.

Policies Properties

You can view the following properties and metering information in the Policies Details south panel tabs:

Properties

Each Policy includes the following properties:

Property

Description

General

ID

An internally assigned unique ID

Name

The name of policy

Type

Object or Account/Container

State

Not Configured / Configuring / Initialized

Description

A user defined policy description

Ring Version

Ring Database version

Default

Yes/No

Redundancy Level

2-way/EC

Policy Capacity

Total usable capacity of the storage policy

Policy Used Capacity

Total capacity used in the storage policy

Data Usage

Amount of written data in the storage policy by the Object Storage account’s users

Containers Count

Total amount of containers created within the policy

Objects Count

Total amount of objects created within the policy

GB per Month Price

Price of used capacity for charge back purposes

Added

The date and time when the policy was added

Modified

The date and time when the policy was last modified

Policy Health

Health Status

Normal / Degraded / Critical

Balance

Indicates the progress of the rebalance process

Rebalance Paused

Yes/No

Last Data Rebalance

The date and time of the last data rebalance

Last Ring Rebalance

The date and time of the last ring rebalance

Note

Please note that Policy Used Capacity attribute will always reflect higher capacity consumption than Data Usage. The Policy Used Capacity also include the underlying object allocated metadata (4K) and the actual object storage allocation (the minimal allocation unit is 4K). Data Usage is the actual account’s data usage representation.

Drives

List the drives assigned with the selected Storage Policy.

Capacity Metering

The Metering Charts provide live metering of the capacity usage associated with the selected Policy.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

Used Capacity

Total storage capacity consumed in the selected policy

Containers

Total numbers of containers that store their objects in the selected policy

Objects

Total numbers of objects stored in the selected policy

Backend Metering

The Metering Charts provide live metering of the IO workload associated with the selected Policy.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

Throughput (OP/s)

The number of operations (PUT/GET/DELETE) issued to the Drives of the selected policy per second

Bandwidth (MB/s)

Total throughput (in MB) of read and write commands issued to the Drives of selected policy per second

Avg. Drive Latency

Average response time of all operations (PUT/GET/DELETE) issued to objects in the selected policy per selected interval

Frontend Metering

The Metering Charts provide live metering of the IO workload associated with the traffic coming to the selected Policy.

The charts display the metering data as it was captured in the past 20 intervals. An interval length can be one of the following: 10 second, 1 minute, 10 minutes, or 1 hour, 1 day, 1 week. The Auto button lets you see continuously-updating live metering info.

The following charts are displayed:

Chart

Description

Throughput (OP/s)

The number of operations (PUT/GET/DELETE) issued to objects in the selected policy per second

Bandwidth (MB/s)

Total throughput (in MB) of read and write commands issued to the selected policy per second

Avg. Latency (ms)

Average response time of all operations (PUT/GET/DELETE) issued to objects in the selected policy per selected interval

Capacity Alerts

The Object Storage administrator can set their own custom configuration for capacity monitoring.

Alert Threshold

Send alert when it is estimated that the policy will be at full capacity within the given time period (default 43,200 minutes - 30 days).

Alert Interval

The period of the sample collection window to calculate the rate at which the free space is consumed in the storage policy.

  • Default 1,440 minutes

  • Minimal value is 10 minutes

  • Minimum used capacity to trigger the alert is 60%

Emergency Threshold

The threshold to trigger administrator capacity emergency state notifications

  • Default is 85%

  • Minimal value is 60%

  • Alert will be triggered hourly as long as the policy is in “Emergency state”

Adding Drives to Policy

Drives are added to an Object Storage policy via the Provisioning Portal. To add drives into a policy, go to the Provisioning Portal, select the Object Storage of interest and click Add Storage. Follow the instruction here: Adding drives.

Note

Drive-related operations in a storage policy will require rebalance that might take several hours/days until completion.

Removing Drives from Policy

If there is a need to reduce the total available capacity of a given policy, or to remove some failed drives that were detached from the policy, you may remove drives from the policy and return them to the cloud for a different use. To remove drives from a policy, go to GUI > Policies, select the policy of interest, and click Remove Drives.

The dialog that opens will list all the drives types and quantities that currently belong to the policy. The system support removing a pair of drives in a single operation. In case a larger volumes of drives needs to be removed please contact Zadara’s support.

Note

Drive-related operations in a storage policy will require rebalance that might take several hours until completion.