Managing Storage Nodes

One of Command centers key roles is to enable Could infrastructure management in which storage nodes are a core component. Command center provides comprehensive management and monitoring capabilities for the clouds storage nodes including all aspects of:

  • Ongoing maintenance

  • Upgrade managing

  • Hardware addition and retirement

  • Health checks

  • Performance monitoring

Understanding The Storage Node Dashboard

The storage node dashboard presents information regarding its configuration, status and resource utilization. The dashboard can be reached by clicking on Storage Nodes from Command center left menu panel and then selecting a specific storage node from the cloud inventory and clicking on it. The storage node dashboard contains multiple panels each providing information on a specific aspect of the SN configuration and status:

Resource Utilization

Provides a current image on SN hardware resources and their level of utilization.

image189

Drive Utilization

Provides a breakdown of the SN drive inventory by type and role and displays the level of utilization per each specific group.

image190

services

Displays the list of services running on the SN and their current status.

image191

In case that some services on the storage note are not in normal status those specific services will also be indicated by a tooltip on the cloud storage nods grid.

image272

Node Information Displays SN hardware, configuration and status information such as: SW version, uptime and serial number.

image192

Network

Displays SN networks and configured IP addresses.

image193

Resource Scheduling

Display information regarding the availability of the Sn resources(VCPUs and disk drives) To be allocated for newly provisioned VPSA/VPSA Object Storage entities by the clouds orchestration framework.

image194

NIC Information Displays hardware and configuration information on the SN data path network card.

image195

Note

Mellanox ConnectX-5 dual port NICs will be presented as 2 sperate network adapters
due to having 2 different PCI addresses.

CPU information

Displays information on the SNs processors.

image196

License information

Displays SN licensing information.

image197

Storage Adapter

Displays information on the SN internal RAID adapter.

image198

Monitoring Storage Node Performance

Command center provides a real time monitoring framework for storage node performance. Monitoring is available from the Performance in tab in the specific VPSA.

Performance statistics displayed per storage nodes include :

  • Storage node CPU utilization

  • Storage node memory consumption

  • Network bandwidth distribution (read/write)

  • Average IO service time per IO type

  • IO throughput distribution

    image199

Monitoring interval can be changed supported intervals are : 1sec ,10sec, 1min, 1hr and 1day. Interactively refreshed charts can be triggered by pushing the Auto refresh button.

Performing Storage Node Operations

Changing storage node resource scheduling configuration

A storage node contains compute and storage resources(CPU,RAM and disks) that are allocated for virtual controllers of VPSA and VPSa object storage entities. Allocation of SN resources for the creation of virtual controllers or allocation of disks from SN to virtual controllers can be enabled or disabled using Command center.

To modify a storage node resource allocation policy go to the resources scheduling panel in the SN dashboard.

  • Click on Enable or Disable for Drive Scheduling to enable/disable allocation of disks from this SN to virtual controllers.

  • Click on Enable or Disable for VC Scheduling to enable/disable creation of virtual controller on this SN.

Any changes made in the SN scheduling policy are immediately applied.

Creating storage node Zsnap

To Trigger Manual creation of Zsnap for a storage node go to its dashboard, click the Actions button and select Create Zsnap from the drop down menu. On the popup dialog that will appear provide the prefix for the Zsnap and confirm creation by clicking on the Create Zsnap button.

Install and import new drives in a storage node

Cloud storage capacity expansion is performed by installing drives into the storage node importing the newly installed drives into the SN. The import operation encapsulates hardware discovery and SN\cloud inventory update. To preform import of newly installed physical drives select the SN to which drives where installed, click on the Actions button and select Import Drives. On the popup dialog that will appear confirm the import operation by clicking on the Confirm button. A resource scan will be performed and any newly installed disks will be imported. The output from the import operation will be presented in a new popup window.

image200

Perform Storage Node drive configuration check

Command Center can trigger a Storage node drive configuration check in which the SN drive configuration is validated against the OS drive configuration. To perform a drive configuration check select the SN to which drives where installed click on the Actions button and select Check Configuration. On the popup dialog that will appear confirm the import operation by clicking on the Confirm button. The configuration check will be immediately performed and its output displayed on a new popup dialog.

image215

Performing storage node version upgrade

Storage node Version upgrade can be performed as part of a complete cloud upgrade workflow from Command centers main dashboard or from the Storage Node dashboard. Upgrade done from the SN dashboard is useful when the cloud upgrade process is performed gradually as a multiple milestone process.

To upgrade A storage node from the SN dashboard navigate to the appropriate SN, click on the Actions button and select Upgrade Version. On the popup dialog that will appear select the software version you would like to upgrade your SN to and click on Upgrade to confirm the process.

Note

Versions marked by asterisk are available but not downloaded to the cloud. Upgrade to those version requires package download and registration beforehand.

During the SN upgrade process status will be displayed on Command center main dashboard, The upgraded Storage node services panel will present all services as offline.

image216

image217

When the Storage node upgrade is finished completion will be indicated on the Command center dashboard.

image218

Performing storage node utilities version upgrade Storage Node utilities can be upgraded in a dedicated process via Command centers SN dashboard. To upgrade Storage node utilities navigate to the appropriate SN, click on the Actions button and select Upgrade Utilities. On the popup dialog that will appear select the utilities version you would like to upgrade your SN to and click on Upgrade to confirm the process.

Note

Versions marked by asterisk are available but not downloaded to the cloud. Upgrade to those version requires package download and registration beforehand.

During the Utilities upgrade process status will be displayed on Command center main dashboard.

image219

When the Storage node utilities upgrade is finished completion will be indicated on the Command center dashboard.

image220

Performing upgrade of storage node disk drives\RAID controller firmware Command center can be used to trigger an update of a Storage node disk drive firmware or RAID controller firmware. Disk drive\RAID controller firmware are bundles with a specific SN software distribution and are updated according to each SW version supported levels.

To update Drive\RAID controller firmware for a Storage node navigate to the required node dashboard, click on the Actions button and select Upgrade Drives & Adapter Firmware. On the popup dialog that will appear select the elements you would like to perform FW upgrade for, available options are : Disk drives, Intel Optane drives and RAID controller. In case of Disk drives or Intel Optane drives FW upgrade : all disk drives and virtual controllers running on the SN will be taken offline.

image222

In case of RAID controller FW upgrade the SN will reboot after FW is installed.

image221

To confirm upgrade type “UPGRADE” in the text box as required and click on the Upgrade button.

The firmware upgrade process status will be displayed on Command center main dashboard. Completion will be also indicated on the Command Centers main dashboard.

Evacuating all virtual controllers from a storage node

To immediately free all storage node compute resources you can use Command center to evacuate all virtual controllers running on it. Evacuation of virtual controllers is useful in preparation for performing activities such as hardware maintenance or refresh.

To evacuate all virtual controllers from a storage note navigate to its dashboard, click on the Actions button and select Evacuate Virtual Controllers.

image223

On the popup dialog that will appear type “EVACUATE” and click on Evacuate VCs to confirm the operation.

Note

VC Evacuation is possible only if there is available compute capacity in other storage notes within the cloud which
is appropriate for receiving the evacuated VCs (maintaining dual controller HA for VPSA and Object storage fault domain demands).

Reboot a storage node

To reboot a Storage node navigate to its dashboard Actions button and select Reboot.

image201

On the popup dialog that will appear confirm the reboot operation by typing reboot on the textbox as requested and clicking on the Reboot button.

Note

Storage Node reboot will be performed only if all underlying VPS/ VPSA Object storage instances status is Normal.
A reboot of CCmaster\CCslave will be performed only if the DRBD service is up to date.
The restrictions noted above can be overridden by checking the Force reboot option. When using Force reboot option
the cloud administrator is considered responsible of verifying and validating the VPSA/DRBD status before rebooting.

Shutdown a storage Node To shut down a Storage node navigate to its dashboard Actions button and select Shutdown.

image202

On the popup dialog that will appear confirm the reboot operation by typing reboot on the text box as requested and clicking on the Shutdown button.

Note

Storage Node shutdown will be performed only if all underlying VPS/ VPSA Object storage instances status is Normal.
A shutdown of CCmaster\CCslave will be performed only if the DRBD service is up to date.
The restrictions noted above can be overridden by checking the Force Shutdown option. When using the Force Shutdown
option the cloud administrator is considered responsible of verifying and validating the VPSA/DRBD status before shutting down the SN.