Node Management

Introduction

Nodes are the physical servers that provide compute, storage, and networking resources in your cloud region. As an MSP administrator, you manage the full lifecycle of these nodes through the Region Management > Nodes page.

A node passes through several states during its lifecycle:

Status lifecycle

• Candidate

  A newly discovered server that has not yet joined the cluster.

• Approved

  A candidate that was accepted but has not finished activation.

• Starting

  The node is booting and initializing services.

• Activating

  The node is completing its activation process.

• Active

  The node is online and running workloads.

• Failed

  The node encountered an error and is not functioning correctly.

• Fenced

  The node was isolated from the cluster to protect data integrity.

• Entering maintenance

  Workloads are migrating off the node in preparation for maintenance.

• In maintenance

  The node is offline for servicing. No workloads run on it.

• Unfencing

  The node is rejoining the cluster after a fencing event.

• Inadequate

  The node has unsupported hardware and cannot join the cluster.

• Rejected

  An administrator declined this candidate node.

• Out

  The node has been removed from the cluster.

The Nodes page

The Nodes page shows all nodes and node candidates in a single table.

You can switch between two views:

List view

A table with sortable columns:

• Name

  The display name of the node.

• Candidate

  Indicates whether the node is still a candidate and has not yet joined the cluster.

• IP

  The management (access) IP address of the node.

• CPU

  The current CPU usage of the node.

• vCPUs

  The total number of virtual CPU cores available on the node.

• Memory

  The total physical memory installed on the node.

• VM Memory (Total)

  The total amount of memory allocated to virtual machines on this node.

• VM Memory (Available)

  The remaining memory available for additional virtual machines.

• Network Rx

  The amount of inbound network traffic received by the node.

• Network Tx

  The amount of outbound network traffic transmitted by the node.

• Storage

  The total storage capacity available on the node.

• Uptime

  The amount of time the node has been running since its last restart.

• Status

  The current lifecycle state of the node (for example, Active, In maintenance, or Failed).

• Tags

  Custom labels assigned to the node for identification and filtering.

CPU, Memory, and Network columns include sparkline charts that show recent usage trends.

Heatmap view

A visual map of nodes grouped and color-coded by resource or status.

You can group nodes by:

• Status

  Groups nodes according to their lifecycle state, such as Active or In maintenance.

• CPU

  Groups nodes based on CPU capacity or usage.

• Memory

  Groups nodes based on memory capacity or usage.

• Storage

  Groups nodes based on storage capacity.

You can color the heatmap by:

• Status

  Applies colors based on lifecycle state.

• Used Memory

  Colors nodes according to the amount of memory currently used.

• Memory

  Colors nodes based on total memory capacity.

• Used CPU

  Colors nodes according to current CPU utilization.

• CPU

  Colors nodes based on total CPU capacity.

• Storage

  Colors nodes based on storage capacity.

• Throughput

  Colors nodes according to network activity levels.

Built-in filters

Use filters to narrow the table to specific node states.

• Active

  Shows only nodes that are currently online and running workloads.

• Candidate

  Shows only nodes that were discovered but have not yet joined the cluster.

• Inadequate

  Shows only nodes that do not meet hardware requirements.

• In Maintenance

  Shows only nodes that are currently in maintenance mode.

Node candidates management

A node candidate is a physical server that has booted, completed hardware discovery, and appeared in the system but has not yet joined the cluster.

Before a candidate can run workloads, you must add it, review its hardware, and join it to the cluster.

Add a candidate node

Add a candidate node when you have a new physical server that you want to register with the cluster. This operation tells the system the server’s network address and credentials so it can connect and inspect the hardware.

Prerequisites:

• The physical server is powered on and network-reachable.

• The server’s access IP address, SSH username, and SSH password.

• The MAC addresses of the server’s converged network interfaces.

Steps:

1. Go to Region Management > Nodes.

2. Select + Add Candidate in the top toolbar.

3. In the Add Candidate Node dialog, complete the Info step:

   • Candidate Access IP: Enter the server’s IPv4 address.

   • SSH User: Enter the SSH username.

   • SSH Password: Enter the SSH password.

   • Node Type: From the dropdown, select the node type:

     • Worker: Runs tenant workloads.

     • Control: Runs cluster management services.

4. Select Next to continue to the MAC Addresses step.

5. Enter one or more Converged MAC Addresses.

   These identify the network interfaces the node will use for converged traffic.

   Select Add to enter another Converged MAC Address.

6. To confirm adding the candidate node, select Finish.

   The system displays “Candidate node creation is in progress.”

   When complete, the system displays “Candidate node added successfully.”

Modify a candidate node

Modify a candidate node when you need to update SSH credentials or change the converged MAC addresses. You cannot change the access IP address after the candidate is created.

Steps:

1. Go to Region Management > Nodes.

2. In the nodes table, locate the candidate node.

3. In the top toolbar, select Modify Candidate.

4. In the Modify Candidate Node dialog, update:

   • SSH User and SSH Password.

   • Converged MAC Addresses (add or remove).

   Candidate Access IP and Access MAC Addresses are read-only.

5. Select Save.

   The system displays “Candidate node modification is in progress.”

   When complete, the system displays “Candidate node modified successfully.”

Join a candidate node to the cluster

Join a candidate node when you are satisfied with its hardware configuration and want it to begin serving workloads. During this step, you choose the node’s role in the cluster.

Steps:

1. Go to Region Management > Nodes.

2. In the nodes table, locate the candidate node.

3. In the top toolbar, select Join.

4. In the Join Candidate Node dialog, review:

   • IP: Displays the node’s access IP (read-only).

   • Node Link Details:

     Expand sections to review interface details including Name, Address (MAC), Vendor, Duplex, Speed, Driver, Device Name, Carrier status (up/down), PCI Slot, Type, and Device ID.

5. In the Node Type dropdown, select a role:

   • Control: Runs cluster management services.

   • Worker: Runs tenant workloads.

6. Select Join and confirm.

   The system displays “Candidate node join is in progress.”

   When complete, the system displays “Candidate node joined successfully.”

Delete a candidate node

Delete a candidate node when the server is no longer needed or was added in error. This removes the candidate record from the system. It does not affect the physical server.

Steps:

1. Go to Region Management > Nodes.

2. Locate the candidate node.

3. In the top toolbar, select Delete Candidate.

4. In the Delete Node dialog, review the node’s IP address.

5. To confirm deletion, select Delete.

   The system displays “Candidate node deleted successfully.”

Node management activities

After a node joins the cluster and becomes active, you manage it through a different set of operations.

Accept (approve) a candidate node

When a node appears with Candidate status, you can accept it to allow it to proceed toward activation.

Steps:

1. Go to Region Management > Nodes.

2. Locate the node with Candidate status.

3. In the top toolbar, select Accept.

The node status changes to Approved, then proceeds through Starting and Activating automatically.

Reject a candidate node

Reject a candidate node when you do not want it to join the cluster. This marks the node as rejected without deleting its record.

Steps:

1. Go to Region Management > Nodes.

2. Locate the node with Candidate status.

3. Open the actions menu and select Reject.

The node status changes to Rejected.

Rename a node

Rename a node to give it a descriptive name or update its description. This does not affect the node’s hostname or functionality.

Steps:

1. Go to Region Management > Nodes.

2. Select the node name to open the detail page, or open the actions menu and select Rename.

3. In the Rename Node dialog, enter:

   • Name: Required.

   • Description: Optional.

4. Select OK.

Put a node into maintenance mode

Put a node into maintenance mode when you need to perform hardware repairs, firmware updates, or other servicing. The system migrates running workloads off the node before it enters maintenance.

Prerequisites:

• The cluster has more than one node.

• No active VMs with GPU passthrough devices are running.

• Remaining nodes have enough memory for migrating VMs.

Steps:

1. Go to Region Management > Nodes.

2. Locate a node with Active, Failed, or Fenced status.

3. Open the actions menu and select Maintenance.

4. In the Node Maintenance dialog, review the node’s ID and Name.

5. Select OK to confirm.

The status changes to Entering maintenance while VMs migrate, then to In maintenance when complete.

Force maintenance

Use force maintenance only as a last resort. It bypasses the normal workload migration process.

Severe warning: Forcing a node into maintenance mode prevents the system from completing some housekeeping actions and may result in full system failure and data loss.

Steps:

1. In the Node Maintenance dialog, select Force Maintenance.

2. Read the warning carefully.

3. Wait for the safety timer to complete.

4. Confirm the action.

The system displays: “Node [name] is being forcefully deactivated.”

Activate a node (exit maintenance mode)

Activate a node to return it to service after maintenance is complete.

Steps:

1. Go to Region Management > Nodes.

2. Locate the node with In maintenance status.

3. Open the actions menu and select Activate.

The node returns to Active status.

Remove a node from the cluster

Remove a node when you want to permanently take it out of the cluster. The node must be in maintenance mode before removal.

Prerequisites:

• The node is in In maintenance status.

• The cluster has more than one node.

Steps:

1. Go to Region Management > Nodes.

2. Locate the node.

3. Open the actions menu and select Remove.

4. Review the node’s IP address.

5. Select Remove.

Configure a node (open web installer)

Use Configure to open the node’s built-in installer interface in a new browser tab. This is useful for initial hardware setup or advanced configuration.

Steps:

1. Go to Region Management > Nodes.

2. Locate the node.

3. Open the actions menu and select Configure.

A new tab opens to: http://<node-ip>/installer.

Manage node tags

Tags help you organize and identify nodes by purpose, location, hardware type, or other categories.

Add a tag:

1. Go to Region Management > Nodes.

2. Select the node name to open the detail page.

3. Select Add tag.

4. Enter the tag name and confirm.

Remove a tag:

1. On the node detail page, locate the tag.

2. Select the remove action on the tag.

Create an alarm for a node

You can set up CloudWatch-style alarms to monitor node metrics and receive notifications when thresholds are exceeded.

Steps:

1. Go to Region Management > Nodes.

2. Locate the node.

3. Open the actions menu and select Create Alarm.

4. Complete the alarm creation form. See Creating an Alarm.

View node details

Select any node name in the table to open its detail page.

Tabs:

• Overview tab

  Displays the node name, description, status, IP address, uptime, memory, disk capacity, and CPU details.

• Events tab

  Shows a chronological log of events related to this node.

• Node Links tab

  Lists network interfaces with MAC address, admin state, operational state, MTU, vendor, model, and bond type. Expand rows to view cluster network interfaces with IP address, VLAN ID, and state.

• Monitoring tab

  Provides performance charts for CPU, memory, and network usage over time.

• Disks tab

  Displays physical disks attached to the node.

• VMs tab

  Lists virtual machines currently running on the node.

• Services tab

  Shows cluster services running on the node, including CPU and memory usage per service.

• PCI Devices tab

  Lists PCI passthrough devices including device name, PCI slot, vendor, device ID, enabled or blocked status, kernel driver, and associated VM if assigned.

Recommended best practices

Cluster sizing

• Maintain at least two nodes in every cluster.

• Keep enough spare memory so any single node can enter maintenance without capacity issues.

Before maintenance

• Check the node’s VMs tab for running workloads.

• Shut down or migrate GPU-attached VMs before maintenance.

• Verify sufficient free memory on remaining nodes.

Monitoring

• Use the Heatmap view to spot imbalances.

• Review sparkline charts for sustained trends.

• Set up alarms for critical metrics.

• Monitor partition free space indicators such as /var/log, /mnt/data, and /mnt/containers.

Tags

• Create a consistent tagging strategy.

• Use tags to help filter and identify nodes quickly.

Node removal

• Always put a node into maintenance mode before removal.

• If degraded data pools are detected, contact support.

Network verification

• Verify converged MAC addresses when adding candidates.

• Review the Node Links tab after a node joins.

Troubleshooting

A candidate node does not appear in the table

Possible causes:

• Hardware discovery not complete.

• Server is not network-reachable.

• Table filter excludes Candidate view.

“Failed to add candidate node” error

Possible causes:

• Candidate Access IP is incorrect or unreachable.

• SSH User or SSH Password credentials are wrong.

• SSH service is not running.

Resolution: Verify IP and credentials. Confirm SSH connectivity.

A node shows “Inadequate” status

Hardware does not meet cluster requirements. Review specifications on the Overview tab.

Maintenance mode is blocked by GPU VMs

Shut down affected GPU VMs before retrying.

Maintenance mode is blocked by insufficient memory

Shut down or migrate VMs to free cluster memory.

A node shows “Failed” status

Hover over Status to see the failure reason.

If recoverable: Put into maintenance, fix the issue, then activate.

If not recoverable: Put into maintenance and remove from cluster.

A node shows “Fenced” status

The node was isolated to protect data integrity.

Possible next steps:

• Wait for recovery.

• If not recovering, use maintenance mode and investigate.

“Failed to join candidate node” error

Possible causes:

• Network links not in expected state.

• Selected Node Type is invalid for cluster configuration.

Node removal shows a “data loss” warning

Do not proceed. Contact support to assess data pool health.

The “Remove” and “Maintenance” actions are not available

• Remove requires In maintenance status and more than one node.

• Maintenance requires Active, Failed, or Fenced status and more than one node.

• Single-node clusters do not support these actions.