Load Balancing¶

Load balancing refers to distributing incoming traffic across a group of backend server. In order to serve a high volume of concurrent requests from users or clients it is required to have a “traffic distributer” sitting in front of the backend servers and routing requests across all workers capable of fulfilling those requests.

The load balance is responsible for the following functions:

Distributes client requests or network load efficiently across multiple backend servers
Ensures high availability and reliability by sending requests only to servers that are online and can handle the request in a timely manner
Provides the flexibility to scale-up or scale-down as demand dictates

In the context of the VPSA Object Storage, a Proxy+Storage Virtual Controller is referred as a backend server.

How load is balanced in the VPSA Object Storage?¶

The VPSA Object Storage load-balancer has three modes of operation:

Internal load balancer (default)¶

Any VPSA Object Storage is provisioned with a built-in Load Balancer (referred as Internal Load Balancer) solution which will work seamlessly out of the box. The internal load balancer is fully configured to provide a secured, reliable and highly-available endpoint. Upon provisioning a new Object Storage instance, the system will be have a globally unique FQDN matching the Object Storage front-end network, along with a matching well-known CA certificate that will be used for a secured communication over HTTPS.

In case a public IP/additional VNI (Virtual Network Interface) is allocated to the object storage, the internal load balancer will be used to distribute the workload. The main difference between Frontend network and Public/VNI handling is the load balancer mode of operation:

Frontend - DRS (Direct Server Return), packets from the Object Storage Virtual Controller bypass the Load Balancer, maximizing egress throughput.
Public/VNI - The load balancer will be used as a gateway for all traffic from/to the Object Storage Virtual Controller.

Load balancing algorithm¶

The load balancer is configured to use weighted load balancer algorithm, where Proxy Only will have the highest weight and Proxy+Storage controllers will be chosen. Controller VCs are excluded from handling client/users operations.

High availability¶

As all other core services in the system, the internal load balancer is highly available. The internal load-balancer service will be hosted in a Controller VC (vc-1) and in case of a failure will failover to vc-0 to ensure service continuity.

Load balancer metering¶

While the actual object storage operations will be distributed to the “backend servers”, in some cases the system administrator would like to review the amount of concurrent sessions their object storage is handling.

In the VPSA Object Storage the active session count can be reviewed in the management interface:

Virtual Controllers view - navigate to the Virtual Controller view and select the virtual controller with vc-1 as ID. In the south pane, select the Frontend Metering tab.
Performance section - under the system section, navigate to the Performance section. Expand the Virtual Controller > Controller > vc-1 view and drag and drop the Active Connections pane.

The Active Connections connections graph will provide a breakdown of connections within the object storage (per VC). The user can toggle between the graph view and a the table view using the TABLES view.

ZELB (Zadara Elastic Load Balancer)¶

For large (multi-PB) deployments where the object storage consist of dozens or even hundreds of virtual controllers, the object storage administrator may want to consider using either:

Dedicated Load Balancer (software/hardware) appliance in-front of the Object Storage in order to gain full control on the balancing mechanism, traffic shaping policy and security protocols.
Integrate the application directly with the VPSA Object Storage.

A multi-PB VPSA Object Storage deployments will be consist of dozens or even hundreds of Virtual Controllers which will be challenging to manage using an external Load Balancer or via application integration. In order to simplify the management overhead it is possible to enable a dedicated Load Balancer (Elastic Load Balancer).

Elastic Load Balancer will break down the Virtual Controllers into groups of 12 VCs. For each Elastic Load Balancer Group (ZELBG) additional two virtual controllers will be provisioned in order to orchestrate the load within the group.

ZELB Terminology¶

ZELB - Zadara Elastic Load Balancer
ZELBG - Zadara Elastic Load Balancer Group (12 VCs per group)
ZELB-Master - proxy VC that will be used as the ZELB active load balancer
ZELB-Slave - proxy VC that will be used as the backup load-balancer
VRRP - Virtual Router Redundancy Protocol
VRID - Virtual Router Identifier
FIP - floating IP

Enable/Disable ZELB¶

The cloud administrator can enable/disable the ZELB component on demand. Since ZELB has networking pre-requisites, it is possible to enable it from Zadara’s Command Center only.

Upon successful provisioning of the ZELB, a new section will be visible in the management interface named Load Balancer Groups.

The Load Balancer Groups view will provide extended visibility to the ZELB components.

Important

Enabling ZELB will not disable the internal load balancer mechanism. The internal load balancer endpoint will continue to serve the object storage management interface using port 8443 (HTTPS).

Pre-requisites

Ensure you have sufficient free IP addresses for allocation in your Frontend network scope (can be either the default cloud FE network or custom virtual network)
Ensure you have allocated a VRID range for the VLAN in use (acceptable range between values between 1-255). A single and unique VRID will be allocated per ZELBG.

Important

VRID usage should be unique within a given VLAN; conflicting VRID(s) might cause networking conflicts and routing issues within the object storage.

Understanding the ZELB architecture¶

The ZELB main concept is to group the VCs into ZELB group(s) - ZELBG. Each ZELBG will be formed out of up to 12 VCs and additional pair of VCs that will constitute an HA group for its corresponding group using the VRRP protocol

The ZELBG will be allocated with a unique FIP that will be used as the ZELBG endpoint for incoming requests. DNS name will not be registered with the specific endpoint.

Grouping the VCs into smaller group will aid the system administrator to distribute the load efficiently using a smaller amount of endpoints.

Load balancing algorithm¶

Similar to the internal load balancer the ZELB is configured to use weighted load balancer algorithm, where Proxy Only will have the highest weight and Proxy+Storage controllers will be chosen. Controller VCs are excluded from handling client/users operations.

Load balancer metering¶

While the actual object storage operations will be distributed to the “backend servers”, in some cases the system administrator would like to review the amount of concurrent sessions their object storage is handling.

Similar to the internal load balancer, the active session count can be reviewed in the management interface for each ZELBG:

Load Balancer Groups view - select one of the ZELBG. In the south pane, select the Frontend Metering tab.
Performance section - under the system section, navigate to the Performance section. Expand the Load Balancer Groups > lbg-0000X view and drag and drop the Active Connections pane.

The Active Connections connections graph will provide a breakdown of connections within the object storage (per VC). The user can toggle between the graph view and a the table view using the TABLES view.

Using an external load balancer without ZELB¶

The VPSA Object Storage provides an easy way to integrate with an existing or newly provisioned software/hardware load balancer even without the need to enable ZELB.

The instructions below are example for setting up an external load balancer to terminate SSL connections and distribute the across all VCs.

There are many load balancer solutions in the market, setting them all up is quite similar procedure. This appendix gives an example of HAproxy, an open-source TCP/HTTP load-balancing proxy server that can be found in www.haproxy.org

The recommended configuration below will allow the following:

SSL Termination is done on the external load balancer for both object operation API’s and GUI connections. Authentication connections are always terminated in Object Storage.
Custom SSL certificate (PEM) located on the load balancer is used for SSL connections
Object operation connections are redirected to VPSA Object Storage proxy VC’s
Object operation connections are distributed between VC’s unevenly (proxy-only VCs to take more load than storage VC’s, and HA VCs to take the lowest load)
Redirected object operation connections will include the original client IP in a special header added by the load balancer (for logging in VPSA Object Storage proxy)
HTTP-based health check is performed by the load balancer to probe all VPSA Object Storage proxy VC’s
Authentication connections are redirected to ZObject Storage floating IP (SSL pass-through terminated on the VPSA Object Storage, Custom SSL certificate must be uploaded to the VPSA Object Storage as well).
GUI connections are redirected to the VPSA Object Storage floating IP
Graphical statistics interface is enabled on the load balancer

Apply the following configuration to your VPSA Object Storage Settings:

Set the internet-facing domain-name/IP of the external LoadBalancer as VPSA Object Storage API Hostname / IP (zadara-qa.com which resolves to the external LB IP 180.80.2.217, is set in this example as VPSA Object Storage API Hostname)
Upload your custom SSL certificate (will be used for authentication connections). The certificate should match the custom domain name.
Set SSL Termination to “External”

HAProxy Installation and configuration instructions:

Install HA Proxy:

sudo add-apt-repository -y ppa:vbernat/haproxy-1.5
sudo apt-get update
sudo apt-get install -y haproxy

Upload your custom SSL certificate to HAProxy server. In this example the certificate PEM file is placed under /etc/ssl/private/zadara\_custom.pem

Edit /etc/haproxy/haproxy.cfg to include the following:

global
    maxconn 2048
    log /dev/log  local0
    log /dev/log  local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    tune.ssl.default-dh-param 2048
    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private
    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ssl-default-bind-ciphers
    ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3

 defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http
    frontend fe-object-operations
    bind 180.80.2.217:443 ssl crt /etc/ssl/private/zadara\_custom.pem
    mode http
    default\_backend be-zios-object-operations

 frontend fe-auth
    bind 180.80.2.217:5000
    option tcplog
    mode tcp
    default\_backend be-floating-zios-auth

 frontend fe-gui
    bind 180.80.2.217:8443 ssl crt /etc/ssl/private/zadara\_custom.pem
    mode http
    default\_backend be-floating-zios-gui

 backend be-zios-object-operations
    mode http
    balance roundrobin
    option forwardfor
    option  httpclose
    option  httpchk HEAD /healthcheck HTTP/1.0
    server ziosStorageProxy0 190.90.2.102:8080 weight 10 check
    server ziosStorageProxy1 190.90.2.104:8080 weight 10 check
    server ziosStorageProxy2 190.90.2.114:8080 weight 50 check
    server ziosProxyOnly3 190.90.2.106:8080 weight 100 check
    server ziosProxyOnly4 190.90.2.109:8080 weight 100 check

 backend be-floating-zios-auth
    mode tcp
    server ziosFloating 190.90.2.118:5000

 backend be-floating-zios-gui
    mode http
    server ziosFloating 190.90.2.118:80

 listen stats \*:1936
    stats enable
    stats uri /
    stats auth zadara:zadara

Enable HAProxy logging (Optional)

Edit rsyslog.conf

sudo vi /etc/rsyslog.conf
# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Restart the service:
```
sudo service rsyslog restart
```

Restart HAProxy service:
```
sudo service haproxy restart
```
Monitor statistics by browsing to http://**<HAProxy server IP>**:1936/ Credentials: zadara/zadara