Load balancing refers to distributing incoming traffic across a group of backend server. In order to serve a high volume of concurrent requests from users or clients it is required to have a “traffic distributer” sitting in front of the backend servers and routing requests across all workers capable of fulfilling those requests.
The load balance is responsible for the following functions:
Distributes client requests or network load efficiently across multiple backend servers
Ensures high availability and reliability by sending requests only to servers that are online and can handle the request in a timely manner
Provides the flexibility to scale-up or scale-down as demand dictates
In the context of the Object Storage, a Proxy Virtual Controller is referred as a backend server.
How load is balanced in the Object Storage?¶
The NextGen Object Storage load-balancer has two modes of operation:
Internal load balancer (default)¶
Any NextGen Object Storage is provisioned with a built-in Load Balancer (referred as Internal Load Balancer) solution which will work seamlessly out of the box. The internal load balancer is fully configured to provide a secured, reliable and highly-available endpoint. Upon provisioning a new Object Storage instance, the system will be have a globally unique FQDN matching the Object Storage front-end network, along with a matching well-known CA certificate that will be used for a secured communication over HTTPS.
In case a public IP/additional VNI (Virtual Network Interface) is allocated to the object storage, the internal load balancer will be used to distribute the workload. The main difference between Frontend network and Public/VNI handling is the load balancer mode of operation:
Frontend - DRS (Direct Server Return), packets from the Object Storage Virtual Controller bypass the Load Balancer, maximizing egress throughput.
Public/VNI - The load balancer will be used as a gateway for all traffic from/to the Object Storage Virtual Controller.
Load balancing algorithm¶
The load balancer is configured to use weighted least connection (wlc) - new connections go to the worker with the least number of connections. Reminder - Controller VCs & Storage VCs are excluded from handling client/users operations.
As all other core services in the system, the internal load balancer is highly available. The internal load-balancer service will be hosted in a Controller VC (vc-1) and in case of a failure will failover to vc-0 to ensure service continuity.
Load balancer metering¶
While the actual object storage operations will be distributed to the “backend servers”, in some cases the system administrator would like to review the amount of concurrent sessions their object storage is handling.
In the NextGen Object Storage the active session count can be reviewed in the management interface:
Virtual Controllers view - navigate to the Virtual Controller view and select the virtual controller with
vc-1as ID. In the south pane, select the Frontend Metering tab.
Performance section - under the system section, navigate to the
Performancesection. Expand the
Virtual Controller > Controller > vc-1view and drag and drop the Active Connections pane.
The Active Connections connections graph will provide a breakdown of
connections within the object storage (per VC). The user can toggle between
the graph view and a the table view using the
Using an external load balancer¶
Zadara’s Object Storage provides an easy way to integrate with an existing or newly provisioned software/hardware load balancer.
The instructions below are example for setting up an external load balancer to terminate SSL connections and distribute the across all VCs.
There are many load balancer solutions in the market, setting them all up is quite similar procedure. This appendix gives an example of HAproxy, an open-source TCP/HTTP load-balancing proxy server that can be found in www.haproxy.org
The external load-balancer recommended configuration below will allow the following:
SSL Termination is done on the external load balancer for both object operation API’s and GUI connections. Authentication connections are always handled in the Object Storage itself.
Custom TLS certificate located on the load balancer is used for TLS connections
Object operation connections are redirected to NextGen Object Storage proxy VC’s
Object operation connections are distributed between VC’s unevenly (proxy VCs to take more load than storage VC’s, and HA VCs to take the lowest load)
Redirected object operation connections will include the original client IP in a special header added by the load balancer (for logging in NextGen Object Storage proxy)
HTTP-based health check is performed by the load balancer to probe all NextGen Object Storage proxy VC’s
Authentication connections are redirected to the Object Storage floating IP (SSL pass-through terminated on the NextGen Object Storage, Custom TLS certificate must be uploaded to the NextGen Object Storage as well).
UI requests are redirected to the NextGen Object Storage floating IP (over port 8443)
Graphical statistics interface is enabled on the load balancer
Apply the following configuration to your NextGen Object Storage Settings:
Set the internet-facing domain-name/IP of the external LoadBalancer as NextGen Object Storage API Hostname / IP (zadara-qa.com which resolves to the external LB IP 22.214.171.124, is set in this example as NextGen Object Storage API Hostname)
Upload your custom SSL certificate (will be used for authentication connections). The certificate should match the custom domain name.
Set SSL Termination to “External”
HAProxy Installation and configuration instructions:
Install HA Proxy:
sudo add-apt-repository -y ppa:vbernat/haproxy-1.5 sudo apt-get update sudo apt-get install -y haproxy
Upload your custom SSL certificate to HAProxy server. In this example the certificate PEM file is placed under
Edit /etc/haproxy/haproxy.cfg to include the following:
global maxconn 2048 log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon tune.ssl.default-dh-param 2048 # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1SSL). This list is from: # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS ssl-default-bind-options no-sslv3 defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http frontend fe-object-operations bind 126.96.36.199:443 ssl crt /etc/ssl/private/zadara\_custom.pem mode http default\_backend be-zios-object-operations frontend fe-auth bind 188.8.131.52:5000 option tcplog mode tcp default\_backend be-floating-zios-auth frontend fe-gui bind 184.108.40.206:8443 ssl crt /etc/ssl/private/zadara\_custom.pem mode http default\_backend be-floating-zios-gui backend be-zios-object-operations mode http balance roundrobin option forwardfor option httpclose option httpchk HEAD /healthcheck HTTP/1.0 server ziosStorageProxy0 220.127.116.11:8080 weight 10 check server ziosStorageProxy1 18.104.22.168:8080 weight 10 check server ziosStorageProxy2 22.214.171.124:8080 weight 50 check server ziosProxyOnly3 126.96.36.199:8080 weight 100 check server ziosProxyOnly4 188.8.131.52:8080 weight 100 check backend be-floating-zios-auth mode tcp server ziosFloating 184.108.40.206:5000 backend be-floating-zios-gui mode http server ziosFloating 220.127.116.11:80 listen stats \*:1936 stats enable stats uri / stats auth zadara:zadara
Enable HAProxy logging (Optional)
sudo vi /etc/rsyslog.conf # provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 # provides TCP syslog reception $ModLoad imtcp $InputTCPServerRun 514
Restart the service:
sudo service rsyslog restart
Restart HAProxy service:
sudo service haproxy restart
Monitor statistics by browsing to
http://**<HAProxy server IP>**:1936/Credentials: