VPSA Object Storage is created by default with internal load balancer to distribute object operations between the proxy virtual controllers (VCs). Any VPSA Object Storage is provisioned with a built-in Load Balancer (referred as Internal Load Balancer) solution which will work seamlessly out of the box. For large (multi-PB) deployments, the VPSA Object Storage administration may want to consider using either:
Dedicated Load Balancer (software/hardware) appliance in-front of the Object Storage in order to gain full control on the balancing mechanism, traffic policy and security protocols.
Integrate the application directly with the VPSA Object Storage.
A multi-PB VPSA Object Storage deployments will handle dozens or even hundreds of Virtual Controllers which will be challenging to manage using an external Load Balancer or via application integration. In order to simplify the management overhead it is possible to enable a dedicated Load Balancer (Elastic Load Balancer).
ZELB (Zadara Elastic Load Balancer)¶
Elastic Load Balancer will break down the Virtual Controllers into groups of 12 VCs. For each Elastic Load Balancer Group (ZELBG) two additional HA Proxy VCs will be provisioned.
cluster scales out and connections load is increasing, switching to an external load balancer might be required.
VPSA Object Storage provides a very smooth and easy way to switch between internal and external load balancer setups.
Internal Load Balancer runs in one of the VC’s
External Load Balancer(s) runs outside of the VPSA Object Storage VC’s
Below are instructions for setting up an external load balancer to terminate SSL connections and distribute the load over all VC’s.
There are many load balancer solutions in the market, setting them all up is quite similar procedure. This appendix gives an example of HAproxy, an open-source TCP/HTTP load-balancing proxy server that can be found in www.haproxy.org
The recommended configuration below will allow the following:
SSL Termination is done on the external load balancer for both object operation API’s and GUI connections. Authentication connections are always terminated in Object Storage.
Custom SSL certificate (PEM) located on the load balancer is used for SSL connections
Object operation connections are redirected to VPSA Object Storage proxy VC’s
Object operation connections are distributed between VC’s unevenly (proxy-only VCs to take more load than storage VC’s, and HA VCs to take the lowest load)
Redirected object operation connections will include the original client IP in a special header added by the load balancer (for logging in VPSA Object Storage proxy)
HTTP-based health check is performed by the load balancer to probe all VPSA Object Storage proxy VC’s
Authentication connections are redirected to ZObject Storage floating IP (SSL pass-through terminated on the VPSA Object Storage, Custom SSL certificate must be uploaded to the VPSA Object Storage as well).
GUI connections are redirected to the VPSA Object Storage floating IP
Graphical statistics interface is enabled on the load balancer
Apply the following configuration to your VPSA Object Storage Settings:
Set the internet-facing domain-name/IP of the external LoadBalancer as VPSA Object Storage API Hostname / IP (zadara-qa.com which resolves to the external LB IP 126.96.36.199, is set in this example as VPSA Object Storage API Hostname)
Upload your custom SSL certificate (will be used for authentication connections). The certificate should match the custom domain name.
Set SSL Termination to “External”
HAProxy Installation and configuration instructions:
sudo add-apt-repository -y ppa:vbernat/haproxy-1.5 sudo apt-get update sudo apt-get install -y haproxy
- Upload your custom SSL certificate to HAProxy server. In this example
the certificate PEM file is placed under /etc/ssl/private/zadara_custom.pem
Edit /etc/haproxy/haproxy.cfg to include the following:
| global | maxconn 2048 | log /dev/log local0 | log /dev/log local1 notice | chroot /var/lib/haproxy | stats socket /run/haproxy/admin.sock mode 660 level admin | stats timeout 30s | user haproxy | group haproxy | daemon | tune.ssl.default-dh-param 2048 | # Default SSL material locations | ca-base /etc/ssl/certs | crt-base /etc/ssl/private | # Default ciphers to use on SSL-enabled listening sockets. | # For more information, see ciphers(1SSL). This list is from: | # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ssl-default-bind-ciphers | ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS | ssl-default-bind-options no-sslv3 | | defaults | log global | mode http | option httplog | option dontlognull | timeout connect 5000 | timeout client 50000 | timeout server 50000 | errorfile 400 /etc/haproxy/errors/400.http | errorfile 403 /etc/haproxy/errors/403.http | errorfile 408 /etc/haproxy/errors/408.http | errorfile 500 /etc/haproxy/errors/500.http | errorfile 502 /etc/haproxy/errors/502.http | errorfile 503 /etc/haproxy/errors/503.http | errorfile 504 /etc/haproxy/errors/504.http | frontend fe-object-operations | bind 188.8.131.52:443 ssl crt /etc/ssl/private/zadara\_custom.pem | mode http | default\_backend be-zios-object-operations | | frontend fe-auth | bind 184.108.40.206:5000 | option tcplog | mode tcp | default\_backend be-floating-zios-auth | | frontend fe-gui | bind 220.127.116.11:8443 ssl crt /etc/ssl/private/zadara\_custom.pem | mode http | default\_backend be-floating-zios-gui | | backend be-zios-object-operations | mode http | balance roundrobin | option forwardfor | option httpclose | option httpchk HEAD /healthcheck HTTP/1.0 | server ziosStorageProxy0 18.104.22.168:8080 weight 10 check | server ziosStorageProxy1 22.214.171.124:8080 weight 10 check | server ziosStorageProxy2 126.96.36.199:8080 weight 50 check | server ziosProxyOnly3 188.8.131.52:8080 weight 100 check | server ziosProxyOnly4 184.108.40.206:8080 weight 100 check | | backend be-floating-zios-auth | mode tcp | server ziosFloating 220.127.116.11:5000 | | backend be-floating-zios-gui | mode http | server ziosFloating 18.104.22.168:80 | | listen stats \*:1936 | stats enable | stats uri / | stats auth zadara:zadara
Enable HAProxy logging (Optional)
Edit rsyslog conf:
| sudo vi /etc/rsyslog.conf | # provides UDP syslog reception | $ModLoad imudp | $UDPServerRun 514 | # provides TCP syslog reception | $ModLoad imtcp | $InputTCPServerRun 514
Restart the service:
sudo service rsyslog restart
Restart HAProxy service:
sudo service haproxy restart
Monitor statistics by browsing to http://<HAProxy server IP>:1936/ Credentials: zadara/zadara