Appendix A: Setting External Load-Balancer

VPSA Object Storage is created by default with internal load balancer to distribute object operations between proxy virtual controllers (VC’s). When VPSA Object Storage cluster scales out and connections load is increasing, switching to an external load balancer might be required.

VPSA Object Storage provides a very smooth and easy way to switch between internal and external load balancer setups.

image76

Internal Load Balancer runs in one of the VC’s

image77

External Load Balancer(s) runs outside of the VPSA Object Storage VC’s

Below are instructions for setting up an external load balancer to terminate SSL connections and distribute the load over all VC’s.

There are many load balancer solutions in the market, setting them all up is quite similar procedure. This appendix gives an example of HAproxy, an open-source TCP/HTTP load-balancing proxy server that can be found in www.haproxy.org

The recommended configuration below will allow the following:

  • SSL Termination is done on the external load balancer for both object operation API’s and GUI connections. Authentication connections are always terminated in Object Storage.

  • Custom SSL certificate (PEM) located on the load balancer is used for SSL connections

  • Object operation connections are redirected to VPSA Object Storage proxy VC’s

  • Object operation connections are distributed between VC’s unevenly (proxy-only VCs to take more load than storage VC’s, and HA VCs to take the lowest load)

  • Redirected object operation connections will include the original client IP in a special header added by the load balancer (for logging in VPSA Object Storage proxy)

  • HTTP-based health check is performed by the load balancer to probe all VPSA Object Storage proxy VC’s

  • Authentication connections are redirected to ZObject Storage floating IP (SSL pass-through terminated on VPSA Object Storage, Custom SSL certificate must be uploaded to VPSA Object Storage as well).

  • GUI connections are redirected to VPSA Object Storage floating IP

  • Graphical statistics interface is enabled on the load balancer

Apply the following configuration to your VPSA Object Storage Settings:

  1. Set the internet-facing domain-name/IP of the external LoadBalancer as VPSA Object Storage API Hostname / IP (zadara-qa.com which resolves to the external LB IP 180.80.2.217, is set  in this example as VPSA Object Storage API Hostname)

  2. Upload your custom SSL certificate (will be used for authentication connections). The certificate should match the custom domain name.

  3. Set SSL Termination to “External”

image78

HAProxy Installation and configuration instructions:

  • Install HAProxy:

    sudo add-apt-repository -y ppa:vbernat/haproxy-1.5

sudo apt-get update

sudo apt-get install -y haproxy
  • Upload your custom SSL certificate to HAProxy server. In this example

    the certificate PEM file is placed under /etc/ssl/private/zadara_custom.pem

  • Edit /etc/haproxy/haproxy.cfg to include the following:

| global
|    maxconn 2048
|    log /dev/log    local0
|    log /dev/log    local1 notice
|    chroot /var/lib/haproxy
|    stats socket /run/haproxy/admin.sock mode 660 level admin
|    stats timeout 30s
|    user haproxy
|    group haproxy
|    daemon
|    tune.ssl.default-dh-param 2048
|    # Default SSL material locations
|    ca-base /etc/ssl/certs
|    crt-base /etc/ssl/private
|    # Default ciphers to use on SSL-enabled listening sockets.
|    # For more information, see ciphers(1SSL). This list is from:
|    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ssl-default-bind-ciphers
|    ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
|    ssl-default-bind-options no-sslv3
|
| defaults
|    log     global
|    mode    http
|    option  httplog
|    option  dontlognull
|    timeout connect 5000
|    timeout client  50000
|    timeout server  50000
|    errorfile 400 /etc/haproxy/errors/400.http
|    errorfile 403 /etc/haproxy/errors/403.http
|    errorfile 408 /etc/haproxy/errors/408.http
|    errorfile 500 /etc/haproxy/errors/500.http
|    errorfile 502 /etc/haproxy/errors/502.http
|    errorfile 503 /etc/haproxy/errors/503.http
|    errorfile 504 /etc/haproxy/errors/504.http
|    frontend fe-object-operations
|    bind 180.80.2.217:443 ssl crt /etc/ssl/private/zadara\_custom.pem
|    mode http
|    default\_backend be-zios-object-operations
|
| frontend fe-auth
|    bind 180.80.2.217:5000
|    option tcplog
|    mode tcp
|    default\_backend be-floating-zios-auth
|
| frontend fe-gui
|    bind 180.80.2.217:8443 ssl crt /etc/ssl/private/zadara\_custom.pem
|    mode http
|    default\_backend be-floating-zios-gui
|
| backend be-zios-object-operations
|    mode http
|    balance roundrobin
|    option forwardfor
|    option  httpclose
|    option  httpchk HEAD /healthcheck HTTP/1.0
|    server ziosStorageProxy0 190.90.2.102:8080 weight 10 check
|    server ziosStorageProxy1 190.90.2.104:8080 weight 10 check
|    server ziosStorageProxy2 190.90.2.114:8080 weight 50 check
|    server ziosProxyOnly3 190.90.2.106:8080 weight 100 check
|    server ziosProxyOnly4 190.90.2.109:8080 weight 100 check
|
| backend be-floating-zios-auth
|    mode tcp
|    server ziosFloating 190.90.2.118:5000
|
| backend be-floating-zios-gui
|    mode http
|    server ziosFloating 190.90.2.118:80
|
| listen stats \*:1936
|    stats enable
|    stats uri /
|    stats auth zadara:zadara

  • Enable HAProxy logging (Optional)

    1. Edit rsyslog conf:

      | sudo vi /etc/rsyslog.conf
| # provides UDP syslog reception
| $ModLoad imudp
| $UDPServerRun 514
| # provides TCP syslog reception
| $ModLoad imtcp
| $InputTCPServerRun 514

    2. Restart the service:

      sudo service rsyslog restart

  • Restart HAProxy service:

    sudo service haproxy restart

  • Monitor statistics by browsing to http://<HAProxy server IP>:1936/ Credentials: zadara/zadara