Load balancing

From securityrouter.org, an OpenBSD-based firewall
Revision as of 09:33, 21 November 2012 by Erik (talk | contribs) (Net-SNMP guide)
Jump to: navigation, search

The load balancer can dynamically redirect and route traffic.

Introduction

The load balancer can operate as

  • Load balancer
  • Application layer gateway
  • SSL accelerator
  • Transparent proxy
  • Internet failover

It's most commonly used to forward traffic to multiple servers with load distribution and health checking. This functionality can, with some generalization, be divided into layer 3 (called redirects) and layer 4+ (called relays). They are configured in much the same way, but have some striking technical differences. Both the redirects and relays are sometimes referred to as "virtual servers" in other products.

Layer 3 (redirects)

Layer 3 load balancing is implemented as a firewall port forward, which is very efficient, and therefore gives layer 3 load balancing a performance advantage over layer 4+. Because of its implementation, it requires the firewall to be started (simply check if there are any firewall rules in the configuration). Also because if this, it's normally not necessary to add any additional firewall rules. Further, it's necessary that the load balancer is the default route (gateway) of the servers being load balanced, since the source packets of the incoming traffic is maintained. In order words, the topology would look something like:

Internet -> Halon -> Switch -> Server1
                            -> Server2
                            -> Server3                

To add a layer 3 load balancer, follow these steps

  1. Check that the firewall is enabled by looking for a firewall statement in the configuration, or rules on the Network > Firewall page (if not, add a rule)
  2. Go to the Network > Load balancer page, click the "Pools (tables)" button and then click "Add pool"
  3. Just give it a name, type an IP address in the table's node column (add multiple hosts by pressing the + icon to the right in the table header) and finally click "Apply"
  4. If you like, click the "Node (hosts)" button and give the IP addresses names
  5. Click the "Relays (servers)" button, click "Add..." and select "Redirect (layer 3)"
  6. Just give it a name, listen address (the external virtual IP), listen port, forward pool (the pool you created in step 3) and then click "Apply"
  7. Click the "Cancel" button, then click the newly added redirect in the table, and watch the health check status

Redirect sticky address

Some protocols need sticky-session in order to load-balance properly, like FTP where the control- (21) and data-connection must end up on the same backend server, or web-sessions which doesn't have a shared session store. The load-balancer uses the firewalls source tracking cache to keep track of previous redirects from a particular source address (IP). This cache has two parameters, one of which you may need to change primarily for web applications.

In the configuration editor, you may set these values in the firewall { section.

set timeout src.track X
set limit src-nodes Y

src.track defines the number of seconds a src-track entry is stored in memory after the last active state has expired. By default this feature is not used. For protocol like FTP this is exactly what you want. However HTTP may suffer from bad behavior if this value doesn't match you web applications session timeout (eg. PHP session timeout). Do not set this value higher than what is necessary.

src-nodes defines how many src-track (nodes) entries are stored in memory. By default 10000. This value should not be changed if you do not hit this limitation.

Caveats

Source tracking is stored per-rule. Thus in the load-balancer each redirect has it's own source tracking. So in order to load-balance FTP both listen on has to be added to the same redirect.

load-balancer {
    table <ftp_servers> { 10.2.0.30 10.2.0.31 }
    redirect "FTP" {
         listen on 1.2.3.4 port 21
         listen on 1.2.3.4 port 5000:5100
         forward to <ftp_servers> check tcp
         sticky-address
    }
}

Layer 4+ (relays)

Unlike redirects, relays doesn't require the firewall to be enabled, nor that the load balancer is the servers' default route (gateway). In fact, the most common scenario is to perform layer 4 load balancing with the firewall disabled (removing the firewall configuration) and to not use the load balancer as a router. Please keep in mind that if the firewall is enabled (simply check if there are any firewall rules in the configuration), you most likely have to add rules on your own allowing traffic to the relay. The topology would look something like:

Internet -> Router -> Switch <-> Halon
                              -> Server1
                              -> Server2                

To add a layer 4+ load balancer, follow these steps

  1. Check if the firewall is enabled by looking for a firewall statement in the configuration, or rules on the Network > Firewall page (if it is, you might have to add rules)
  2. Go to the Network > Load balancer page, click the "Pools (tables)" button and then click "Add pool"
  3. Just give it a name, type an IP address in the table's node column (add multiple hosts by pressing the + icon to the right in the table header) and finally click "Apply"
  4. If you like, click the "Node (hosts)" button and give the IP addresses names
  5. Click the "Relays (servers)" button, click "Add..." and select "Relay (layer 4+)"
  6. Just give it a name, listen port, forward pool (the pool you created in step 3) and then click "Apply"
  7. Click the "Cancel" button, then click the newly added relay in the table, and watch the health check status

Configuration examples

Below are a few clear-text configuration examples. The load balancer is implemented using OpenBSD's relayd and therefore its manual page[1] is useful for clear-text configuration.

HTTPS (SSL) acceleration

This very simple example provides an HTTPS accelerator. If you are using the 64-bit version (amd64) on a router with AES-NI instructions, you can expect gigabit performance. Below is a more or less complete example, using the router exclusively as a layer 7 load balancer, utilizing only one Ethernet interface.

interface em0 {
	address 192.168.0.100/24
	route default 192.168.0.1
}
load-balancer {
	table <servers> { 192.168.0.101 192.168.0.102 }
	relay "webservers" {
		listen on 192.168.0.100 port 443 ssl
		forward to <servers> port 80 check tcp
	}
}
system {
	http-server {
		port 4433
	}
	authentication {
		root-password "extremelyhardpassword"
		user "admin" {
			password "veryhardpassword"
		}
	}
	dns {
		name-server 8.8.8.8
	}
}

Then, upload the certificate and private key. Currently, these are not in the configuration file. Instead, enable root access (already enabled by the above example) and upload the file using for example sep according to the skeleton files guidelines. You can also try out the load balancer by using the web administration's self-signed certificate, by issuing the following commands when logged in as root:

# cp /etc/ssl/server.crt /etc/ssl/192.168.0.100.crt
# cp /etc/ssl/private/server.key /etc/ssl/private/192.168.0.100.key

Internet failover

The load balancer can be used to select one of several default routers (gateways) which is useful for outbound internet failover when more sophisticated protocols such as BGP is unavailable. Below is an incomplete example of such a a configuration.

load-balancer {
	gw1 = 212.37.18.193
	gw2 = 213.12.48.1
	table <gateways> { $gw1 ip ttl 1, $gw2 ip ttl 1 }
	router "internetfailover" {
		route 0.0.0.0/0
		forward to <gateways> check icmp
	}
}

SSL stripping

If you would like to strip the SSL from a service that is only available over SSL (eg. the web administration, even thou it's not recommended nor good practice), this example shows how to make the web administration available for unsecure HTTP connections.

load-balancer {
	relay "webui" {
		listen on 0.0.0.0 port 80
		forward with ssl to 127.0.0.1 port 443
	}
}

SNMP traps

The preferred way of creating notifications is using SNMP traps; notifying all trap receivers about hostUp and hostDown event. Configure a trap receiver on the System > SNMP page, and then enable SNMP traps on the Network > Load balancer page. In raw configuration, this equals to

load-balanacer {
   send traps
   ...
}
system {
   snmp-server {
      trap receiver X.X.X.X
   }
   ...
}
...

The relayd daemon sends a enterprise.30155.3.1 (relaydStateChange), with descriptive object values. Please note that this MIB file's layout is not final and may be revised in the future. The MIB file is available from the Network > Load balancer page.

+--openBSD(30155)
   |
   +--relaydMIBObjects(3)
      |
      +--relaydStateChange(1)
         +-- -R-- String    hostName(1)
         +-- -R-- EnumVal   hostStatus(2)
         |        Values: hostDown(-1), hostUnknown(0), hostUp(1)
         +-- -R-- EnumVal   hostLastStatus(3)
         |        Values: hostDown(-1), hostUnknown(0), hostUp(1)
         +-- -R-- INTEGER   checksSuccessful(4)
         +-- -R-- INTEGER   checks(5)
         +-- -R-- String    tableName(6)
         +-- -R-- INTEGER   tableHosts(7)
         +-- -R-- INTEGER   hostRetriesLeft(8)
         +-- -R-- INTEGER   hostRetries(9)

Net-SNMP guide

This short guide describes how to set up a trap receiver and run a custom script when a relayd host is up/down. SNMP authentication and security is out of scope for this guide.

Install the HALON-RELAYD-TRAP-MIB.txt file to your mibs/ folder.

Create a configuration file for snmptrap (snmptrapd.conf). Run whatever script/interpreter you prefer (eg. PHP).

 authCommunity log,execute,net public
 traphandle HALON-RELAYD-TRAP-MIB::relaydStateChange /usr/local/bin/php /opt/trap.php

Start snmptrapd.

Example trap.php script file.

<?php
 $result = array();
 $result['host'] = fgets(STDIN);
 $result['ip'] = fgets(STDIN);
 while(!feof(STDIN)) {
  list($type, $value) = explode(' ', trim(fgets(STDIN)), 2);
  if ($type == "") break;
  $result[$type] = $value;
 }
 file_put_contents(var_export($result, true), "/tmp/trap-output");
?>