From securityrouter.org, an OpenBSD-based firewall
Revision as of 09:10, 7 September 2015 by Anders (talk | contribs)
Jump to: navigation, search

Internet Protocol Security (IPsec) is a protocol suite for securing Internet Protocol (IP) communications by authenticating and encrypting each IP packet of a communication session. This article mainly deals with the configuration of so-called site-to-site VPN. For remote client VPN, please see the VPN server page. When configuring IPsec, you can choose between

  • Automatic keying (IKE)
  • Manual keying

Today, IKE is the most common method because it automatically rotate keys, in order to ensure that the encryption is not compromised. The disadvantage is that the relatively complicated protocol can result in unstable tunnels in case of minor misconfiguration or vendor incompatibility. Manual keys have the advantage of being statically configured; once you successfully configured a tunnel, it "cannot" go down by itself (there's really nothing that can fail). In other words; use IKE if you need maximum security without having to think about key rotation, and use manual keys if you need maximum stability and vendor interoperability.

IKE Manual key
Security Very good Good
Replay protection[1] Yes No
Key rotation Automatic Manual
Ease of use Poor Good
Stability Mixed Good
Vendor interoperability Mixed Good

Regardless of which tunnel type you choose, you may need configure the firewall to allow the traffic as well. However, the default configuration allows the traffic on the encapsulation (enc0) interface and interfaces in the lan group by default.


Before creating an IKE tunnel, please read the comparison between IKE and manual keying at the top of this page. Some notes on how to properly add a tunnel;

  • Most required fields are coloured red; however, you probably want to use a pre-shared key, typed in the "phase 1" box
  • You can specify several "from" and "to" networks into the fields in the "flow" box, separated by (for example) a space
  • If a local gateway is not specified it will use the primary IP as default

Dynamic address

Sometimes, one of the routers receives a dynamic "WAN" address using for example DHCP or PPPoE. In that case, you can either use Dynamic DNS and create a regular tunnel, or use aggressive mode. In the aggressive mode case, use the following configuration (but substitute the IP addresses) on the router with dynamic address

ike dynamic from to peer X.X.X.X aggressive quick psk "badpassword" ping

and on the other end (with static IP)

ike passive from to aggressive quick psk "badpassword" ping

When setting the mode to dynamic on the router with the dynamic address it also activates DPD (dead peer detection) and uses the local hostname (if configured) as the local ID.

Note that if you have multiple dynamic tunnels going to the same endpoint, they all need to share the same phase 1 settings; such as the pre-shared key (because of the way that IPsec works). Because of this you also need to explicitly set the local ID to be the same on all the dynamic hosts if they have local hostnames assigned to them, otherwise you can leave it as default. Also note that aggressive mode IPSec tunnels clashes with L2TP (because it also uses aggressive mode IPsec). Our recommendation is not to mix L2TP with dynamic tunnels.


  • The IKE identities are important; phase 1 IDs defaults to the external (gateway) IPs (but can be changed) whereas phase 2 IDs always equals to your flows
    • When you use single hosts as "from" or "to", please notice that IP addresses ( and single-node networks ( differs in terms of IKE identity
    • When you specify multiple "from" or "to" networks, beware that it affects the phase 2 IDs, and the order in which you type the networks might be of relevance
    • You might experience NAT-T version/implementation differences, try disabling it in the other end's configuration
  • Because tunnel establishment and rekeying/keep alive differs, tunnels might establish but anyway behave in an unstable manner
    • The easy way out is to switch to manual keys if supported by the other endpoint
    • Try establishing the connection from the other end; make the endpoint that usually initiates passive and check for configuration mismatches
    • If dealing with unstable connections or firewalls with old or buggy software; try disabling DPD (dead peer detection) on the other end, or enable it in the Halon system by adding DPD-check-interval=30 to the [General] section of the settings file.
    • If you see messages from either side of the tunnel that say "Invalid cookie" in the IKE debug logs it means that one of the sides is using an SA thas is no longer in use, try resetting the tunnel on both sides. You can reset the tunnel on the Halon by restarting the IKE daemon.

There are several IKE debugging options on the Network > IPsec page. For example, you can view the IKE log, view phase 1 or decrypted phase 2 packets, enable debugging, etc. You can manually specify filters for the IKE packet dumping, by specifying filters such as

tcpdump -i em0 -npvvs 1530 host

where em0 should be replaced with the interface that the IKE traffic originates from (usually the WAN port) and should be the other endpoint.

Vendor interoperability

Cisco ASA

When setting up an IPSec IKE tunnel between a Halon SR and some models of Cisco ASA (Cisco ASA 5505, for example) you can sometimes run into issues where even though the settings are correct, the tunnel still doesn't go up. On reason for this can be that they are using different versions/implementations of NAT-T, try disabling NAT-T on the Cisco ASA and see if the tunnel goes up.

Restarting IKE

Normally, the tunnels are added/deleted/started/stopped individually; for example by removing and "test" committing, or using the isakmpctl CLI program. However, sometimes you might feel like restarting the entire IKE daemon (called isakmpd), perhaps in order to completely reset its internal state. If using the web administration, It can be achieved by removing all tunnels (most efficiently performed using the plain-text configuration editor) and then "Test" deploying the configuration. The corresponding CLI commands are

> configure
# delete ipsec {
# commit -time 1

Advanced settings

Some settings can only be changed in the skeleton files using the root access. The IKE daemon's configuration file[2] is

vi /cfg/skel/isakmpd/isakmpd.conf

and uses a so-called INI syntax. Make sure to run

chmod 600 /cfg/skel/isakmpd/isakmpd.conf

and then restarting the IKE daemon after creating or changing the file.

Manual keys

Before creating a manual key tunnel, please read the comparison between IKE and manual keying at the top of this page. Some notes on how to properly add a tunnel;

  • The "Local gateway" field should be the firewall's own WAN IP address
  • The SPI values are decimal by default, but can be typed as "0x1000" (for hexadecimal 1000) if desired
  • The SPI values should be "swapped" between each site; the outgoing on one side should be the incoming on the other side
  • Some vendors only offers one SPI value; if so, type the same value on both fields
  • If you have several tunnels, make sure they are all using different SPI values
  • Press the "key" icons right to the authentication and encryption key fields to automatically generate high-quality encryption keys and then copy those to the other side
  • If populated manually, the authentication and encryption key fields should have hexadecimal values starting with "0x"
  • Most required fields are coloured red; however, you should probably press the plus (+) icon in the lower right corner to add at least one "out" flow (specifying the "from" and "to" networks)


ZyXEL's ZyWALL firewalls can use manual keys, but you might have to press an "advanced" button. They have only one SPI value, which is decimal. Therefore, if you type 256 in the ZyWALL, type 256 or 0x100 in both incoming and outgoing in the Halon. The ZyWall uses ASCII keys, which complicates things. First of all, you cannot convert from hexadecimal to ASCII, because not every possible hex "character" is a printable ASCII character. Secondly, ZyWALL doesn't have a key generator. Consequently, you need to somehow "come up with" high-quality random ASCII keys of the correct length, and then convert them to hex.

Setting Example of ZyWALL value Corresponding in Halon
Authentication or encryption key kkkkkkkkkkkkkkkk 0x6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b
SPI 256 0x100 (or 256, both incoming and outgoing)

Please note that the key above is extremely bad, and should never be used.


Creating a manual key tunnel with a Sonicwall is straightforward. Sonicwall uses hexadecimal, and by using the same values but prefixed with "0x" in the Halon it should work.

Setting Example of Sonicwall value Corresponding in Halon
Authentication or encryption key 6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b 0x6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b
SPI 100 0x100

Please note that the key above is extremely bad, and should never be used.


Creating a manual key tunnel with a Juniper is straightforward. Juniper uses hexadecimal, and by using the same values but prefixed with "0x" in the Halon it should work.

Setting Example of Juniper value Corresponding in Halon
Authentication or encryption key 6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b 0x6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b
SPI 100 0x100

Please note that the key above is extremely bad, and should never be used.

Halon SX

Creating a manual key tunnel with an older Halon SX is straightforward. The only thing to keep in mind is that Halon SR uses decimal by default while Halon SX uses hexadecimal, and by using the same values but prefixed with "0x" in the Halon SR it should work.

Setting Example of Halon SX value Corresponding in Halon SR
Authentication or encryption key 6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b 0x6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b6b
SPI 100 0x100

Please note that the key above is extremely bad, and should never be used.


IPsec VPN throughput is typically slower than plain-text routing/firewalling, because of the encryption overhead. In order to address encryption speed issues, several CPUs offer encryption acceleration (most commonly AES). In the list of supported hardware, several systems with AES acceleration capabilities are mentioned, even some of the low end $100-systems. If IPsec performance is important to you; choose a system with AES acceleration, and make sure you use AES as encryption algorithm.

Intel AES-NI

Intel's AES-NI[3] is becoming increasingly common, and is present in most servers sold today. We benchmarked two standard servers with AES-NI capable Xeon CPUs doing IPsec tunneling between two low-end computers (Pentium 4) with Linux and Iperf[4] (with standard configuration and without extra command-line options) in order to show the importance of choosing the right algorithms. With 3DES (non-accelerated), we achieved 110 Mbps. AES-128 gave 650 Mbps. AES-128-CGM gave 720 Mbps. The differences are quite drastic.


You can monitor the traffic that goes over the tunnels by clicking on the "Encrypted traffic" button on the IPsec page. You can filter the traffic that you're monitoring by their encapsulated IP by using the following command (From the terminal):

tcpdump -i enc0 | grep "XX.XX.XX.XX"

where "XX.XX.XX.XX" is the encapsulated IP that you want to filter for.

You can manually ping a computer on the other side of the tunnel from the Halon by specifying the correct source IP with the ping command:


where the first IP-address is the local IP of the Halon (for that particular tunnel) and the other IP is the local IP on the other side that you want to ping.

Example setups

There are, as always with networking, many ways of accomplishing the same thing. Below are a few examples.

Failover with OSPF

In order to establish multiple redundant tunnels, you can use OSPF and GRE over IPsec.

interface gre0 {
	address OSPF1_ME/30 OSPF1_REMOTE
	firewall {
interface gre1 {
	address OSPF2_ME/30 OSPF2_REMOTE
	firewall {
interface lo1 {
	address TRANSPORT1_ME/32
	address TRANSPORT2_ME/32
	firewall {
ipsec {
	ike from TRANSPORT1_ME to TRANSPORT1_REMOTE peer REMOTE_GW1 psk "secret" label tunnel1
	ike from TRANSPORT2_ME to TRANSPORT2_REMOTE peer REMOTE_GW2 psk "secret" label tunnel2
ospf {
	redistribute default
	area {
		interface gre0 {
		interface gre1 {
		interface LAN {

Failover with DPD and scripted checks

If dynamic routing (such as OSPF or BGP) is unavailable, you can use shell scripting with the configure command by enabling root access and skeleton files. Below is an example of such as script, created as /cfg/whatever.sh. In the example, we check for the presence of some part of an IP address in the list of established SAs. The check could of course be something else, such as pinging. We recommend enabling dead peer detection (DPD) for this setup to work.

is_up() {
	ipsecctl -ss | grep 1.2.3
	return $?
while true
	sleep 10

	# if tunnel is up, don't do anything
	is_up && continue

	# determine which gateway is active
	configure -- show ipsec | grep $gw1 && new=$gw2 && old=$gw1
	logger Tunnel is down, try changing gateway

	# make the switch
	id=`configure -- edit ipsec -- replace peer $old with peer $new -- commit -time 60 Change $old to $new`
	[ $? -ne 0 ] && continue

	# we have test commit, allow 30 seconds for tunnel to establish
	sleep 30

	# if tunnel is up, confirm (save) the commit
	is_up && configure -- commit-confirm $id

It should be started on boot by adding the following to /cfg/skel/rc.local

sh /cfg/whatever.sh > /dev/null &

SNMP traps

There are several ways to determine the state of an IPsec tunnel, but in our experience the most accurate indication is a ping. If DPD is enabled for the tunnels, the phase 2 SAD list can be queried instead of ping. Begin by enabling root access and create a file such as /cfg/ipsectrap.sh (which you auto-start using skeleton files) which uses snmpctl to send a trap. The example below is equivalent to what the web administration does when presenting the (red/green) status on tunnels.

while true
		ping -q -w1 -c1 -I SOURCEIP $IP && continue
		# ipsecctl -s sa | grep $IP && continue
		echo fail $IP
		snmpctl trap send SNMP_IOD
	sleep 10