Informations in this document mainly come from a document made by Leonardo Ciociano

Racoon is an Internet Key Exchanger (IKE). Racoon job it to automatically negotiate the keys that are going to be used to encrypt traffic. This could be accomplished with PreShared Keys (PSK), X.509 Certificates, or Kerberos. The daemon could use different methods of PSK exchange. Main Mode, Aggresive Mode (insecure) or Base Mode for IKE first phase.

Remember: If Kernel 2.6.10 is being used, ipsec-tools 0.5 will be required.

The easiest way to connect two servers with IKE, is PSK method. PSK method bases its authentication on a shared password (stored on file /etc/racoon/psk.txt - always with 400 perms). Once both ends are authenticated, encryption keys are negotiatied.

This file is organized in columns. First column has a server identity, everything next to first column is the PSK.

# IPv4 Adressen
192.168.2.100          simple psk
5.0.0.1                0xe10bd52b0529b54aac97db63462850f3
# USER_FQDN
ralf@spenneberg.net    This is a psk for an email address
# FQDN
www.spenneberg.net     This is a psk

Racoon configuration should be something like this (/etc/racoon/racoon.conf):

path pre_shared_key "/etc/psk.txt";

remote 192.168.2.100 {
        exchange_mode main;
        proposal {
                encryption_algorithm aes;
                hash_algorithm sha256;
                authentication_method pre_shared_key;
                dh_group modp4096;
        }
}

sainfo address 172.16.1.0/24 any address 172.16.2.0/24 any {
        pfs_group modp4096;
        encryption_algorithm aes;
        authentication_algorithm hmac_sha1;
        compression_algorithm deflate;
}

First line in this file is where racoon should find PSK.

Then, it defines an end as 192.168.2.100 and the paremeters to use on the Phase 1 IKE negotiation. Second Block specifies the parameters for the SA setting. Also encryption, authentication and compression algorithms are defined. It is imperative to define all three algorithms to avoid errors.

Racoon Daemon does not start tunnel negotiation at startup. It waits till the tunnel is needed. For this notification occur, kernel needs to know when to notify racoon. So, we need to define security policies without SA. When the kernel needs to protect a packet, defined in the security policies, and the SA are not available, the kernel calls racoon requesting the required SA. Racoon will start an IKE negotiation, and will create the SA. The kernel, then, will send the packet.

We need the following policies for this configuration.

#!/usr/sbin/setkey -f
#
# Flush SAD and SPD
flush;
spdflush;

# Create policies for racoon
spdadd 172.16.1.0/24 172.16.2.0/24 any -P out ipsec
           esp/tunnel/192.168.1.100-192.168.2.100/require;

spdadd 172.16.2.0/24 172.16.1.0/24 any -P in ipsec
           esp/tunnel/192.168.2.100-192.168.1.100/require;

Once the policies are loaded with 'setkey -f /etc/setkey.conf' command, racoon can start.

For testing racoon daemon, it could be started on Foreground mode: racoon -F -f /etc/racoon/racoon.conf

Again, the other end configuration, should reflect the direction of the packets. Also, IP addresses should be checked on files /etc/racoon/psk.txt, /etc/setkey.conf and /etc/racoon/racoon.conf.