Shorewall and Proxmox VE Cluster Configuration

This is a follow up article describing how to use Proxmox VE and Shorewall together. This article focus on using Shorewall within your Proxmox cluster. If you have not read the first article I recommend that you do so, it will aid your understanding with what is going in this one.

Network Layout and Shorewall Configuration

We are going to be using a bridging configuration. This is what Proxmox VE uses with by default. Bridging allows for easy migration of hosts without having to re-configure the firewall each time a machine is migrated.

The default Proxmox network configuration looks similar to this:
/etc/network/interfaces

iface eth0 inet manual

auto vmbr0 
iface vmbr0 inet static
	address 192.0.2.10
	netmask 255.255.255.0
	gateway 192.0.2.1 
	bridge_ports eth0
	bridge_stp off 
	bridge_fd 0

Limitations

Shorewall 4.4.3 will only support one bridge device. This is not a huge drawback and add several advantages. The first advantage that having a single bridge device adds is a more abstracted network layout. You have cluster nodes that have a different number of network interfaces and still be free to migrate between them and know that your virtual machine will continue to work. With a cluster setup this is a very important property. Proxmox allows you to bond multiple NICs together so that they look as one logical interface. I personally use 802.3ad bonding mode because my switches support it. This bonding mode gives you increased throughput and redundancy. Other bonding modes will work as well. If you think you need multiple bridges you don't; there is a solution using VLANs. We will discuss VLANs more later in this article.

Installing Shorewall

In the last article we just used the Shorewall version that was in the repos; Shorewall version 4.0.15. That version of Shorewall does not have all the features we need. Thankfully the package maintainer has provided an up to date package on his personal site. Intrusions on how to install that can be found here and work perfectly with Proxmox VE.
The configuration files for Shorewall are stored in the /etc/shorewall/ directory. To get started, copy the example configuration files from the ones installed with the Shorewall package.
cp /usr/share/doc/shorewall/default-config/* /etc/shorewall/
cp /usr/share/doc/shorewall/examples/two-interfaces/* /etc/shorewall/

These files give you a good base to begin customizing the firewall to meet your needs.
Configuring Shorewall

First thing is to edit the shorewall.conf file. You want to enable IP forwarding. Change it from Keep to On. Also make sure that startup is enabled.

IP_FORWARDING=On
STARTUP_ENABLED=Yes

Then edit the zones file:
fw		firewall 
world		ipv4 
net:world	bport 
dmz:world	bport 
vz		ipv4 

The zones file defines what zones are available so that you can make rules and policies based on them.

Next edit the interfaces file:

world		vmbr0			detect		bridge 
net		vmbr0:bond0 
dmz		vmbr0:vmtab+ 
vz		venet0		detect		routeback

This file just tells Shorewall what interfaces are connected to what zones. This also allows you to make policies based where traffic is traveling to or from. The policy file is where you define what those policies are. In this article the following polices will be defined.

Traffic from the firewall:
1.To the Internet is permitted
2.From the Internet is prohibited

Traffic from the DMZ:
1.To the Internet is permitted
2.To the firewall is prohibited

# Internet Connections 
dmz		net		ACCEPT 
vz		world		ACCEPT 

# Allow FW to use internet 
$FW		world		ACCEPT 
net		all		DROP		info 
# THE FOLLOWING POLICY MUST BE LAST 
all		all		REJECT	info

Next we need to define rules. Rules are the exceptions to the policies defined above. We are just going to create a very basic set. You can easily expand on these on your own.

#	Accept SSH connections for administration 
# 
SSH(ACCEPT)	net		$FW 
SSH(ACCEPT)	dmz		$FW 
SSH(ACCEPT)	vz		$FW 

# Permit access to Proxmox Manager and Console 
ACCEPT		net		$FW	tcp	5900:5999 
HTTPS(ACCEPT)	net		$FW 
HTTP(ACCEPT)	net		$FW 

# 
#	Allow Ping from the local network 
# 
Ping(ACCEPT)	dmz		$FW 
Ping(ACCEPT)	vz		$FW 
Ping(ACCEPT)	net		dmz 
Ping(ACCEPT)	net		vz 

# 
# Drop Ping from the "bad" net zone.. and prevent your log from being flooded.. 
# 
Ping(DROP)	net		$FW 
# 
#
#	VMID:	120 
#	Type:	KVM 
#	Name:	test0.example.com 
#	IP:	192.0.2.11 
# 
HTTP(ACCEPT)		net		dmz:192.0.2.11

#       VMID:   113 
#       Type:   OpenVZ 
#       Name:   test1.example.com 
#       IP:     192.0.2.12 
# 
HTTP(ACCEPT)		net		vz:192.0.2.12

This set of rules allows SSH and Ping to work from the net zone. The last part of the file shows a web server running at 192.0.2.11 and at 192.0.2.12. As you can see the syntax is slightly different for a KVM machine versus a OpenVZ machine.

We then need to make some changes from the default two interface setup. We need to edit the masq file and change it to fit your environment. I am not using masquerading so I removed the entries in that file. We also need to edit the routestopped file again customize this to your environment. See http://shorewall.net/Documentation.htm#Routestopped for more information.
The last things to do are to check your syntax using: shorewall check and to try your firewall out using: shorewall try /etc/shorewall 60 The first command just does basic checks of your syntax and shows any errors. The second command apply your Shorewall configuration for 60 seconds for you try out make sure that everything works as expected. If you lock yourself out you just have to wait until the 60 seconds is over and then you are able to get back in. Once you are satisfied that everything is working edit the /etc/default/shorewall file and change startup=0 to startup=1.

Advanced Networking

The issue with having more than one bridge with Proxmox VE and Shorewall is all the bridges generated have names that all start off the same way. There is a patch to Shorewall that allows you to define multiple bridges as discussed here: http://tinyurl.com/ykb33jr However this is not an ideal solution. The ideal solution would be to change the way Proxmox names the generated bridge ports to reflect what bridge on the host they are connecting to. Another workaround is to just define VLANs and use those VLANs within your virtual machine. Defining VLANs is distro dependent so I won't go into it but by using VLANs you are able to migrate your virtual machine to any cluster node and have it work as long as your switch port is part of those VLANs. Also as of Shorewall version 4.4.4 it is possible to specify multiple bridges.

Applying the Firewall Cluster Wide
Keeping your configuration files in sync cluster wide can easily be done with an rsync script or even just with scp. If there is interest I will write part 3 that keeps your Shorewall files updated with Puppet (http://reductivelabs.com/trac/puppet/). So leave me a comment to see that.
Final Notes
The final gotcha is you need to restart the firewall every time a virtual machine is migrated. I have not found a good way to do this yet but for right now just doing it by hand is working. If anybody has ideas on a way to have it happen automatically I would be interested to hear.

Andrew Niemantsverdriet
Linux Systems Administrator
Rocky Mountain College
andrew@rocky.edu