Showing posts with label server. Show all posts
Showing posts with label server. Show all posts

Monday, April 04, 2011

Reduce firewall configuration complexity using iptables with chains

Introduction

Setting up a firewall on your *nix box, being it a workstation, laptop, or server, is always a good idea. In most cases, you can do with some simple firewall rules, f.e. on your laptop, block all incoming requests (except the established connections, i.e. the replies on the outgoing requests you made), or on a simple webserver (allow port 80 only).

But if you need more complex rules, f.e. a server that hosts a website available for the entire internet, but with an ssh and samba service that should only be available for the local subnet, or even some specific IP addresses, it becomes a bit more complex.
And if you want to filter the outgoing traffic as well, your iptables rules get a mess after a while, and when you want to change anything, chances of a mistake or forgetting something are high, which may result in locking yourself out of your box (at least for remote access), or leaving something open that shouldn't.

To make your rules more manageable, you can make use of chains in your iptables rules. I got some inspiration in an article that uses chains to make iptables more efficient (faster). My goal was to get easier to read and configure iptables rules, but it will result in faster handling of packets as well.

Setup

  • A web service should be available from all networks (i.e. internet) on port 80 (http) and 443 (https)
  • The server can be managed remotely using ssh (port 20) and webmin (port 10000), but only from a limited set of IP addresses (admin PC's).
  • The server hosts a samba service (several TCP and UDP ports), that should only be available from a limited set of IP addresses (admin + webmaster PC's).
  • Outgoing connections will be filtered, but some services should be allowed (dns, dhcp, smtp, ntp) and some external websites should be available to get updates.

Concepts

ESTABLISHED state
When using this option, you can filter for established connections. If you define it in both the INPUT and OUTPUT rules, you only have to define in the INPUT rules which NEW incoming requests should be allowed, and in the OUTPUT rules which NEW outgoing request are allowed. The established connections will be allowed and should not be redefined (making the configuration a lot more readable and maintainable). An example allowing only an ssh service without using the ESTABLISHED state would be :

# iptables -A INPUT -p tcp --dport ssh -j ACCEPT
# iptables -A INPUT -j REJECT
# iptables -A OUTPUT -p tcp --sport ssh -j ACCEPT
# iptables -A OUTPUT -j REJECT

Basically, every incoming/outgoing connection is dropped, except if the incoming packet has port 22 (ssh) as destination, or if the outgoing packet was sent from port 22 (which is the reply of the ssh server).

When using ESTABLISHED state, this will be :

# iptables -A INPUT -p tcp --dport ssh -j ACCEPT
# iptables -A INPUT -j REJECT
# iptables -A OUTPUT -m state --state ESTABLISHED -j ACCEPT
# iptables -A OUTPUT -j REJECT

Now, every incoming/outgoing connection is dropped, except if the incoming packet has port 22 (ssh) as destination, or if the packet belongs to an established connection. Because incoming connections to port 22 are allowed, the firewall will remember a packet coming in, creating a 'connection' for the host/port the packet originates from when the ssh server replies to it. So when the reply of the ssh server is sent out, it matches an 'established' connection and will be allowed out.

In this example, the benefit of using the connection state is not clear, but when more allowed incoming services are added, they only have to be added on the INPUT chain, but not on the OUTPUT chain, because they are covered by the ESTABLISHED rule.
In the first example (without the ESTABLISHED rule), every allowed incoming connection should be repeated in the OUTPUT chain, matching the packets sent for the outgoing connection, which results in an equal amount of rules on both chains.
If you want to do filtering in both directions (allowing incoming request for listening services and outgoing request for remote services), this can become very messy, and almost unmaintainable without making mistakes.

Introducing chains
When two services (on different ports) should be available to a limited but identical list of IP addresses.
Without using chains, for every combination of port and IP a rule should be created :

# iptables -A INPUT -p tcp -m tcp -s 10.100.2.3 --dport 22 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.4 --dport 22 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.7 --dport 22 -j ACCEPT

# iptables -A INPUT -p tcp -m tcp -s 10.100.2.3 --dport 10000 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.4 --dport 10000 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.7 --dport 10000 -j ACCEPT

Resulting in a lot of rules, and when an IP address has to be changed/added/removed, this has to be done for every corresponding rule.

When using chains, this can be much easier. Imagine, that you first check if the packet matches the destination port, and if it does, jump to a new chain, where a list of IP addresses is checked. :

// create new chain admin_IP
# iptables -N admin_IP

// add rules to chain admin_IP
# iptables -A admin_IP -s 10.100.2.3 -j ACCEPT
# iptables -A admin_IP -s 10.100.2.4 -j ACCEPT
# iptables -A admin_IP -s 10.100.2.7 -j ACCEPT
// drop all packets that are not matched by previous rules
# iptables -A admin_IP -j DROP

// filter ports in INPUT chain
# iptables -A INPUT -p tcp -m tcp --dport 22 -j admin_IP
# iptables -A INPUT -p tcp -m tcp --dport 10000 -j admin_IP 

As you can see, there is are several benefits of putting the IP addresses in a separate chain :
  • the list of IP addresses in the separate chain can be reused for both ports, so they have to be defined only once.
  • adding/changing/removing an IP address is much easier
  • there is a better overview of the firewall rules.

Actual configuration

  • INPUT chain

    # iptables -A INPUT -i lo -j ACCEPT
    # iptables -A INPUT -p tcp -m tcp --tcp-flags ACK ACK -j ACCEPT
    # iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
    # iptables -A INPUT -m state --state RELATED -j ACCEPT
    # iptables -A INPUT -p icmp -j icmp_in
    # iptables -A INPUT -p tcp -m tcp --dport 22 -j admin_IP
    # iptables -A INPUT -p tcp -m tcp --dport 10000 -j admin_IP
    # iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
    # iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
    # iptables -A INPUT -p tcp -m tcp --dport 139 -j webmaster_IP
    # iptables -A INPUT -p tcp -m tcp --dport 445 -j webmaster_IP
    # iptables -A INPUT -p udp -m udp --dport 137:138 -j webmaster_IP
    # iptables -A INPUT -j DROP

    Basically, this is the input filter, allowing :
    • all local traffic (not leaving the physical PC)
    • a check for tcp-connections
    • established and related connections
    • ICMP packets (ping, etc.) are handled in a seperate chain icmp_in 
    • some services
      • ssh (tcp 22) and webmin (tcp 10000) allowed for admins (admin_IP chain)
      • website (tcp 80 and 443) for everybody
      • samba (tcp 139 and 445, udp 137-138) for webmasters (and admins, see definition of webmaster_IP chain)
    • everything else is not allowed (dropped) 
    Very structured and readable, I must say. :)

  • OUTPUT chain

    # iptables -A OUTPUT -o lo -j ACCEPT
    # iptables -A OUTPUT -m state --state ESTABLISHED -j ACCEPT
    # iptables -A OUTPUT -m state --state RELATED -j ACCEPT
    # iptables -A OUTPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.2 -p udp -m udp --dport 53 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.3 -p udp -m udp --dport 53 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.4 -p udp -m udp --dport 67 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.5 -p tcp -m tcp --dport 25 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.6 -p udp -m udp --dport 123 -j ACCEPT
    # iptables -A OUTPUT -p tcp -m tcp --dport 80 -j ext_websites
    # iptables -A OUTPUT -j DROP

    The output filter, allowing :
    • all local traffic (not leaving the physical PC)
    • established and related connections
    • ICMP replies
    • some remote services (hosted by different servers, IP addresses do not represent actual situation)
      • 2 dns (udp 53) servers (a separate chain could have been created)
      • dhcp (udp 67)
      • smtp (tcp 25)
      • ntp (udp 123)
    • external websites (tcp 80), listed in chain ext_websites
    • everything else is not allowed (dropped)
  • icmp_in chain

    # iptables -N icmp_in
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 8 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 0 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 3 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 4 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 11 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 12 -j ACCEPT
    # iptables -A icmp_in -j DROP

    Basically, all allowed incoming ICMP message types.

  • admin_IP chain

    # iptables -N admin_IP
    # iptables -A admin_IP -s 10.100.2.3 -j ACCEPT
    # iptables -A admin_IP -s 10.100.2.4 -j ACCEPT
    # iptables -A admin_IP -s 10.100.2.7 -j ACCEPT
    # iptables -A admin_IP -j DROP

    A list of allowed IP addresses of admin PC's.
    Everything else is not allowed.

  • webmaster_IP chain

    # iptables -N webmaster_IP
    # iptables -A webmaster_IP -s 10.100.2.11 -j ACCEPT
    # iptables -A webmaster_IP -s 10.100.2.17 -j ACCEPT
    # iptables -A webmaster_IP -s 10.100.2.34 -j ACCEPT
    # iptables -A webmaster_IP -s 10.100.2.50 -j ACCEPT
    # iptables -A webmaster_IP -j admin_IP

    A list of allowed IP addresses of webmaster PC's.
    At the end of the list, it jumps to the admin_IP chain, actually combining both chains.

  • ext_websites chain

    # iptables -N ext_websites
    # iptables -A ext_websites -d 212.211.132.250 -j ACCEPT
    # iptables -A ext_websites -d 212.211.132.32 -j ACCEPT
    # iptables -A ext_websites -d 195.20.242.89 -j ACCEPT
    # iptables -A ext_websites -d 130.89.149.225 -j ACCEPT
    # iptables -A ext_websites -d 86.59.118.153 -j ACCEPT
    # iptables -A ext_websites -d 130.89.149.227 -j ACCEPT
    # iptables -A ext_websites -d 128.31.0.51 -j ACCEPT
    # iptables -A ext_websites -d 86.59.118.153 -j ACCEPT
    # iptables -A ext_websites -d 67.228.198.100 -j ACCEPT
    # iptables -A ext_websites -d 140.211.166.6 -j ACCEPT
    # iptables -A ext_websites -d 140.211.166.21 -j ACCEPT
    # iptables -A ext_websites -j LOG

    A list of allowed external websites for updates (mirrors of Debian, webmin and Drupal, in this example).
    All other requests for external websites are logged. This can be useful for monitoring : notification of abuse, or if you forgot to add an allowed website.

This is of course, just an example. There are many variations possible. Maybe you want to add an ftp server, that is only accessible from a specific subnet and some other IP addresses. Or you can allow outgoing ssh-connections to a pool of servers. If you want to protect your server from scanning techniques (XMAS, NULL, ...) you can create a seperate chain for that as well. Options are limitless and depending on your needs.

But the idea is that the firewall rules are now much easier to understand and change. For example, if you want to make ssh (port 22) available also for webmasters, or for the entire internet, just change the -j option to webmaster_IP or ACCEPT. Adding an IP address for an admin PC, is just adding one line to the admin_IP chain.

BTW: I didn't go into every detail of iptables options that are used in my examples, so if you like more information on all available options of iptables and on how iptables works, you can take a look at this tutorial.

Monday, February 07, 2011

Debian 6 (Squeeze) is released.

A few days ago, Debian 6.0, codename Squeeze, was released after a 2 year development period.

Of course this new release contains a lot of new stuff : lots of new and upgraded packages, faster booting because processes are started in parallel (if dependencies allow it), integration of grub2, volatile repository is replaced by squeeze-updates, ...

Because I want to use Tomcat 6 on a production server, I wanted to try the upgrade on a testbox. Everything went quite smoothly. All steps for upgrading from Debian 5.0 (Lenny) are clearly explained in the release notes (Chapter 4). The only thing that went wrong, in my case was the update of mysql-server. For some reason it was uninstalled, but the replacement package for the new version 5.1 was not installed.

After the upgrade process, I rebooted and I noticed grub 0.97 (now called grub-legacy) was still used, but it transfered the bootprocess to grub2. This worked like a charm, so then I ran  (as root) :

# upgrade-from-grub-legacy

to install grub2 in the MBR and boot from it. Rebooting went perfectly afterwards.

So far I'm quite pleased with squeeze and the upgrade process. I'll do the upgrade with my other boxes in the next few days.

Friday, January 21, 2011

XFS check

I wanted to check the consistency of the data partition on one of my servers. It is 6.5TB and formatted with XFS, so I ran :

#xfs_check /dev/sdb1

And I got :

xfs_check: out of memory
 
After some searching, it turns out that a lot of memory is needed to perform the xfs_check on a large file system : >6GB and you need to run it on 64bit, to able to address that memory.

My system is a 32-bit with only 4GB, so I would probably not be able to run xfs_check on my system, but there is another way :

#xfs_repair -n /dev/sdb1

This tools tries to repair a XFS filesystem, but with the -n switch no changes are written to the file system, so the effect is quite the same. It still uses a lot of memory if you have a lot of files/inodes on the file system, but 3GB on a 32-bit system should be sufficient.

Of course, if xfs_repair finds a problem, you can still run it without the -n switch, to repair the filesystem.

Tuesday, October 19, 2010

Microcode

This morning, one of my servers (Dell Poweredge R510, with a quad-core Intel Xeon E5520 CPU running Debian 5.0.4) had crashed. It didn't respond to anything : ping, ssh, smb. Not even when physically attaching a screen and a USB keyboard to it, did it respond.
After a reboot, the system started without a problem and everything seemed to work again.

Skimming through the logs I found some things occuring :

kernel: BUG: soft lockup - CPU#7 stuck for 61s!
and
kernel: __ratelimit: 35 messages suppressed
kernel: nf_conntrack: table full, dropping packet.

After Googling, I didn't really find a sound explanation, but some of the things that was mentioned was a bug in the Intel CPU, which could be solved by updating the CPU Microcode.

I'm not sure this will solve my problem, which only happened once since I started using the server about half a year ago, but as was mentioned, it does little risk, doesn't slow down your machine and might solve a few  problems.

So I installed two packages (you need contrib and non-free repositories):
apt-get install intel-microcode microcode.ctl
Package intel-microcode contains the updated microcode for Intel CPU's, while microcode.ctl does the update. Because the update is done in memory, the update is lost after a reboot, so you will have to do it again, but this package takes care of that.

Update 06Dec2010 : The microcode is automatically updated after a reboot. :)

Sunday, March 28, 2010

New server : MicroClient Jr DX

I've got my new server up and running. It took me some time and effort to get it working, but now it's ready to replace my current server.

Norhtec MicroClient  Jr DX

First some specs :

  • Norhtec MicroClient  Jr DX
  • CPU : Vortex86DX Soc (System on Chip)
  • OS : Debian server 5.0.4 (Lenny)
  • Kernel : 2.6.33.1
  • Memory : 512 MB
  • Storage :
    • Compact Flash (CF) 8 GB (root, system files)
    • 2,5" HD 160 GB (swap and data files)
    • USB HD 320 GB (backup)
  • 100 Mbit LAN
And running services :
  • Fileserver : Samba (3.2.5)
  • Webserver : Apache (2.2.9) + PHP (5.2.6)
  • Database : MySQL (5.0.51)
  • Mailserver : Postfix (2.5.5)
  • DNS : Bind 9
  • DHCP 3
  • NTP
 
And now a small summary of the hurdles and how I overcame them to get it all working :
I got this new 'toy' a few months ago, in december 2009.
It doesn't have a cd-rom drive (the housing of the server is too small to hold one), so I decided to try PXE to install the OS. I added a PXE service to my network and soon was able to boot a Debian netinstall.
But the installer didn't recognise the hardware (CPU, network, harddrive). It turned out the default kernel didn't support it, so I had to use a custom one.
I followed this manual on how to get it and finally install the custom kernel. The only thing I did different is using a PXE-boot instead of the USB pendrive.

Small success here, but the hard drive controller still didn't work, so I was only able to install Debian on an USB harddrive. It worked, but I wanted to use the internal harddrive and CF-card.
Tinkering a bit with the BIOS settings didn't help much either. I found several possible settings on different websites. In the end, these settings worked for me :
  • PCI IDE Bus Master : Enable (HD doesn't work when Disabled)
  • Onboard IDE operate mode : Legacy (could be set to Native as well, but this has the best performance so far)
Then I stumbled upon the book Linux Kernel in a Nutshell written by Greg Kroah-Hartman. In Chapter 7  (p. 52-56) he describes how you can find a kernel module for hardware you are using. I found that module ata/pata_rdc.c was the one I needed to get the IDE-controller (RDC Semiconductor, Inc. Device 1011 (rev 01)) working.

It was introduced in kernel 2.6.32.* and is still experimental. I get a lot of warnings when booting or accessing the CF or HD, and it only runs at UDMA/33, which is much slower than the possible UDMA/100. But I hope the driver will get better in future kernel releases.
So, after building a custom kernel with all necessary modules included, I was able to boot and install Debian on the CF card, with the internal hard drive containing data.

After installing and configuring all services, copying data from my old server, and setting up the backup scripts, the new server is ready to replace the current one.
After more than 6 years of loyal service and an uptime of (currently) 90 days, it will retire, or at least, it will be used less frequently.

To summarize : the new server isn't more performant than the previous one, and it doesn't have that much more memory or storage, but : It's much smaller, less power consuming and less noisy. It think it might even pass the wife test. ;)