Wednesday, April 20, 2011

git subtree module

Update (3Jun2013) : An improved version the git subtree module is available, now with a config file to define your subtrees and imported projects. There is an updated blog post describing how to install and use it.

Did you ever want to merge an external git tree with yours, while keeping the commit history? Or do you want to create a new git repository from a folder in your project, keeping the history?

Then the git subtree module is what you need. It let's you import a complete git repo (with commit history) into your project, and add the files to a folder you specify.

For example :

$git subtree add --prefix=other_project \
     git://github.com/your_tree/your_project.git master

imports the master branch of your git repository located in git://github.com/your_tree/your_project.git into the folder other_project.

If you make changes to this imported project and you want to push them back to the original project, you can use this :

$git subtree push --prefix=other_project \
     git://github.com/your_tree/your_project.git master

BTW: the subtree module is not part of the core git package. So if you want to use it, you will have to install the module first.

Download the git subtree module and extract it, or clone it :

$git clone https://github.com/apenwarr/git-subtree

In the git-subtree directory, run

$chmod u+x install.sh

and as root :

#./install.sh

This will copy the git subtree module to the git script folder. You can now use the git subtree module.

Monday, April 04, 2011

Reduce firewall configuration complexity using iptables with chains

Introduction

Setting up a firewall on your *nix box, being it a workstation, laptop, or server, is always a good idea. In most cases, you can do with some simple firewall rules, f.e. on your laptop, block all incoming requests (except the established connections, i.e. the replies on the outgoing requests you made), or on a simple webserver (allow port 80 only).

But if you need more complex rules, f.e. a server that hosts a website available for the entire internet, but with an ssh and samba service that should only be available for the local subnet, or even some specific IP addresses, it becomes a bit more complex.
And if you want to filter the outgoing traffic as well, your iptables rules get a mess after a while, and when you want to change anything, chances of a mistake or forgetting something are high, which may result in locking yourself out of your box (at least for remote access), or leaving something open that shouldn't.

To make your rules more manageable, you can make use of chains in your iptables rules. I got some inspiration in an article that uses chains to make iptables more efficient (faster). My goal was to get easier to read and configure iptables rules, but it will result in faster handling of packets as well.

Setup

  • A web service should be available from all networks (i.e. internet) on port 80 (http) and 443 (https)
  • The server can be managed remotely using ssh (port 20) and webmin (port 10000), but only from a limited set of IP addresses (admin PC's).
  • The server hosts a samba service (several TCP and UDP ports), that should only be available from a limited set of IP addresses (admin + webmaster PC's).
  • Outgoing connections will be filtered, but some services should be allowed (dns, dhcp, smtp, ntp) and some external websites should be available to get updates.

Concepts

ESTABLISHED state
When using this option, you can filter for established connections. If you define it in both the INPUT and OUTPUT rules, you only have to define in the INPUT rules which NEW incoming requests should be allowed, and in the OUTPUT rules which NEW outgoing request are allowed. The established connections will be allowed and should not be redefined (making the configuration a lot more readable and maintainable). An example allowing only an ssh service without using the ESTABLISHED state would be :

# iptables -A INPUT -p tcp --dport ssh -j ACCEPT
# iptables -A INPUT -j REJECT
# iptables -A OUTPUT -p tcp --sport ssh -j ACCEPT
# iptables -A OUTPUT -j REJECT

Basically, every incoming/outgoing connection is dropped, except if the incoming packet has port 22 (ssh) as destination, or if the outgoing packet was sent from port 22 (which is the reply of the ssh server).

When using ESTABLISHED state, this will be :

# iptables -A INPUT -p tcp --dport ssh -j ACCEPT
# iptables -A INPUT -j REJECT
# iptables -A OUTPUT -m state --state ESTABLISHED -j ACCEPT
# iptables -A OUTPUT -j REJECT

Now, every incoming/outgoing connection is dropped, except if the incoming packet has port 22 (ssh) as destination, or if the packet belongs to an established connection. Because incoming connections to port 22 are allowed, the firewall will remember a packet coming in, creating a 'connection' for the host/port the packet originates from when the ssh server replies to it. So when the reply of the ssh server is sent out, it matches an 'established' connection and will be allowed out.

In this example, the benefit of using the connection state is not clear, but when more allowed incoming services are added, they only have to be added on the INPUT chain, but not on the OUTPUT chain, because they are covered by the ESTABLISHED rule.
In the first example (without the ESTABLISHED rule), every allowed incoming connection should be repeated in the OUTPUT chain, matching the packets sent for the outgoing connection, which results in an equal amount of rules on both chains.
If you want to do filtering in both directions (allowing incoming request for listening services and outgoing request for remote services), this can become very messy, and almost unmaintainable without making mistakes.

Introducing chains
When two services (on different ports) should be available to a limited but identical list of IP addresses.
Without using chains, for every combination of port and IP a rule should be created :

# iptables -A INPUT -p tcp -m tcp -s 10.100.2.3 --dport 22 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.4 --dport 22 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.7 --dport 22 -j ACCEPT

# iptables -A INPUT -p tcp -m tcp -s 10.100.2.3 --dport 10000 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.4 --dport 10000 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.7 --dport 10000 -j ACCEPT

Resulting in a lot of rules, and when an IP address has to be changed/added/removed, this has to be done for every corresponding rule.

When using chains, this can be much easier. Imagine, that you first check if the packet matches the destination port, and if it does, jump to a new chain, where a list of IP addresses is checked. :

// create new chain admin_IP
# iptables -N admin_IP

// add rules to chain admin_IP
# iptables -A admin_IP -s 10.100.2.3 -j ACCEPT
# iptables -A admin_IP -s 10.100.2.4 -j ACCEPT
# iptables -A admin_IP -s 10.100.2.7 -j ACCEPT
// drop all packets that are not matched by previous rules
# iptables -A admin_IP -j DROP

// filter ports in INPUT chain
# iptables -A INPUT -p tcp -m tcp --dport 22 -j admin_IP
# iptables -A INPUT -p tcp -m tcp --dport 10000 -j admin_IP 

As you can see, there is are several benefits of putting the IP addresses in a separate chain :
  • the list of IP addresses in the separate chain can be reused for both ports, so they have to be defined only once.
  • adding/changing/removing an IP address is much easier
  • there is a better overview of the firewall rules.

Actual configuration

  • INPUT chain

    # iptables -A INPUT -i lo -j ACCEPT
    # iptables -A INPUT -p tcp -m tcp --tcp-flags ACK ACK -j ACCEPT
    # iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
    # iptables -A INPUT -m state --state RELATED -j ACCEPT
    # iptables -A INPUT -p icmp -j icmp_in
    # iptables -A INPUT -p tcp -m tcp --dport 22 -j admin_IP
    # iptables -A INPUT -p tcp -m tcp --dport 10000 -j admin_IP
    # iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
    # iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
    # iptables -A INPUT -p tcp -m tcp --dport 139 -j webmaster_IP
    # iptables -A INPUT -p tcp -m tcp --dport 445 -j webmaster_IP
    # iptables -A INPUT -p udp -m udp --dport 137:138 -j webmaster_IP
    # iptables -A INPUT -j DROP

    Basically, this is the input filter, allowing :
    • all local traffic (not leaving the physical PC)
    • a check for tcp-connections
    • established and related connections
    • ICMP packets (ping, etc.) are handled in a seperate chain icmp_in 
    • some services
      • ssh (tcp 22) and webmin (tcp 10000) allowed for admins (admin_IP chain)
      • website (tcp 80 and 443) for everybody
      • samba (tcp 139 and 445, udp 137-138) for webmasters (and admins, see definition of webmaster_IP chain)
    • everything else is not allowed (dropped) 
    Very structured and readable, I must say. :)

  • OUTPUT chain

    # iptables -A OUTPUT -o lo -j ACCEPT
    # iptables -A OUTPUT -m state --state ESTABLISHED -j ACCEPT
    # iptables -A OUTPUT -m state --state RELATED -j ACCEPT
    # iptables -A OUTPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.2 -p udp -m udp --dport 53 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.3 -p udp -m udp --dport 53 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.4 -p udp -m udp --dport 67 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.5 -p tcp -m tcp --dport 25 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.6 -p udp -m udp --dport 123 -j ACCEPT
    # iptables -A OUTPUT -p tcp -m tcp --dport 80 -j ext_websites
    # iptables -A OUTPUT -j DROP

    The output filter, allowing :
    • all local traffic (not leaving the physical PC)
    • established and related connections
    • ICMP replies
    • some remote services (hosted by different servers, IP addresses do not represent actual situation)
      • 2 dns (udp 53) servers (a separate chain could have been created)
      • dhcp (udp 67)
      • smtp (tcp 25)
      • ntp (udp 123)
    • external websites (tcp 80), listed in chain ext_websites
    • everything else is not allowed (dropped)
  • icmp_in chain

    # iptables -N icmp_in
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 8 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 0 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 3 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 4 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 11 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 12 -j ACCEPT
    # iptables -A icmp_in -j DROP

    Basically, all allowed incoming ICMP message types.

  • admin_IP chain

    # iptables -N admin_IP
    # iptables -A admin_IP -s 10.100.2.3 -j ACCEPT
    # iptables -A admin_IP -s 10.100.2.4 -j ACCEPT
    # iptables -A admin_IP -s 10.100.2.7 -j ACCEPT
    # iptables -A admin_IP -j DROP

    A list of allowed IP addresses of admin PC's.
    Everything else is not allowed.

  • webmaster_IP chain

    # iptables -N webmaster_IP
    # iptables -A webmaster_IP -s 10.100.2.11 -j ACCEPT
    # iptables -A webmaster_IP -s 10.100.2.17 -j ACCEPT
    # iptables -A webmaster_IP -s 10.100.2.34 -j ACCEPT
    # iptables -A webmaster_IP -s 10.100.2.50 -j ACCEPT
    # iptables -A webmaster_IP -j admin_IP

    A list of allowed IP addresses of webmaster PC's.
    At the end of the list, it jumps to the admin_IP chain, actually combining both chains.

  • ext_websites chain

    # iptables -N ext_websites
    # iptables -A ext_websites -d 212.211.132.250 -j ACCEPT
    # iptables -A ext_websites -d 212.211.132.32 -j ACCEPT
    # iptables -A ext_websites -d 195.20.242.89 -j ACCEPT
    # iptables -A ext_websites -d 130.89.149.225 -j ACCEPT
    # iptables -A ext_websites -d 86.59.118.153 -j ACCEPT
    # iptables -A ext_websites -d 130.89.149.227 -j ACCEPT
    # iptables -A ext_websites -d 128.31.0.51 -j ACCEPT
    # iptables -A ext_websites -d 86.59.118.153 -j ACCEPT
    # iptables -A ext_websites -d 67.228.198.100 -j ACCEPT
    # iptables -A ext_websites -d 140.211.166.6 -j ACCEPT
    # iptables -A ext_websites -d 140.211.166.21 -j ACCEPT
    # iptables -A ext_websites -j LOG

    A list of allowed external websites for updates (mirrors of Debian, webmin and Drupal, in this example).
    All other requests for external websites are logged. This can be useful for monitoring : notification of abuse, or if you forgot to add an allowed website.

This is of course, just an example. There are many variations possible. Maybe you want to add an ftp server, that is only accessible from a specific subnet and some other IP addresses. Or you can allow outgoing ssh-connections to a pool of servers. If you want to protect your server from scanning techniques (XMAS, NULL, ...) you can create a seperate chain for that as well. Options are limitless and depending on your needs.

But the idea is that the firewall rules are now much easier to understand and change. For example, if you want to make ssh (port 22) available also for webmasters, or for the entire internet, just change the -j option to webmaster_IP or ACCEPT. Adding an IP address for an admin PC, is just adding one line to the admin_IP chain.

BTW: I didn't go into every detail of iptables options that are used in my examples, so if you like more information on all available options of iptables and on how iptables works, you can take a look at this tutorial.