5 years

Exactly 5 years ago, I posted a 'First Post' on this blog. Back then I was wondering how long this blog would survive. Apparently it still exists. Going over a few posts of the past 5 years I noticed there were some active periods and some slower ones. Times were I posted things that I saw on the streets or read in the newspaper, but I think my blog has evolved into a space were I share IT and technology related stuff that I discover, am working on or I shouldn't forget.
By the way, the top 3 of most visited posts over the last 5 years are IT related :

1. Two-way MySQL replication
2. Rename multiple tables in MySQL database
   
3. Gadget Keyboard

BTW : A top 5 of the most popular post of the last 30 days is in the column on the right.

Under construction

Today my holidays start and I will spend them ... working : Tomorrow I will start redecorating my new house. Basically it will be just some cleaning, painting, and small repairs, no major de- and reconstruction.
In two weeks I plan to move to the new place and I hope to be ready by then, and I still have to do a lot of packing.
The next three weeks will be quite busy, just like the last 3 months actually. ;)

House

The front

For the last few weeks and months, I've been looking for a new place to live, but today I found a house that looked nice and affordable. So I bought it.
It sounds very simply put that way, but basically that's what I did. I visited it, inspected, decided to buy and came to an agreement with the current owner.

It's big enough for me, situated in a quiet neighborhood, close to my work, in a good state, and for a fair price.

But this is of course only the beginning of it. Now starts the hassle of securing a loan, planning to move, looking for cardboard boxes (to move things in, or if something goes wrong, to live in ;) ), start packing, notify my current landlord, and a lot of administration, ...

A lot of things to do in the next few weeks, that and going to work, mentoring for GSoC, and my normal every day and social live. :)

But it's all very exciting!

The view, old picture, the weather is much nicer at the moment ;)

The living room (not my furniture)

The stairs, quite fond of them actually.

git subtree module

Update (3Jun2013) : An improved version the git subtree module is available, now with a config file to define your subtrees and imported projects. There is an updated blog post describing how to install and use it.

Did you ever want to merge an external git tree with yours, while keeping the commit history? Or do you want to create a new git repository from a folder in your project, keeping the history?

Then the git subtree module is what you need. It let's you import a complete git repo (with commit history) into your project, and add the files to a folder you specify.

For example :

$git subtree add --prefix=other_project \
     git://github.com/your_tree/your_project.git master

imports the master branch of your git repository located in git://github.com/your_tree/your_project.git into the folder other_project.

If you make changes to this imported project and you want to push them back to the original project, you can use this :

$git subtree push --prefix=other_project \
     git://github.com/your_tree/your_project.git master

BTW: the subtree module is not part of the core git package. So if you want to use it, you will have to install the module first.

Download the git subtree module and extract it, or clone it :

$git clone https://github.com/apenwarr/git-subtree

In the git-subtree directory, run

$chmod u+x install.sh

and as root :

#./install.sh

This will copy the git subtree module to the git script folder. You can now use the git subtree module.

Reduce firewall configuration complexity using iptables with chains

Introduction

Setting up a firewall on your *nix box, being it a workstation, laptop, or server, is always a good idea. In most cases, you can do with some simple firewall rules, f.e. on your laptop, block all incoming requests (except the established connections, i.e. the replies on the outgoing requests you made), or on a simple webserver (allow port 80 only).

But if you need more complex rules, f.e. a server that hosts a website available for the entire internet, but with an ssh and samba service that should only be available for the local subnet, or even some specific IP addresses, it becomes a bit more complex.
And if you want to filter the outgoing traffic as well, your iptables rules get a mess after a while, and when you want to change anything, chances of a mistake or forgetting something are high, which may result in locking yourself out of your box (at least for remote access), or leaving something open that shouldn't.

To make your rules more manageable, you can make use of chains in your iptables rules. I got some inspiration in an article that uses chains to make iptables more efficient (faster). My goal was to get easier to read and configure iptables rules, but it will result in faster handling of packets as well.

Setup

• A web service should be available from all networks (i.e. internet) on port 80 (http) and 443 (https)
• The server can be managed remotely using ssh (port 20) and webmin (port 10000), but only from a limited set of IP addresses (admin PC's).
• The server hosts a samba service (several TCP and UDP ports), that should only be available from a limited set of IP addresses (admin + webmaster PC's).
• Outgoing connections will be filtered, but some services should be allowed (dns, dhcp, smtp, ntp) and some external websites should be available to get updates.

Concepts

ESTABLISHED state

When using this option, you can filter for established connections. If you define it in both the INPUT and OUTPUT rules, you only have to define in the INPUT rules which NEW incoming requests should be allowed, and in the OUTPUT rules which NEW outgoing request are allowed. The established connections will be allowed and should not be redefined (making the configuration a lot more readable and maintainable). An example allowing only an ssh service without using the ESTABLISHED state would be :

# iptables -A INPUT -p tcp --dport ssh -j ACCEPT
# iptables -A INPUT -j REJECT
# iptables -A OUTPUT -p tcp --sport ssh -j ACCEPT
# iptables -A OUTPUT -j REJECT

Basically, every incoming/outgoing connection is dropped, except if the incoming packet has port 22 (ssh) as destination, or if the outgoing packet was sent from port 22 (which is the reply of the ssh server).

When using ESTABLISHED state, this will be :

# iptables -A INPUT -p tcp --dport ssh -j ACCEPT
# iptables -A INPUT -j REJECT
# iptables -A OUTPUT -m state --state ESTABLISHED -j ACCEPT
# iptables -A OUTPUT -j REJECT

Now, every incoming/outgoing connection is dropped, except if the incoming packet has port 22 (ssh) as destination, or if the packet belongs to an established connection. Because incoming connections to port 22 are allowed, the firewall will remember a packet coming in, creating a 'connection' for the host/port the packet originates from when the ssh server replies to it. So when the reply of the ssh server is sent out, it matches an 'established' connection and will be allowed out.

In this example, the benefit of using the connection state is not clear, but when more allowed incoming services are added, they only have to be added on the INPUT chain, but not on the OUTPUT chain, because they are covered by the ESTABLISHED rule.
In the first example (without the ESTABLISHED rule), every allowed incoming connection should be repeated in the OUTPUT chain, matching the packets sent for the outgoing connection, which results in an equal amount of rules on both chains.
If you want to do filtering in both directions (allowing incoming request for listening services and outgoing request for remote services), this can become very messy, and almost unmaintainable without making mistakes.

Introducing chains

When two services (on different ports) should be available to a limited but identical list of IP addresses.
Without using chains, for every combination of port and IP a rule should be created :

# iptables -A INPUT -p tcp -m tcp -s 10.100.2.3 --dport 22 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.4 --dport 22 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.7 --dport 22 -j ACCEPT

# iptables -A INPUT -p tcp -m tcp -s 10.100.2.3 --dport 10000 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.4 --dport 10000 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.7 --dport 10000 -j ACCEPT

Resulting in a lot of rules, and when an IP address has to be changed/added/removed, this has to be done for every corresponding rule.

When using chains, this can be much easier. Imagine, that you first check if the packet matches the destination port, and if it does, jump to a new chain, where a list of IP addresses is checked. :

// create new chain admin_IP
# iptables -N admin_IP

// add rules to chain admin_IP
# iptables -A admin_IP -s 10.100.2.3 -j ACCEPT
# iptables -A admin_IP -s 10.100.2.4 -j ACCEPT
# iptables -A admin_IP -s 10.100.2.7 -j ACCEPT
// drop all packets that are not matched by previous rules
# iptables -A admin_IP -j DROP

// filter ports in INPUT chain
# iptables -A INPUT -p tcp -m tcp --dport 22 -j admin_IP
# iptables -A INPUT -p tcp -m tcp --dport 10000 -j admin_IP 

As you can see, there is are several benefits of putting the IP addresses in a separate chain :

• the list of IP addresses in the separate chain can be reused for both ports, so they have to be defined only once.
• adding/changing/removing an IP address is much easier
• there is a better overview of the firewall rules.

Actual configuration

• INPUT chain
  
  

  # iptables -A INPUT -i lo -j ACCEPT
  # iptables -A INPUT -p tcp -m tcp --tcp-flags ACK ACK -j ACCEPT
  # iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
  # iptables -A INPUT -m state --state RELATED -j ACCEPT
  # iptables -A INPUT -p icmp -j icmp_in
  # iptables -A INPUT -p tcp -m tcp --dport 22 -j admin_IP
  # iptables -A INPUT -p tcp -m tcp --dport 10000 -j admin_IP
  # iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
  # iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
  # iptables -A INPUT -p tcp -m tcp --dport 139 -j webmaster_IP
  # iptables -A INPUT -p tcp -m tcp --dport 445 -j webmaster_IP
  # iptables -A INPUT -p udp -m udp --dport 137:138 -j webmaster_IP
  # iptables -A INPUT -j DROP

  
  Basically, this is the input filter, allowing :
  
  • all local traffic (not leaving the physical PC)
  • a check for tcp-connections
  • established and related connections
  • ICMP packets (ping, etc.) are handled in a seperate chain icmp_in 
  • some services
    • ssh (tcp 22) and webmin (tcp 10000) allowed for admins (admin_IP chain)
    • website (tcp 80 and 443) for everybody
    • samba (tcp 139 and 445, udp 137-138) for webmasters (and admins, see definition of webmaster_IP chain)
  • everything else is not allowed (dropped) 
  Very structured and readable, I must say. :)
  
  
• OUTPUT chain
  
  

  # iptables -A OUTPUT -o lo -j ACCEPT
  # iptables -A OUTPUT -m state --state ESTABLISHED -j ACCEPT
  # iptables -A OUTPUT -m state --state RELATED -j ACCEPT
  # iptables -A OUTPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
  # iptables -A OUTPUT -d 10.1.1.2 -p udp -m udp --dport 53 -j ACCEPT
  # iptables -A OUTPUT -d 10.1.1.3 -p udp -m udp --dport 53 -j ACCEPT
  # iptables -A OUTPUT -d 10.1.1.4 -p udp -m udp --dport 67 -j ACCEPT
  # iptables -A OUTPUT -d 10.1.1.5 -p tcp -m tcp --dport 25 -j ACCEPT
  # iptables -A OUTPUT -d 10.1.1.6 -p udp -m udp --dport 123 -j ACCEPT
  # iptables -A OUTPUT -p tcp -m tcp --dport 80 -j ext_websites
  # iptables -A OUTPUT -j DROP

  
  The output filter, allowing :
  
  • all local traffic (not leaving the physical PC)
  • established and related connections
  • ICMP replies
  • some remote services (hosted by different servers, IP addresses do not represent actual situation)
    • 2 dns (udp 53) servers (a separate chain could have been created)
    • dhcp (udp 67)
    • smtp (tcp 25)
    • ntp (udp 123)
  • external websites (tcp 80), listed in chain ext_websites
  • everything else is not allowed (dropped)
• icmp_in chain
  
  

  # iptables -N icmp_in
  # iptables -A icmp_in -p icmp -m icmp --icmp-type 8 -j ACCEPT
  # iptables -A icmp_in -p icmp -m icmp --icmp-type 0 -j ACCEPT
  # iptables -A icmp_in -p icmp -m icmp --icmp-type 3 -j ACCEPT
  # iptables -A icmp_in -p icmp -m icmp --icmp-type 4 -j ACCEPT
  # iptables -A icmp_in -p icmp -m icmp --icmp-type 11 -j ACCEPT
  # iptables -A icmp_in -p icmp -m icmp --icmp-type 12 -j ACCEPT
  # iptables -A icmp_in -j DROP

  
  Basically, all allowed incoming ICMP message types.
  
  
• admin_IP chain
  
  

  # iptables -N admin_IP
  # iptables -A admin_IP -s 10.100.2.3 -j ACCEPT
  # iptables -A admin_IP -s 10.100.2.4 -j ACCEPT
  # iptables -A admin_IP -s 10.100.2.7 -j ACCEPT
  # iptables -A admin_IP -j DROP

  
  A list of allowed IP addresses of admin PC's.
  Everything else is not allowed.
  
  
• webmaster_IP chain
  
  

  # iptables -N webmaster_IP
  # iptables -A webmaster_IP -s 10.100.2.11 -j ACCEPT
  # iptables -A webmaster_IP -s 10.100.2.17 -j ACCEPT
  # iptables -A webmaster_IP -s 10.100.2.34 -j ACCEPT
  # iptables -A webmaster_IP -s 10.100.2.50 -j ACCEPT
  # iptables -A webmaster_IP -j admin_IP

  
  A list of allowed IP addresses of webmaster PC's.
  At the end of the list, it jumps to the admin_IP chain, actually combining both chains.
  
  
• ext_websites chain
  
  

  # iptables -N ext_websites
  # iptables -A ext_websites -d 212.211.132.250 -j ACCEPT
  # iptables -A ext_websites -d 212.211.132.32 -j ACCEPT
  # iptables -A ext_websites -d 195.20.242.89 -j ACCEPT
  # iptables -A ext_websites -d 130.89.149.225 -j ACCEPT
  # iptables -A ext_websites -d 86.59.118.153 -j ACCEPT
  # iptables -A ext_websites -d 130.89.149.227 -j ACCEPT
  # iptables -A ext_websites -d 128.31.0.51 -j ACCEPT
  # iptables -A ext_websites -d 86.59.118.153 -j ACCEPT
  # iptables -A ext_websites -d 67.228.198.100 -j ACCEPT
  # iptables -A ext_websites -d 140.211.166.6 -j ACCEPT
  # iptables -A ext_websites -d 140.211.166.21 -j ACCEPT
  # iptables -A ext_websites -j LOG

  
  A list of allowed external websites for updates (mirrors of Debian, webmin and Drupal, in this example).
  All other requests for external websites are logged. This can be useful for monitoring : notification of abuse, or if you forgot to add an allowed website.
  

This is of course, just an example. There are many variations possible. Maybe you want to add an ftp server, that is only accessible from a specific subnet and some other IP addresses. Or you can allow outgoing ssh-connections to a pool of servers. If you want to protect your server from scanning techniques (XMAS, NULL, ...) you can create a seperate chain for that as well. Options are limitless and depending on your needs.

But the idea is that the firewall rules are now much easier to understand and change. For example, if you want to make ssh (port 22) available also for webmasters, or for the entire internet, just change the -j option to webmaster_IP or ACCEPT. Adding an IP address for an admin PC, is just adding one line to the admin_IP chain.

BTW: I didn't go into every detail of iptables options that are used in my examples, so if you like more information on all available options of iptables and on how iptables works, you can take a look at this tutorial.

My first Python script

I've started reading a Python tutorial and this is the first little script I wrote :

#!/usr/bin/env python

a="internationalization"
print a[0] + str(len(a[1:-1])) + a[-1]

a="localization"
print a[0] + str(len(a[1:-1])) + a[-1]

If it is executed, you get :

i18n
l10n

It's not much, but slightly less boring than the usual "Hello World!".
Basically, it just takes the first and last character of each word, and replaces the characters in between with the total number of characters in between. The exciting thing about it, is the sheer compactness of the syntax. In any other language a lot more code would be needed. :)

BTW : i18n and l10n are widely used numeronyms, that refer to internationalization and localization.

bash treats numbers with leading 0 as octal

In a bash script, the number of the current week is used to select a different tape every week (rotation of 4 tapes) :

#!/bin/bash 

WEEK=`date +%W` # Week number (01..52)
TAPE=$((WEEK % 4))

This worked fine so far, but this weekend (it's the 8th weekend of the year) there was an error :

value too great for base (error token is "08")

After some digging, I found that bash treats numbers with a leading 0 as octal. But value 8 (and 9) don't exist in octal, so that's why it throws an error. With smaller numbers (01-07), there is no problem, because they are valid octal numbers, and for bigger numbers (10-...), the number is treated in decimal format, because there is no leading 0 anymore.

This problem can be fixed in two ways :

1) strip leading 0 from value

In bash you can do this with this command :

WEEK=${WEEK#0} # strip leading 0

This only removes a leading zero, a zero that is not leading will not be removed.

2) convert value to decimal

TAPE=$((10#$WEEK % 4)) # convert $WEEK to base-10 with '10#'

Note that you have to add the $ in front of the variable WEEK, to treat it is a variable, in order for the conversion (with 10#) to work.