California

Waiting in the lobby of the hotel for the shuttle to take me back to the airport. And that will be the conclusion of a 2 week adventure starting at the Google campus for the GSoC Mentor Summit, a road trip visiting several places in North California and an extensive exploration of San Francisco city.

To summarize : it has been amazing. The things I've seen, the nature, the wildlife, the atmosphere, the people I've met and talked to.


So what did I learn :

• One can pay almost anything with a credit card, even drinks in a bar or a parking meter.
• Having an insurance is a good thing. If you have one, you will be helped immediately and with a smile. If you don't have one, you will still get a smile, but you are presented a bill before you are helped.
• San Francisco is a nice city, full of friendly and patient people (exception for cab drivers, but that counts for probably all cab drivers in the world) but also a city of controversy. Rich and poor live together, or more precisely, next to each other, and you can take that quite literally. One moment you're walking down a fancy shopping street, and one block further you find yourself surrounded by homeless people.
• Automatic gearboxes are nice for driving in the city and on freeways. Cruise control is also a very nice feature. But when doing some hills you just crave for a manual gearbox.
• Americans are very good at putting signs for anything, usually stating the obvious. It can be a helpful reminder, or in my case, being a tourist visiting the US for the first time, very convenient. But when you need a sign you can't find one.
• I still haven't come to grasp with traffic rules for crossroads. On crossroads without a traffic light and stop signs on all corners, I haven't figured out yet who gets to drive first. The one who came to the crossroad first, or the one coming from the right.
  On cross roads with traffic lights, it is usually allowed to drive when it's red if you are going right, but not always. So I usually stopped for red anyway, just to be safe. But this sometimes resulted in being notified by the cars behind me that I should be going.
• Driving by bike in San Francisco is a very nice experience. The challenge is to find streets that are not too steep.

It was a nice experience and I hope to return some day.

GSoC Mentor Summit 2011

I'm on the bus back to the hotel, after a weekend of meeting and collaboration with a lot of inspired Open Source developers. We met at Google Headquarters for the annual GSoC Mentor Summit, discussing last year's Google Summer of Code in particular and Open Source collaboration in general. More than 300 people of 175 participating organisations, from all over the world, were present.

It was my first time, both as a GSoC mentor for the phpMyAdmin project as on attending the summit, but it was great (or awesome, like some would state it). I've met and talked with many people. I learned a lot from the different sessions I attended and was pleasantly surprised about how much drive, knowledge and experience was present.

Awesome!

5 years

Exactly 5 years ago, I posted a 'First Post' on this blog. Back then I was wondering how long this blog would survive. Apparently it still exists. Going over a few posts of the past 5 years I noticed there were some active periods and some slower ones. Times were I posted things that I saw on the streets or read in the newspaper, but I think my blog has evolved into a space were I share IT and technology related stuff that I discover, am working on or I shouldn't forget.
By the way, the top 3 of most visited posts over the last 5 years are IT related :

1. Two-way MySQL replication
2. Rename multiple tables in MySQL database
   
3. Gadget Keyboard

BTW : A top 5 of the most popular post of the last 30 days is in the column on the right.

Under construction

Today my holidays start and I will spend them ... working : Tomorrow I will start redecorating my new house. Basically it will be just some cleaning, painting, and small repairs, no major de- and reconstruction.
In two weeks I plan to move to the new place and I hope to be ready by then, and I still have to do a lot of packing.
The next three weeks will be quite busy, just like the last 3 months actually. ;)

House

The front

For the last few weeks and months, I've been looking for a new place to live, but today I found a house that looked nice and affordable. So I bought it.
It sounds very simply put that way, but basically that's what I did. I visited it, inspected, decided to buy and came to an agreement with the current owner.

It's big enough for me, situated in a quiet neighborhood, close to my work, in a good state, and for a fair price.

But this is of course only the beginning of it. Now starts the hassle of securing a loan, planning to move, looking for cardboard boxes (to move things in, or if something goes wrong, to live in ;) ), start packing, notify my current landlord, and a lot of administration, ...

A lot of things to do in the next few weeks, that and going to work, mentoring for GSoC, and my normal every day and social live. :)

But it's all very exciting!

The view, old picture, the weather is much nicer at the moment ;)

The living room (not my furniture)

The stairs, quite fond of them actually.

git subtree module

Update (3Jun2013) : An improved version the git subtree module is available, now with a config file to define your subtrees and imported projects. There is an updated blog post describing how to install and use it.

Did you ever want to merge an external git tree with yours, while keeping the commit history? Or do you want to create a new git repository from a folder in your project, keeping the history?

Then the git subtree module is what you need. It let's you import a complete git repo (with commit history) into your project, and add the files to a folder you specify.

For example :

$git subtree add --prefix=other_project \
     git://github.com/your_tree/your_project.git master

imports the master branch of your git repository located in git://github.com/your_tree/your_project.git into the folder other_project.

If you make changes to this imported project and you want to push them back to the original project, you can use this :

$git subtree push --prefix=other_project \
     git://github.com/your_tree/your_project.git master

BTW: the subtree module is not part of the core git package. So if you want to use it, you will have to install the module first.

Download the git subtree module and extract it, or clone it :

$git clone https://github.com/apenwarr/git-subtree

In the git-subtree directory, run

$chmod u+x install.sh

and as root :

#./install.sh

This will copy the git subtree module to the git script folder. You can now use the git subtree module.

Reduce firewall configuration complexity using iptables with chains

Introduction

Setting up a firewall on your *nix box, being it a workstation, laptop, or server, is always a good idea. In most cases, you can do with some simple firewall rules, f.e. on your laptop, block all incoming requests (except the established connections, i.e. the replies on the outgoing requests you made), or on a simple webserver (allow port 80 only).

But if you need more complex rules, f.e. a server that hosts a website available for the entire internet, but with an ssh and samba service that should only be available for the local subnet, or even some specific IP addresses, it becomes a bit more complex.
And if you want to filter the outgoing traffic as well, your iptables rules get a mess after a while, and when you want to change anything, chances of a mistake or forgetting something are high, which may result in locking yourself out of your box (at least for remote access), or leaving something open that shouldn't.

To make your rules more manageable, you can make use of chains in your iptables rules. I got some inspiration in an article that uses chains to make iptables more efficient (faster). My goal was to get easier to read and configure iptables rules, but it will result in faster handling of packets as well.

Setup

• A web service should be available from all networks (i.e. internet) on port 80 (http) and 443 (https)
• The server can be managed remotely using ssh (port 20) and webmin (port 10000), but only from a limited set of IP addresses (admin PC's).
• The server hosts a samba service (several TCP and UDP ports), that should only be available from a limited set of IP addresses (admin + webmaster PC's).
• Outgoing connections will be filtered, but some services should be allowed (dns, dhcp, smtp, ntp) and some external websites should be available to get updates.

Concepts

ESTABLISHED state

When using this option, you can filter for established connections. If you define it in both the INPUT and OUTPUT rules, you only have to define in the INPUT rules which NEW incoming requests should be allowed, and in the OUTPUT rules which NEW outgoing request are allowed. The established connections will be allowed and should not be redefined (making the configuration a lot more readable and maintainable). An example allowing only an ssh service without using the ESTABLISHED state would be :

# iptables -A INPUT -p tcp --dport ssh -j ACCEPT
# iptables -A INPUT -j REJECT
# iptables -A OUTPUT -p tcp --sport ssh -j ACCEPT
# iptables -A OUTPUT -j REJECT

Basically, every incoming/outgoing connection is dropped, except if the incoming packet has port 22 (ssh) as destination, or if the outgoing packet was sent from port 22 (which is the reply of the ssh server).

When using ESTABLISHED state, this will be :

# iptables -A INPUT -p tcp --dport ssh -j ACCEPT
# iptables -A INPUT -j REJECT
# iptables -A OUTPUT -m state --state ESTABLISHED -j ACCEPT
# iptables -A OUTPUT -j REJECT

Now, every incoming/outgoing connection is dropped, except if the incoming packet has port 22 (ssh) as destination, or if the packet belongs to an established connection. Because incoming connections to port 22 are allowed, the firewall will remember a packet coming in, creating a 'connection' for the host/port the packet originates from when the ssh server replies to it. So when the reply of the ssh server is sent out, it matches an 'established' connection and will be allowed out.

In this example, the benefit of using the connection state is not clear, but when more allowed incoming services are added, they only have to be added on the INPUT chain, but not on the OUTPUT chain, because they are covered by the ESTABLISHED rule.
In the first example (without the ESTABLISHED rule), every allowed incoming connection should be repeated in the OUTPUT chain, matching the packets sent for the outgoing connection, which results in an equal amount of rules on both chains.
If you want to do filtering in both directions (allowing incoming request for listening services and outgoing request for remote services), this can become very messy, and almost unmaintainable without making mistakes.

Introducing chains

When two services (on different ports) should be available to a limited but identical list of IP addresses.
Without using chains, for every combination of port and IP a rule should be created :

# iptables -A INPUT -p tcp -m tcp -s 10.100.2.3 --dport 22 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.4 --dport 22 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.7 --dport 22 -j ACCEPT

# iptables -A INPUT -p tcp -m tcp -s 10.100.2.3 --dport 10000 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.4 --dport 10000 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.7 --dport 10000 -j ACCEPT

Resulting in a lot of rules, and when an IP address has to be changed/added/removed, this has to be done for every corresponding rule.

When using chains, this can be much easier. Imagine, that you first check if the packet matches the destination port, and if it does, jump to a new chain, where a list of IP addresses is checked. :

// create new chain admin_IP
# iptables -N admin_IP

// add rules to chain admin_IP
# iptables -A admin_IP -s 10.100.2.3 -j ACCEPT
# iptables -A admin_IP -s 10.100.2.4 -j ACCEPT
# iptables -A admin_IP -s 10.100.2.7 -j ACCEPT
// drop all packets that are not matched by previous rules
# iptables -A admin_IP -j DROP

// filter ports in INPUT chain
# iptables -A INPUT -p tcp -m tcp --dport 22 -j admin_IP
# iptables -A INPUT -p tcp -m tcp --dport 10000 -j admin_IP 

As you can see, there is are several benefits of putting the IP addresses in a separate chain :

• the list of IP addresses in the separate chain can be reused for both ports, so they have to be defined only once.
• adding/changing/removing an IP address is much easier
• there is a better overview of the firewall rules.

Actual configuration

• INPUT chain
  
  

  # iptables -A INPUT -i lo -j ACCEPT
  # iptables -A INPUT -p tcp -m tcp --tcp-flags ACK ACK -j ACCEPT
  # iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
  # iptables -A INPUT -m state --state RELATED -j ACCEPT
  # iptables -A INPUT -p icmp -j icmp_in
  # iptables -A INPUT -p tcp -m tcp --dport 22 -j admin_IP
  # iptables -A INPUT -p tcp -m tcp --dport 10000 -j admin_IP
  # iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
  # iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
  # iptables -A INPUT -p tcp -m tcp --dport 139 -j webmaster_IP
  # iptables -A INPUT -p tcp -m tcp --dport 445 -j webmaster_IP
  # iptables -A INPUT -p udp -m udp --dport 137:138 -j webmaster_IP
  # iptables -A INPUT -j DROP

  
  Basically, this is the input filter, allowing :
  
  • all local traffic (not leaving the physical PC)
  • a check for tcp-connections
  • established and related connections
  • ICMP packets (ping, etc.) are handled in a seperate chain icmp_in 
  • some services
    • ssh (tcp 22) and webmin (tcp 10000) allowed for admins (admin_IP chain)
    • website (tcp 80 and 443) for everybody
    • samba (tcp 139 and 445, udp 137-138) for webmasters (and admins, see definition of webmaster_IP chain)
  • everything else is not allowed (dropped) 
  Very structured and readable, I must say. :)
  
  
• OUTPUT chain
  
  

  # iptables -A OUTPUT -o lo -j ACCEPT
  # iptables -A OUTPUT -m state --state ESTABLISHED -j ACCEPT
  # iptables -A OUTPUT -m state --state RELATED -j ACCEPT
  # iptables -A OUTPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
  # iptables -A OUTPUT -d 10.1.1.2 -p udp -m udp --dport 53 -j ACCEPT
  # iptables -A OUTPUT -d 10.1.1.3 -p udp -m udp --dport 53 -j ACCEPT
  # iptables -A OUTPUT -d 10.1.1.4 -p udp -m udp --dport 67 -j ACCEPT
  # iptables -A OUTPUT -d 10.1.1.5 -p tcp -m tcp --dport 25 -j ACCEPT
  # iptables -A OUTPUT -d 10.1.1.6 -p udp -m udp --dport 123 -j ACCEPT
  # iptables -A OUTPUT -p tcp -m tcp --dport 80 -j ext_websites
  # iptables -A OUTPUT -j DROP

  
  The output filter, allowing :
  
  • all local traffic (not leaving the physical PC)
  • established and related connections
  • ICMP replies
  • some remote services (hosted by different servers, IP addresses do not represent actual situation)
    • 2 dns (udp 53) servers (a separate chain could have been created)
    • dhcp (udp 67)
    • smtp (tcp 25)
    • ntp (udp 123)
  • external websites (tcp 80), listed in chain ext_websites
  • everything else is not allowed (dropped)
• icmp_in chain
  
  

  # iptables -N icmp_in
  # iptables -A icmp_in -p icmp -m icmp --icmp-type 8 -j ACCEPT
  # iptables -A icmp_in -p icmp -m icmp --icmp-type 0 -j ACCEPT
  # iptables -A icmp_in -p icmp -m icmp --icmp-type 3 -j ACCEPT
  # iptables -A icmp_in -p icmp -m icmp --icmp-type 4 -j ACCEPT
  # iptables -A icmp_in -p icmp -m icmp --icmp-type 11 -j ACCEPT
  # iptables -A icmp_in -p icmp -m icmp --icmp-type 12 -j ACCEPT
  # iptables -A icmp_in -j DROP

  
  Basically, all allowed incoming ICMP message types.
  
  
• admin_IP chain
  
  

  # iptables -N admin_IP
  # iptables -A admin_IP -s 10.100.2.3 -j ACCEPT
  # iptables -A admin_IP -s 10.100.2.4 -j ACCEPT
  # iptables -A admin_IP -s 10.100.2.7 -j ACCEPT
  # iptables -A admin_IP -j DROP

  
  A list of allowed IP addresses of admin PC's.
  Everything else is not allowed.
  
  
• webmaster_IP chain
  
  

  # iptables -N webmaster_IP
  # iptables -A webmaster_IP -s 10.100.2.11 -j ACCEPT
  # iptables -A webmaster_IP -s 10.100.2.17 -j ACCEPT
  # iptables -A webmaster_IP -s 10.100.2.34 -j ACCEPT
  # iptables -A webmaster_IP -s 10.100.2.50 -j ACCEPT
  # iptables -A webmaster_IP -j admin_IP

  
  A list of allowed IP addresses of webmaster PC's.
  At the end of the list, it jumps to the admin_IP chain, actually combining both chains.
  
  
• ext_websites chain
  
  

  # iptables -N ext_websites
  # iptables -A ext_websites -d 212.211.132.250 -j ACCEPT
  # iptables -A ext_websites -d 212.211.132.32 -j ACCEPT
  # iptables -A ext_websites -d 195.20.242.89 -j ACCEPT
  # iptables -A ext_websites -d 130.89.149.225 -j ACCEPT
  # iptables -A ext_websites -d 86.59.118.153 -j ACCEPT
  # iptables -A ext_websites -d 130.89.149.227 -j ACCEPT
  # iptables -A ext_websites -d 128.31.0.51 -j ACCEPT
  # iptables -A ext_websites -d 86.59.118.153 -j ACCEPT
  # iptables -A ext_websites -d 67.228.198.100 -j ACCEPT
  # iptables -A ext_websites -d 140.211.166.6 -j ACCEPT
  # iptables -A ext_websites -d 140.211.166.21 -j ACCEPT
  # iptables -A ext_websites -j LOG

  
  A list of allowed external websites for updates (mirrors of Debian, webmin and Drupal, in this example).
  All other requests for external websites are logged. This can be useful for monitoring : notification of abuse, or if you forgot to add an allowed website.
  

This is of course, just an example. There are many variations possible. Maybe you want to add an ftp server, that is only accessible from a specific subnet and some other IP addresses. Or you can allow outgoing ssh-connections to a pool of servers. If you want to protect your server from scanning techniques (XMAS, NULL, ...) you can create a seperate chain for that as well. Options are limitless and depending on your needs.

But the idea is that the firewall rules are now much easier to understand and change. For example, if you want to make ssh (port 22) available also for webmasters, or for the entire internet, just change the -j option to webmaster_IP or ACCEPT. Adding an IP address for an admin PC, is just adding one line to the admin_IP chain.

BTW: I didn't go into every detail of iptables options that are used in my examples, so if you like more information on all available options of iptables and on how iptables works, you can take a look at this tutorial.