Monday, November 07, 2011

California

Waiting in the lobby of the hotel for the shuttle to take me back to the airport. And that will be the conclusion of a 2 week adventure starting at the Google campus for the GSoC Mentor Summit, a road trip visiting several places in North California and an extensive exploration of San Francisco city.

To summarize : it has been amazing. The things I've seen, the nature, the wildlife, the atmosphere, the people I've met and talked to.


So what did I learn :
  • One can pay almost anything with a credit card, even drinks in a bar or a parking meter.
  • Having an insurance is a good thing. If you have one, you will be helped immediately and with a smile. If you don't have one, you will still get a smile, but you are presented a bill before you are helped.
  • San Francisco is a nice city, full of friendly and patient people (exception for cab drivers, but that counts for probably all cab drivers in the world) but also a city of controversy. Rich and poor live together, or more precisely, next to each other, and you can take that quite literally. One moment you're walking down a fancy shopping street, and one block further you find yourself surrounded by homeless people.
  • Automatic gearboxes are nice for driving in the city and on freeways. Cruise control is also a very nice feature. But when doing some hills you just crave for a manual gearbox.
  • Americans are very good at putting signs for anything, usually stating the obvious. It can be a helpful reminder, or in my case, being a tourist visiting the US for the first time, very convenient. But when you need a sign you can't find one.
  • I still haven't come to grasp with traffic rules for crossroads. On crossroads without a traffic light and stop signs on all corners, I haven't figured out yet who gets to drive first. The one who came to the crossroad first, or the one coming from the right.
    On cross roads with traffic lights, it is usually allowed to drive when it's red if you are going right, but not always. So I usually stopped for red anyway, just to be safe. But this sometimes resulted in being notified by the cars behind me that I should be going.
  • Driving by bike in San Francisco is a very nice experience. The challenge is to find streets that are not too steep.
It was a nice experience and I hope to return some day.

Monday, October 24, 2011

GSoC Mentor Summit 2011

I'm on the bus back to the hotel, after a weekend of meeting and collaboration with a lot of inspired Open Source developers. We met at Google Headquarters for the annual GSoC Mentor Summit, discussing last year's Google Summer of Code in particular and Open Source collaboration in general. More than 300 people of 175 participating organisations, from all over the world, were present.

It was my first time, both as a GSoC mentor for the phpMyAdmin project as on attending the summit, but it was great (or awesome, like some would state it). I've met and talked with many people. I learned a lot from the different sessions I attended and was pleasantly surprised about how much drive, knowledge and experience was present.

Awesome!

Monday, August 15, 2011

5 years

Exactly 5 years ago, I posted a 'First Post' on this blog. Back then I was wondering how long this blog would survive. Apparently it still exists. Going over a few posts of the past 5 years I noticed there were some active periods and some slower ones. Times were I posted things that I saw on the streets or read in the newspaper, but I think my blog has evolved into a space were I share IT and technology related stuff that I discover, am working on or I shouldn't forget.
By the way, the top 3 of most visited posts over the last 5 years are IT related :

  1. Two-way MySQL replication
  2. Rename multiple tables in MySQL database
  3. Gadget Keyboard
BTW : A top 5 of the most popular post of the last 30 days is in the column on the right.

Friday, July 08, 2011

Under construction

Today my holidays start and I will spend them ... working : Tomorrow I will start redecorating my new house. Basically it will be just some cleaning, painting, and small repairs, no major de- and reconstruction.
In two weeks I plan to move to the new place and I hope to be ready by then, and I still have to do a lot of packing.
The next three weeks will be quite busy, just like the last 3 months actually. ;)

Tuesday, May 03, 2011

House

The front
For the last few weeks and months, I've been looking for a new place to live, but today I found a house that looked nice and affordable. So I bought it.
It sounds very simply put that way, but basically that's what I did. I visited it, inspected, decided to buy and came to an agreement with the current owner.

It's big enough for me, situated in a quiet neighborhood, close to my work, in a good state, and for a fair price.

But this is of course only the beginning of it. Now starts the hassle of securing a loan, planning to move, looking for cardboard boxes (to move things in, or if something goes wrong, to live in ;) ), start packing, notify my current landlord, and a lot of administration, ...

A lot of things to do in the next few weeks, that and going to work, mentoring for GSoC, and my normal every day and social live. :)

But it's all very exciting!

The view, old picture, the weather is much nicer at the moment ;)
The living room (not my furniture)
The stairs, quite fond of them actually.

Wednesday, April 20, 2011

git subtree module

Update (3Jun2013) : An improved version the git subtree module is available, now with a config file to define your subtrees and imported projects. There is an updated blog post describing how to install and use it.

Did you ever want to merge an external git tree with yours, while keeping the commit history? Or do you want to create a new git repository from a folder in your project, keeping the history?

Then the git subtree module is what you need. It let's you import a complete git repo (with commit history) into your project, and add the files to a folder you specify.

For example :

$git subtree add --prefix=other_project \
     git://github.com/your_tree/your_project.git master

imports the master branch of your git repository located in git://github.com/your_tree/your_project.git into the folder other_project.

If you make changes to this imported project and you want to push them back to the original project, you can use this :

$git subtree push --prefix=other_project \
     git://github.com/your_tree/your_project.git master

BTW: the subtree module is not part of the core git package. So if you want to use it, you will have to install the module first.

Download the git subtree module and extract it, or clone it :

$git clone https://github.com/apenwarr/git-subtree

In the git-subtree directory, run

$chmod u+x install.sh

and as root :

#./install.sh

This will copy the git subtree module to the git script folder. You can now use the git subtree module.

Monday, April 04, 2011

Reduce firewall configuration complexity using iptables with chains

Introduction

Setting up a firewall on your *nix box, being it a workstation, laptop, or server, is always a good idea. In most cases, you can do with some simple firewall rules, f.e. on your laptop, block all incoming requests (except the established connections, i.e. the replies on the outgoing requests you made), or on a simple webserver (allow port 80 only).

But if you need more complex rules, f.e. a server that hosts a website available for the entire internet, but with an ssh and samba service that should only be available for the local subnet, or even some specific IP addresses, it becomes a bit more complex.
And if you want to filter the outgoing traffic as well, your iptables rules get a mess after a while, and when you want to change anything, chances of a mistake or forgetting something are high, which may result in locking yourself out of your box (at least for remote access), or leaving something open that shouldn't.

To make your rules more manageable, you can make use of chains in your iptables rules. I got some inspiration in an article that uses chains to make iptables more efficient (faster). My goal was to get easier to read and configure iptables rules, but it will result in faster handling of packets as well.

Setup

  • A web service should be available from all networks (i.e. internet) on port 80 (http) and 443 (https)
  • The server can be managed remotely using ssh (port 20) and webmin (port 10000), but only from a limited set of IP addresses (admin PC's).
  • The server hosts a samba service (several TCP and UDP ports), that should only be available from a limited set of IP addresses (admin + webmaster PC's).
  • Outgoing connections will be filtered, but some services should be allowed (dns, dhcp, smtp, ntp) and some external websites should be available to get updates.

Concepts

ESTABLISHED state
When using this option, you can filter for established connections. If you define it in both the INPUT and OUTPUT rules, you only have to define in the INPUT rules which NEW incoming requests should be allowed, and in the OUTPUT rules which NEW outgoing request are allowed. The established connections will be allowed and should not be redefined (making the configuration a lot more readable and maintainable). An example allowing only an ssh service without using the ESTABLISHED state would be :

# iptables -A INPUT -p tcp --dport ssh -j ACCEPT
# iptables -A INPUT -j REJECT
# iptables -A OUTPUT -p tcp --sport ssh -j ACCEPT
# iptables -A OUTPUT -j REJECT

Basically, every incoming/outgoing connection is dropped, except if the incoming packet has port 22 (ssh) as destination, or if the outgoing packet was sent from port 22 (which is the reply of the ssh server).

When using ESTABLISHED state, this will be :

# iptables -A INPUT -p tcp --dport ssh -j ACCEPT
# iptables -A INPUT -j REJECT
# iptables -A OUTPUT -m state --state ESTABLISHED -j ACCEPT
# iptables -A OUTPUT -j REJECT

Now, every incoming/outgoing connection is dropped, except if the incoming packet has port 22 (ssh) as destination, or if the packet belongs to an established connection. Because incoming connections to port 22 are allowed, the firewall will remember a packet coming in, creating a 'connection' for the host/port the packet originates from when the ssh server replies to it. So when the reply of the ssh server is sent out, it matches an 'established' connection and will be allowed out.

In this example, the benefit of using the connection state is not clear, but when more allowed incoming services are added, they only have to be added on the INPUT chain, but not on the OUTPUT chain, because they are covered by the ESTABLISHED rule.
In the first example (without the ESTABLISHED rule), every allowed incoming connection should be repeated in the OUTPUT chain, matching the packets sent for the outgoing connection, which results in an equal amount of rules on both chains.
If you want to do filtering in both directions (allowing incoming request for listening services and outgoing request for remote services), this can become very messy, and almost unmaintainable without making mistakes.

Introducing chains
When two services (on different ports) should be available to a limited but identical list of IP addresses.
Without using chains, for every combination of port and IP a rule should be created :

# iptables -A INPUT -p tcp -m tcp -s 10.100.2.3 --dport 22 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.4 --dport 22 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.7 --dport 22 -j ACCEPT

# iptables -A INPUT -p tcp -m tcp -s 10.100.2.3 --dport 10000 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.4 --dport 10000 -j ACCEPT
# iptables -A INPUT -p tcp -m tcp -s 10.100.2.7 --dport 10000 -j ACCEPT

Resulting in a lot of rules, and when an IP address has to be changed/added/removed, this has to be done for every corresponding rule.

When using chains, this can be much easier. Imagine, that you first check if the packet matches the destination port, and if it does, jump to a new chain, where a list of IP addresses is checked. :

// create new chain admin_IP
# iptables -N admin_IP

// add rules to chain admin_IP
# iptables -A admin_IP -s 10.100.2.3 -j ACCEPT
# iptables -A admin_IP -s 10.100.2.4 -j ACCEPT
# iptables -A admin_IP -s 10.100.2.7 -j ACCEPT
// drop all packets that are not matched by previous rules
# iptables -A admin_IP -j DROP

// filter ports in INPUT chain
# iptables -A INPUT -p tcp -m tcp --dport 22 -j admin_IP
# iptables -A INPUT -p tcp -m tcp --dport 10000 -j admin_IP 

As you can see, there is are several benefits of putting the IP addresses in a separate chain :
  • the list of IP addresses in the separate chain can be reused for both ports, so they have to be defined only once.
  • adding/changing/removing an IP address is much easier
  • there is a better overview of the firewall rules.

Actual configuration

  • INPUT chain

    # iptables -A INPUT -i lo -j ACCEPT
    # iptables -A INPUT -p tcp -m tcp --tcp-flags ACK ACK -j ACCEPT
    # iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
    # iptables -A INPUT -m state --state RELATED -j ACCEPT
    # iptables -A INPUT -p icmp -j icmp_in
    # iptables -A INPUT -p tcp -m tcp --dport 22 -j admin_IP
    # iptables -A INPUT -p tcp -m tcp --dport 10000 -j admin_IP
    # iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
    # iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
    # iptables -A INPUT -p tcp -m tcp --dport 139 -j webmaster_IP
    # iptables -A INPUT -p tcp -m tcp --dport 445 -j webmaster_IP
    # iptables -A INPUT -p udp -m udp --dport 137:138 -j webmaster_IP
    # iptables -A INPUT -j DROP

    Basically, this is the input filter, allowing :
    • all local traffic (not leaving the physical PC)
    • a check for tcp-connections
    • established and related connections
    • ICMP packets (ping, etc.) are handled in a seperate chain icmp_in 
    • some services
      • ssh (tcp 22) and webmin (tcp 10000) allowed for admins (admin_IP chain)
      • website (tcp 80 and 443) for everybody
      • samba (tcp 139 and 445, udp 137-138) for webmasters (and admins, see definition of webmaster_IP chain)
    • everything else is not allowed (dropped) 
    Very structured and readable, I must say. :)

  • OUTPUT chain

    # iptables -A OUTPUT -o lo -j ACCEPT
    # iptables -A OUTPUT -m state --state ESTABLISHED -j ACCEPT
    # iptables -A OUTPUT -m state --state RELATED -j ACCEPT
    # iptables -A OUTPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.2 -p udp -m udp --dport 53 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.3 -p udp -m udp --dport 53 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.4 -p udp -m udp --dport 67 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.5 -p tcp -m tcp --dport 25 -j ACCEPT
    # iptables -A OUTPUT -d 10.1.1.6 -p udp -m udp --dport 123 -j ACCEPT
    # iptables -A OUTPUT -p tcp -m tcp --dport 80 -j ext_websites
    # iptables -A OUTPUT -j DROP

    The output filter, allowing :
    • all local traffic (not leaving the physical PC)
    • established and related connections
    • ICMP replies
    • some remote services (hosted by different servers, IP addresses do not represent actual situation)
      • 2 dns (udp 53) servers (a separate chain could have been created)
      • dhcp (udp 67)
      • smtp (tcp 25)
      • ntp (udp 123)
    • external websites (tcp 80), listed in chain ext_websites
    • everything else is not allowed (dropped)
  • icmp_in chain

    # iptables -N icmp_in
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 8 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 0 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 3 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 4 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 11 -j ACCEPT
    # iptables -A icmp_in -p icmp -m icmp --icmp-type 12 -j ACCEPT
    # iptables -A icmp_in -j DROP

    Basically, all allowed incoming ICMP message types.

  • admin_IP chain

    # iptables -N admin_IP
    # iptables -A admin_IP -s 10.100.2.3 -j ACCEPT
    # iptables -A admin_IP -s 10.100.2.4 -j ACCEPT
    # iptables -A admin_IP -s 10.100.2.7 -j ACCEPT
    # iptables -A admin_IP -j DROP

    A list of allowed IP addresses of admin PC's.
    Everything else is not allowed.

  • webmaster_IP chain

    # iptables -N webmaster_IP
    # iptables -A webmaster_IP -s 10.100.2.11 -j ACCEPT
    # iptables -A webmaster_IP -s 10.100.2.17 -j ACCEPT
    # iptables -A webmaster_IP -s 10.100.2.34 -j ACCEPT
    # iptables -A webmaster_IP -s 10.100.2.50 -j ACCEPT
    # iptables -A webmaster_IP -j admin_IP

    A list of allowed IP addresses of webmaster PC's.
    At the end of the list, it jumps to the admin_IP chain, actually combining both chains.

  • ext_websites chain

    # iptables -N ext_websites
    # iptables -A ext_websites -d 212.211.132.250 -j ACCEPT
    # iptables -A ext_websites -d 212.211.132.32 -j ACCEPT
    # iptables -A ext_websites -d 195.20.242.89 -j ACCEPT
    # iptables -A ext_websites -d 130.89.149.225 -j ACCEPT
    # iptables -A ext_websites -d 86.59.118.153 -j ACCEPT
    # iptables -A ext_websites -d 130.89.149.227 -j ACCEPT
    # iptables -A ext_websites -d 128.31.0.51 -j ACCEPT
    # iptables -A ext_websites -d 86.59.118.153 -j ACCEPT
    # iptables -A ext_websites -d 67.228.198.100 -j ACCEPT
    # iptables -A ext_websites -d 140.211.166.6 -j ACCEPT
    # iptables -A ext_websites -d 140.211.166.21 -j ACCEPT
    # iptables -A ext_websites -j LOG

    A list of allowed external websites for updates (mirrors of Debian, webmin and Drupal, in this example).
    All other requests for external websites are logged. This can be useful for monitoring : notification of abuse, or if you forgot to add an allowed website.

This is of course, just an example. There are many variations possible. Maybe you want to add an ftp server, that is only accessible from a specific subnet and some other IP addresses. Or you can allow outgoing ssh-connections to a pool of servers. If you want to protect your server from scanning techniques (XMAS, NULL, ...) you can create a seperate chain for that as well. Options are limitless and depending on your needs.

But the idea is that the firewall rules are now much easier to understand and change. For example, if you want to make ssh (port 22) available also for webmasters, or for the entire internet, just change the -j option to webmaster_IP or ACCEPT. Adding an IP address for an admin PC, is just adding one line to the admin_IP chain.

BTW: I didn't go into every detail of iptables options that are used in my examples, so if you like more information on all available options of iptables and on how iptables works, you can take a look at this tutorial.

Wednesday, March 02, 2011

My first Python script

I've started reading a Python tutorial and this is the first little script I wrote :

#!/usr/bin/env python

a="internationalization"
print a[0] + str(len(a[1:-1])) + a[-1]

a="localization"
print a[0] + str(len(a[1:-1])) + a[-1]

If it is executed, you get :

i18n
l10n

It's not much, but slightly less boring than the usual "Hello World!".
Basically, it just takes the first and last character of each word, and replaces the characters in between with the total number of characters in between. The exciting thing about it, is the sheer compactness of the syntax. In any other language a lot more code would be needed. :)

BTW : i18n and l10n are widely used numeronyms, that refer to internationalization and localization.

Monday, February 28, 2011

bash treats numbers with leading 0 as octal

In a bash script, the number of the current week is used to select a different tape every week (rotation of 4 tapes) :

#!/bin/bash 

WEEK=`date +%W` # Week number (01..52)
TAPE=$((WEEK % 4))

This worked fine so far, but this weekend (it's the 8th weekend of the year) there was an error :

value too great for base (error token is "08")

After some digging, I found that bash treats numbers with a leading 0 as octal. But value 8 (and 9) don't exist in octal, so that's why it throws an error. With smaller numbers (01-07), there is no problem, because they are valid octal numbers, and for bigger numbers (10-...), the number is treated in decimal format, because there is no leading 0 anymore.

This problem can be fixed in two ways :

1) strip leading 0 from value


In bash you can do this with this command :

WEEK=${WEEK#0} # strip leading 0

This only removes a leading zero, a zero that is not leading will not be removed.

2) convert value to decimal

TAPE=$((10#$WEEK % 4)) # convert $WEEK to base-10 with '10#'

Note that you have to add the $ in front of the variable WEEK, to treat it is a variable, in order for the conversion (with 10#) to work.

Sunday, February 13, 2011

FOSDEM 2011

It's been great again. Over 5000 Open Source developers and enthousiasts meeting to share more than 200 hours of knowledge over the course of a weekend at FOSDEM 2011, held at campus Solbosch of ULB in Brussels, Belgium.
Last year I went there for the first time, for just one day, and I liked it so much I immediately decided to go back this year, but for the whole two days.

So there I was, on Saturday afternoon, after taking two trains, a metro and a bus, entering the ULB campus where FOSDEM is held, seeing the signs pointing to all different buildings and rooms where talks were held, the catering trucks, and a busy bunch of enthousiastic people.

I headed for the MySQL & Friends devroom, where I met some of the people of the phpMyAdmin team, which I joined last year. I hadn't met them before, so this was a nice opportunity.
These are some of the talks I attended on Saturday :

After this, we had a phpMyAdmin team dinner and afterwards I introduced them to  some fine Belgian beers.

Sunday was a bit chaotic, I hadn't planned to see that much talks, one of the talks I planned to attend was canceled, and I missed another one, because I left early to catch the canceled one (but I didn't know it at that time), but I still went to some :

After the last talk, it was time to go home. A special FOSDEM bus brought us back to the railway station, were I had some dinner before I headed home.
It was a nice weekend, with a nice Open atmosphere, again perfectly organized by and for the community.  In case you missed it, this was FOSDEM 2011 (a video artists impression).

I'm looking forward to next edition already!

Monday, February 07, 2011

Debian 6 (Squeeze) is released.

A few days ago, Debian 6.0, codename Squeeze, was released after a 2 year development period.

Of course this new release contains a lot of new stuff : lots of new and upgraded packages, faster booting because processes are started in parallel (if dependencies allow it), integration of grub2, volatile repository is replaced by squeeze-updates, ...

Because I want to use Tomcat 6 on a production server, I wanted to try the upgrade on a testbox. Everything went quite smoothly. All steps for upgrading from Debian 5.0 (Lenny) are clearly explained in the release notes (Chapter 4). The only thing that went wrong, in my case was the update of mysql-server. For some reason it was uninstalled, but the replacement package for the new version 5.1 was not installed.

After the upgrade process, I rebooted and I noticed grub 0.97 (now called grub-legacy) was still used, but it transfered the bootprocess to grub2. This worked like a charm, so then I ran  (as root) :

# upgrade-from-grub-legacy

to install grub2 in the MBR and boot from it. Rebooting went perfectly afterwards.

So far I'm quite pleased with squeeze and the upgrade process. I'll do the upgrade with my other boxes in the next few days.

Friday, January 21, 2011

XFS check

I wanted to check the consistency of the data partition on one of my servers. It is 6.5TB and formatted with XFS, so I ran :

#xfs_check /dev/sdb1

And I got :

xfs_check: out of memory
 
After some searching, it turns out that a lot of memory is needed to perform the xfs_check on a large file system : >6GB and you need to run it on 64bit, to able to address that memory.

My system is a 32-bit with only 4GB, so I would probably not be able to run xfs_check on my system, but there is another way :

#xfs_repair -n /dev/sdb1

This tools tries to repair a XFS filesystem, but with the -n switch no changes are written to the file system, so the effect is quite the same. It still uses a lot of memory if you have a lot of files/inodes on the file system, but 3GB on a 32-bit system should be sufficient.

Of course, if xfs_repair finds a problem, you can still run it without the -n switch, to repair the filesystem.

Tuesday, January 04, 2011

All time favorites : games

I've recently rediscovered the wonderful pass time of gaming. And I was thinking of a few great games I played in the past, in alphabetical order :

They earned a position in this list because of the mere of beauty of the game, which was usually a combination of graphics, music, scenery, atmosphere and gameplay.

Saturday, January 01, 2011

Happy 2011!

Happy New Year!

May 2011 be a year of friendship, love, joy and happiness, good health and prosperity!

2010


A lot of things happened in 2010. I'm not going to write an exhaustive list, but will just mention things that spring to mind.
On a technological level : I got my new server and installed linux on it, I learned to use vim and git, and started contributing to the Open Source community by joining the phpMyAdmin development team. I visited FOSDEM for the first time and will certainly do so this year.
On a professional level : I created a new website for my department using Drupal, and installed a new backup server.

One of my resolutions for 2010 was to read more, and I achieved this. If possible I read a chapter every evening, and I hope to continue doing this in the new year.
I also spent quite some time playing games, watching films and series and (re)discovered some music. Some of the highlights of the year : Assissin's Creed II, the best game I played in years; Isbells and Mumford & Sons, the latter doing some great gigs; and 1984, classic by George Orwell. :)

And I met a lot of interesting people, made new friends and got to know some people better.

Resolutions 2011


Bit a more of the same of last year. I made some progress, but there is always room for improvement :

  • Smile (even) more!
  • Enjoy more and worry less
  • Continue reading
  • Be more confident
  • Look for a new place to live