File recovery using foremost

Quick note on using foremost to retrieve files from a harddrive with a faulty partition table :

sudo foremost -dT -i /dev/md2 -t jpeg -o /path/to/restore2/

This scans /dev/md2 hd, looking for jpeg files and puts the restored files in /path/to/restore2/
Foremost can retrieve a lot of different file types (including images, MS Office files, PDF's and zipfiles), look at the man pages for a complete list. It is also possible to define your own filters.

Modify (or copy first) the foremost.conf file and add a line  for every file type fe. :

ex1   y   4096000  \x50\x4b\x03          \x74\x78\x74
ex2   y   4096000  \x50\x4b\x03???\x44   \x74\x78\x74

where you specify :

• file extension
• case sensitive pattern (if ASCII codes are used in the pattern)
• maximum size of a file in bytes
• starting pattern
• end pattern

The pattern can be in hex code or ASCII, use a ? as a wildcard for one character. In the second example above, a wildcard for 3 characters is added.

Now run foremost, using the custom config file :

sudo foremost -dT -i /dev/md2 -c /tmp/foremost.conf -o /path/to/restore2/

This scans /dev/md2 hd, using the rules in /tmp/foremost.conf and puts the restored files in /path/to/restore2/

upgrade Dell PERC6 firmware with Debian

How to upgrade the firmware of a PERC6 RAID controller in a Dell PowerEdge R510 server, that's running Debian 6 (Squeeze).
Dell does not officially support Debian, thus the provided upgrade package does not run automatically. But with some tweaking, you can get it done.

This manual works for both PERC6/E and PERC6/I integrated (onboard chip). I haven't tested it with PERC6/I, but I guess it will work as well.

Step 0 :

Install the latest Dell OpenManage Server Administrator software (OMSA 7.1). This is not necessary for the upgrade, but useful all along, because you get a (web)interface and some command line tools to manage the server hardware. You get log messages also.

There is a community supported port of OMSA for Ubuntu, but it works fine on Debian Squeeze.

Step 1 : get the firmware

You can get the firmware update packages at the Dell support pages. Select the binary for RedHat.

You will have to get the actual firmware upgrade file from this package :

bash SAS-RAID_Firmware_F96NR_LN_6.3.3-0002_X00.BIN --extract perc6e_fw

This will extract the contents of the binary into folder perc6e_fw. Using bash is needed because the scripts are intended for sh, but give errors when run on Ubuntu or Debian. You can also change the intended shell in the first line of the BIN file and run it.

The firmware upgrade file is located in perc6e_fw/payload/*.rom

Step 2 : get the firmware uploader

I couldn't get this Dell binary package to actually update the driver, so I used the firmware updater from LSI. The PERC 6 RAID controllers are rebranded LSI controllers, so this works.

Get the StorCli tool from the LSI website, choose the Linux version, it is a zipped RPM package.

First unzip it :

unzip Storcli_linux_10175.zip

Then retrieve the upgrade tool from the rpm package :

rpm2cpio storcli-1.01.75-1.noarch.rpm | cpio -idmv

This will convert the rpm to a cpio archive and extract the files from that cpio archive.

Now you have the storcli tool to upload the firmware upgrade file to the PERC controller. Here is a manual on how to use it, but we'll only need a few commands.

Step 3 : Upgrade the PERC controller firmware

Now you just have to upload the Dell PERC6 firmware to the RAID controller with the LSI tool :

storcli /c0 download file=FW1371E.rom

This will start downloading the firmware file into the controller.

After the download is finished, you have to reboot your server. The new firmware will be installed when the server is booting.

Please note that the /cX parameter defines which RAID controller you will send the firmware upgrade to. If you have only one PERC RAID controller in your server, you can use /c0. If you have multiple, it is safe to assume that the onboard PERC6/I is in 0 and that a second one is in 1. If you have more than 2, you will have to find out which controller is which. You can do this with this command :

storcli show

Lion Time Machine on Debian

After an upgrade to Lion (Mac OS X 10.7) Time Machine didn't work anymore.
It had been configured to store the Time Machine files on a Debian file server, using samba. But from Lion on, AFP 3 is required by Time Machine.

Solution : install netatalk on the Debian server.
According to a blogpost, at least version of netatalk 2.2.0 is required. The version of the available Debian package is older (2.1.2), so this means compiling from source code.

Use netatalk 2.2.3, get it here, copy it your server and extract :

$ tar -jxpvf netatalk-2.2.3.tar.bz2
$ cd netatalk-2.2.3

To enable DHX2 authentication (which is required), Debian packages libgcrypt11 and libgcrypt11-dev need to be installed. The libgcrypt11-dev package is required otherwise the library will not be detected by the configure script and the DHX2 (UAMS) module will not be compiled in.

$ sudo apt-get install libgcrypt11 libgcrypt11-dev

Default settings would put netatalk in /usr/local/*, so use the settings below to put the files in /usr/bin/, /usr/sbin/, and /etc/

$ ./configure --enable-debian --prefix=/usr --sysconfdir=/etc
$ make
$ sudo make install

Check if installation was successful by running

$ afpd -v

It should show netatalk-2.2.3 on the first line.

Here you can find more information about the config files, but to make the Time Machine work add this line to the end of file /etc/netatalk/AppleVolumes.default

~/timemachine "TimeMachine" options:upriv,usedots,tm

This makes folder timemachine in the homedir of the user (which you need to create) available on AFP as volume TimeMachine.
Note that option tm needs to be added to support Time Machine on this volume.

When all is set, netatalk can be started and your Mac should be able to use volume TimeMachine for backups :

$ sudo /etc/init.d/netatalk start

On your Mac, you will have to execute this instruction in a terminal window to make it recognise the afp share as a timemachine :

Defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1 

Update (12Nov2012) : added section about allowing linux timemachine volumes on a Mac

Cleanup apt cache

Over time, the cache of apt, the Debian package installation application, keeps growing. Every time a package is installed or updated, the original packages is stored in the cache. After a few months of updates, the cache grows quite big, because nothing is deleted from it automatically.
Keeping the original packages from all software you have installed is certainly a good idea, but the packages that are replaced by newer ones are kept as well.

The cache can be cleaned. using this command :

apt-get autoclean

This removes all unneeded (old) package files from the apt cache.

If you want to clear the cache completely, and thus removing all packages, use this command :

apt-get clean

Both commands are quite safe to use, because they only remove the downloaded packages from the apt cache. The installed files naturally remain untouched.
This way you can sometimes free a few hundred megabytes after a few months.

Ubuntu is based on the Debian packaging system, so this applies to every flavor of Ubuntu as well.

Microcode

This morning, one of my servers (Dell Poweredge R510, with a quad-core Intel Xeon E5520 CPU running Debian 5.0.4) had crashed. It didn't respond to anything : ping, ssh, smb. Not even when physically attaching a screen and a USB keyboard to it, did it respond.
After a reboot, the system started without a problem and everything seemed to work again.

Skimming through the logs I found some things occuring :

kernel: BUG: soft lockup - CPU#7 stuck for 61s!

and

kernel: __ratelimit: 35 messages suppressed
kernel: nf_conntrack: table full, dropping packet.

After Googling, I didn't really find a sound explanation, but some of the things that was mentioned was a bug in the Intel CPU, which could be solved by updating the CPU Microcode.

I'm not sure this will solve my problem, which only happened once since I started using the server about half a year ago, but as was mentioned, it does little risk, doesn't slow down your machine and might solve a few  problems.

So I installed two packages (you need contrib and non-free repositories):

apt-get install intel-microcode microcode.ctl

Package intel-microcode contains the updated microcode for Intel CPU's, while microcode.ctl does the update. Because the update is done in memory, the update is lost after a reboot, so you will have to do it again, but this package takes care of that.

Update 06Dec2010 : The microcode is automatically updated after a reboot. :)

change permissions of files and folders

On *nix, when you want to change the access permission of a directory and all files and folders in it, you could use :

cd your_dir
chmod -R 750 *

But this doesn't make a distinction between files and folders. So if you want different permissions for files and folders, you can do this :

cd your_dir
find -type f -exec chmod 640 {} \;
find -type d -exec chmod 750 {} \;