This is half rant, half warning to others.

Relative merrits of the IBM ThinkPad X31 vs the Lenovo ThinkPad X220...

Old IBM ThinkPad X31 which I abused for years:

• Built like a tank
• 4:3 screen at 1024x768
• Fell down several times
• Got doused in coffee and coke
• Has been invaded (as in "filled to the brim") by Indian ants which ate out all the coffee and coke over about a week of constant occupation (even while the machine was running; I had no choice) years later
• Got hit by a truck's rear view mirror (~40 km/h)
• Got stepped upon repeatedly (not by me)
• Experienced everything from -25 to +35-40 degrees Celsius
• Experienced very high and extremely low humidity
• Survived uncounted kilometers of very rough off-road driving
• Got sprayed with sea water repeatedly
• Prolonged exposure to various kinds of abrasive sand
• I probably forgot other abuse

I upgraded the RAM, swapped the battery once and the HDD twice. If not for a loose power socket, I would still use it.

Contrast my new Lenovo ThinkPad X220 after a few months of pampering:

• You can remap the blue button to something useful (nice!)
• Wide-screen 16:9 at 1366 x 768 with no upgrade option (we are talking 5 years of technological advancement being ignored)
• Cardbus slot, but only one mini and one micro PCI slot so Wi-Fi, UMTS, SSD, and spindle disk at the same time are impossible
• Charge LED can only be seen from the outside which is great when my battery starts running low. Other people do care more about my battery status than myself, obviously! Making the LED visible from both sides would cost next to nothing. This is sloppy/bad design and nothing else. I am unable to understand why anyone who ever used a portable computer would do this. Argh!
• Sleep LED can only be seen from outside
• Use of built-in 3G or GPS can totally lock up device and/or make KDE/X quit and go back to login screen
• Random segfaults of programs that cause high network load via Wi-Fi (Chrome is especially queasy)
• Random ssh disconnects when using Wi-Fi
• Random sound artifacts during high network load via Wi-Fi
• All of the above, but less often, when using wired network
• Random segfaults and other magic issues after more than a few hours of uptime and/or warmth
• Three distinct USB ports (1 x USB 3.0, 1 x USB 2.0, 1 x USB 2.0 with power pass-through) which do or do not work correctly depending on uptime and temperature
• Makes weird noises
• On-desk combined power brick and USB hub broken after a few months of sitting on my desktop and being used once every week

Google tells me there are several people with random issues with their X220, especially when the mini-PCI slot is being used in a machine with USB 3.0. Removing the UMTS/GPS module did help, but still, this means no SSD for me as that's the slot I would have to use.

Recently, things have been degrading very quickly to the point where I refuse to do even simple things, like write and push blog posts, on this machine.

I am not sure if I am simply unlucky and received a dud, if the X220 is a model with issues, or if this is indicative of a general trend within Lenovo, but it's time to contact Lenovo support.

If I had known all of this in advance, I would simply have soldered the power socket back into my X31 and held out for the X221. Or something else, entirely.

Intro

RAID sucks, and so do all other Free alternatives in the Linux world.

Having been in exactly the same situation several times in the past, I have been following Russell Coker's posts regarding data integrity with interest. FWIW, the correct answer to "BTRFS (sic) and Xen" is: subvolumes.

Now that Fedora has, once again, decided to postpone their switch to Btrfs as default file system, I decided to write up my own take on this topic.

Any and all my considerations are made under the assumption that important data is backed up while semi-important data is at least living on two different machines. Beware, this is a long post, but I do like to think it's well worth reading.

RAID

Disk failures

Let's recap the current situation for "traditional" file systems like xfs, ext{2..4}, etc. In case you need to freshen up on the technical details go here.

• Single disks: Fine for many use cases like Laptops, Desktops, etc, they are what most of us are using for most storage needs.
• RAID 0:If you are using that, you are most likely doing it wrong. Barring increasing caches for SSD-based volumes on machines which you simply take out of your cluster if there is any kind of problem, I don't know of not a sigle valid use case. Other than to get rid of data, that is.
• RAID 1: The default for small servers and important machines which don't need a lot of disk space.
• RAID 5: OK for personal storage servers, avoid once disks become larger than 500GB-1TB, depending on personal preference. To write data, you need to calculate new checksums and thus read all data from all corresponding slices, impacting write performance significantly.
• RAID 6: The RAID 10 people will disagree, but I like this RAID level best. While your write performance will definitely take a hit, you can mitigate this by decreasing your volume sizes. The extra cost in ways of power, disks, controllers, and rack space are a price we gladly pay for the ability to lose two disks and still retain all data.
• RAID 10: Great write performance until the day when two disks in the same RAID position die right after each other and your company's main mail storage dies. Yes, this has happenend to me and it sucks.
• RAID 2,3,4,50,60,foo: Mostly irrelevant in the real world; disregard.

Silent corruption

All the RAID levels with redundancy are fine and dandy if you lose a whole disk and put in a new one. Of course, that does not help you the tiniest bit once you get read errors. Controllers which manage RAID 1 or 10 may or may not compare the data while it's being read and may or may not tell you about discrepancies. They can then toss a coin and give you either result. There is no way to determine which is correct. RAID 2, 3, 4, 5, and 6 could, in theory, verify the data as it's being read, but that would limit your read rate to that of a single disk so no one does that.

We actually had a massive web presence fail over night, once. No one knew what the cause was as there hadn't been any kind of access logged. As that particular deployment is done directly from a VCS, we simply ran a diff and found one change. We traced the failure down to a syntax error; a one character change made everything fail. Looking at the ASCII table, it was clear that one single bit had flipped. This sucked. A lot.

Mitigation

So you end up scrubbing your RAID sets on a weekly basis, recovering from silent corruption with the help of your RAID 6's two parity stripes. And even though you schedule the scrubbings with the least priority, you still take a performance hit when randomly seeking a lot.

ZFS

Some smart people at Sun set out to fix these problems and more, and fix it they did. The "Z" in ZFS is meant to imply that this is the last file system you will ever need and if not for its license, this might have worked. Of course, "last" and "forever" mean "not now" and "10 years" respectively, in computer terms. The limits within ZFS have been chosen so that the entropy needed to create a disk able to max out a ZFS volume would, literally, require our oceans to boil. Let's just say that you won't encounter these limits any time soon.

While that is a nice thing to know, it does not say anything about data integrity.

There are several mechanisms ensuring data integrity built in ZFS, the most fundamental being extensive checksumming. Checksums are vastly superior to live comparisions which RAID would have to perform as it does not need to sacrifice any significant i/o reading checksum data. With HDD access, you are i/o-bound, and not CPU-bound, anyway, so performing the checksum calculations is laughably cheap when factoring in the added safety. Everything, be data, metadata, inodes, you name it, is checksummed. Every read operation must go through the checksumming functions, ensuring that all data which reaches your user space is correct. If ZFS can't verify the checksum, it will not deliver any data at all, ensuring noticeable, as opposed to silent, corruption. So how does ZFS recover when the checksum does not match the data?

copies={1..3}

The easy way is to store several copies of your data within your volume. Mount a ZFS volume or subvolume with copies=2 or 3 and all data that is being written from then on will be copied twice or three times. Trivial in principle, but powerful when built into a file system.

RAIDz{1..3}

This is where things start to get interesting. RAIDz1, 2, and 3 will allow you to lose 1, 2, or 3 disks respectively while still retaining your data; at the obvious expense of the storage capacity of as many disks. Basically, RAIDz2 is ZFS' variant of RAID 6, but it's so much better than mere RAID 6.

Moo!

RAIDz will not assign fixed slices and fill them up like RAID does. It will look at the current write speeds and use appropriate slice sizes dynamically. As a direct result, data which has been written at roughly the same time will always be near other data written at the same time. This is very nice for functionality like snapshots, writeable subvolumes and other things and is called COW, ZFS' variant of super cow powers. This also enables ZFS to write data, read it back and verify the checksum and only then point to the new data. Atomic commits done right.

Keep rollin'

As ZFS has to keep track of what data is still in use in which volumes and subvolumes anyway, it knows which regions are free to be reclaimed. Instead of filling up random places, ZFS will roll over your disks, overwriting unused data on the fly.

This bears repeating: Contrary to RAID 5 and 6, ZFS will never need to read old data in order to write new data. Write performance galore and anyone still stuck on RAID 10 can finally enjoy the increased data security.

Btrfs

To be completely honest, I do not know Btrfs as well as I know ZFS. If I get anything wrong, correct me. If I offend anyone with my outsider's interpretation, that is not my intention.

Btrfs has been initiated by Oracle way before it ever even thought about buying Sun or history might have run a different course, for worse or a lot worse. The closing of Java, Solaris, ZFS, Hudson, OpenOffice (since then "gifted" aka thrown away), and others makes me think the latter. As there are few technological developments which caused me as much stress, overtime and pain as OCFS2, I am naturally wary of file systems sponsored by Oracle and their data focus integrity/availability. The fact that there's still no way to fsck a Btrfs volume could be birthing pain or yet another facet of Oracle's stance on this topic, I honestly don't know. Either way, it's a good idea to be wary for now.

I can't say too much on Btrfs' technical underpinnings, but I know it uses COW and that it has its own variant of RAID. Toss in snapshots (writeable?), subvolumes and integrated block device management, and you have the building blocks of a decent file system.

Still, Btrfs is not ready for prime time. Btrfs has been "one or two years in the future" for a few years now so I will not be holding my breath. And once the first distributions are starting to use Btrfs by default, people will lose data. A year or three after that, I will feel comfortable to use it myself and in production.

Commercial solutions

You can either hand a lot of money to proprietary vendors who will not tell you a thing about how things work internally (and no way to fix things directly) or you can buy solutions that employ ZFS in the back-end. I prefer my storage hardware to be relatively dumb while keeping the intelligent bits where I can see and poke them so this is not really part of these considerations.

So what can you do?

Test your disks before deploying them

Easy, but vitally important:

disk=/dev/foo
smartctl -a $disk
smartctl -t long $disk
badblocks -swo $vendor_$model_$serial_$timestamp.badblocks.swo $disk
smartctl -a $disk

If you are lazy, get a copy of disktest, a tool I wrote to do exactly this. It should wait for the long self-test to finish, can not read out all vendor names, and does not have a way to document who ran the test, but it's a start. Patches and feedback are, obviously, welcome.

And yes, I will package this soonish.

Using your disks

This list is surprisingly short. You can use

• distinct disks
• RAID as per above
• ZFS-FUSE
• Debian/kFreeBSD
• Nexenta

And that's it.

At work, I prefer small RAID 6 volume sets. It sucks, but there is nothing better.

For personal use, I have a machine with a FUSE-based RAIDz2 mounted with copies=2. Three disks can fail and my data is still secure. While this setup is slow as molasses due to FUSE, speed is not a consideration for me here; data safety is. A migration to Debian/kFreeBSD in the medium term is still likely.

If you have things to add or disagree with me, I would love to hear from you.

Update: Jan Christian Kaessens pointed out that, contrary to my experience, zfsonlinux did not bring down his sytem in flames. As zfsonlinux is a kernel module, potential performance is a lot better. Also, it supports ZFS Pool Version 28 as opposed to FUSE-ZFS' 23. While the really nice features are in v 29 and 30 (which are closed thanks to Oracle) v 28 still has some nice changes.

You know ACTA and related topics have reached the mainstream when the German news show devotes a segment of several minutes(!) to ACTA, complete with background information about the sleazy backroom dealings, and the various political sunday talk shows debate controversially about it, as well. In a round of five 60-plus aged people, only one defenced ACTA; the rest was sceptic or dead against it.

The several tens of thousands of people who walked the streets of Germany yesterday (I couldn't; I am ill) really managed to stir up Germany's political landscape. This. is. awesome.

Now if only the population at large would understand the inherent differences between physical counterfeiting, physical theft, and copyright infrigement.

That's where we all come in: Inform people about the background and actual issues. Now, more than ever, there's a chance that they will be willing to listen.

FOSDEM 2012 was very nice, as expected.

This year, I decided to stop consuming only and getting more involved. I helped staff the token sale and held two Lightning Talks. This turned out to be an awesome experience.

During the beer event, we sold 4600 beer tokens (FOSDEM does not earn anything from this; all proceeds go to Cafe Delirium). The sale was staffed with one to three people at all times which made for a constant but mostly bearable workload. While the others made good-natured fun of me and my German efficiency, I like to think that I helped make things run a tad more smoothly. Along those lines, we will have a sign asking people for exact change and/or to buy stacks that can be paid in bills if possible, next year; that should help streamline the whole thing even more. All in all, it turned out to be quite stressful, but tons of fun. I am looking forward to chuck in again next year.

The two Lighting Talks (vcsh on Saturday and git-annex on Sunday) I held very rather well received. Feedback after the talks and online was very positive which is always nice. There's been a slight increase in help requests and generic questions, so I assume people are trying vcsh and git-annex as a consequence of those talks, which is even nicer.

Also, I met a lot of people. Most of the meetings were pre-arranged, some by chance. Carrying a name tag at all times, I was approached by several people who knew me online but didn't know my face. I will try to always carry a name tag on conferences now and I suggest others do the same.

Finally, a huge thank you to everyone who helped make FOSDEM happen; you rock!

apt-get install vcsh

I finally got around to package vcsh and David Bremner was kind enough to sponsor it. vcsh is available in testing, unstable, and squeeze-backports. Also, Ubuntu seems to have copied it over into their repositories automagically.

As you probably don't know, vcsh is a tool to manage config files in git. Say you want to maintain one repository for zsh, one for vim, one for ssh, one for mplayer, and one for mr, but obviously there can only be one .git in $HOME. vcsh helps by moving $GIT_DIR into $XDG_CONFIG_HOME/vcsh/repo.d/ but keeping $GIT_WORK_TREE in $HOME.

The splitting of configuration sets into separate repositories allows you avoid checking out, say, mplayer's config on your servers, or checking out your ssh config at work.

If this sounds complicated, it's not; vcsh hides all of the dirty details from you. vcsh integrates nicely with mr by means of a plugin, making handling your configurations even more trivial. If you do get stuck along the way, simply drop into #vcs-home on irc.oftc.net or send email to the vcs-home mailing list.

popcon tells me that there are more users than I know about directly, so pipe up if you're one of them. And if you are one of the people who have it installed but don't use it, I am even more interested. I would love to know why so I can improve it.

Long story short, if you care about the integrity and/or history of your configuration, or if you use one or more computers, you should definitely give vcsh a try.

Any kind of feedback appreciated :)

Hello Planet :)

You may have seen my ancient blog posts pop up on the planet when XTaran was nice enough to add me to the planet's aggregation. Not quite sure why ikiwiki/planet decided to do this, but I will try to find out. If you have an idea, which does not involve re-spamming the feed, do let me know. In any case, expect mostly shell-related posts from me.

And, as every year, I will be going to FOSDEM.

Cleaning up a local repo:

git rebase -s ours -i --autosquash HEAD~20

Allowing the remote to be clobbered:

git config receive.denyNonFastForwards false

Pushing from local repo:

git push --force

Safe-guarding the remote once again:

git config receive.denyNonFastForwards true

In case you are rebasing an ikiwiki, you want to clobber the local checkout on your server, as well:

git remote update && git reset --hard @{u}

Done.

This might be interesting if you are using git-annex with one default remote and don't modify files once added:

git config [--global] alias.annex-push \
'!git pull && git annex add . && git annex copy . --to $REMOTE --fast --quiet && git commit -a -m "$HOST $(date +%F--%H-%M-%S-%Z)" && git push'

Optional parameters include --global (you will need to name all your remotes the same for this) and --fast (don't use this if you modify files). The timestamp is not-quite-ISO-8601, but it\'s human-readable and trivial to parse for computers.

Ben Hutchings writes:

There is an alternative to using C-c, and that is M-#. This adds a '#' to the start of the line, commenting it out, and then behaves as if you pressed Return. The result is that the command is cancelled but still remains on-screen and in your history. You can then copy the command later using the mouse or keyboard.

Naturally, there is a zsh equivalent for this (there always is...): pound-insert. Quoth the manpage:

pound-insert
If there is no # character at the beginning of the buffer, add one to the beginning of each line. If there is one, remove a # from each line that has one. In either case, accept the current line. The INTERACTIVE_COMMENTS option must be set for this to have any usefulness.

So if you want the same for zsh, a simple

bindkey '^[#' pound-insert

is all you need.

Update:

Clint Adams noted another way to do (almost) the same.

Here, there be blog posts.

For now, this blog is more or less a placeholder.

In the past, I created a site to share thoughts about life, the universe and everything. Said site has never been picked up by archive.org and a good thing also; the HTML was horrible.

Then, this new craze of "weblogs" arrived and I decided to go against the mainstream and not start another instance of those.

As blogs and microblogs have become a useful addition to partake in some interesting discussions, I decided to give the whole thing another spin.

All that being said, I do not expect to write a lot. You have been warned.

A good place to start are the full lists of all posts and comments, respectively.