Friday, July 8, 2011

Blogger: Having posts skip the RSS feed

Turns out you can make new posts not show up in the RSS feed in Blogger, but the only way to to do it is to backdate them far enough that, chronologically, they are more than 8 entries ago.

Oi, working around the limitations of closed platforms is always frustrating.

Wednesday, June 29, 2011

Spotlight on Open Source: youtube-dl

youtube-dl by Ricardo Garcia Gonzalez (Public Domain)

One of the things I really enjoy watching on Youtube are "Let's Play's": regular people recording their play-throughs of (typically classic) video games in an episodic format. I have a lot of childhood memories associated with various 8- and 16-bit titles, so it's always a fun trip down memory lane.

Unfortunately Youtube's playlist interface can be summarily described as "awful" and the interface still does not support altering playback speed for most videos. Worse, Adobe Flash just sucks and doesn't belong anywhere near Linux.

I used to use an extension for my web browser to download individual videos, but youtube-dl makes life much easier by letting you download an entire playlist into a directory. You can even resume downloading where you left off so it's easy to get partial playlists or keep your local archive up to date as new videos are added.

Is it worth it to waste disk space storing videos you can download for free whenever you want? Of course! For one thing, it's nice if you have a monthly download quota like I do since playlists with HD videos can often exceed 20GB: you can download some content in advance to watch later when you might be near the quota.

For another, mplayer is far superior to Youtube's player. You can set a default (faster) playback speed and apply video filters like unsharp mask and add noise which really improve perceived video quality, or use vdpau to get hardware acceleration for 1080p movies.

Finally, you can move each file into a "watched" subfolder as you finish watching it, which I find to be far more intuitive than any of Youtube's playlist-type features.

The only thing I thought youtube-dl was missing was support for parallel downloads of all files in a playlist: Youtube limits standard defintion video downloads to about 100k/s, so if you want to max out your connection, parallel downloads are necessary. I tend to run it in GNU screen on a server, though, so it doesn't bother me if it takes a while to finish.

Overall, youtube-dl is a fantastic script. Thanks, Ricardo!

Spotlight on Open Source is a periodic feature where I highlight a piece of open-source software that is making my life better. If you liked this entry, why not view the previous features? Open-source projects usually benefit from publicity, so I encourage you to spread the word as well!

Wednesday, June 15, 2011

Hyphenator on Blogger

It feels a little kludgy, but I managed to get Hyphenator installed on this blog so the text can be justified.

I'm not sure if editing your template to add the hyphenate class to the post-body entry-content div is the only way to do it, but it worked for me. For some reason I already had my template under version control so it's pretty easy for me to add stuff to it even though I run into bugs in Blogger every time I try to update the template.

But I'll spare you the boring details of my complaints about Blogger.

Tuesday, June 14, 2011

Air conditioning affects line voltage

This is kinda neat. As May has turned into June and more people are using air conditioning, the line voltage detected by my UPS has started to dip during the day. It's the orange line at the top in this image:

The minimum line voltage corresponds to about 4PM.

Sunday, June 12, 2011

Terrible, then merely bad

Over on the post where Google announced they were going to shut down the Translate API, I found this (abridged) comment:

Google is like NASA of the internet - they have the best and brightest on their payroll - and I'm not just talking about their developers. If there was truly an abuse to the API they would have developed a way to handle it without shutting it down. So, in this case, the abuse is nothing more than a smoke screen.

If you woke up one morning and found out that Google is going to charge for their API you would start bashing Google left and right for charging for something that used to be free and their PR would flat line - essentially - they would look like the monster taking candy from a child.

I don't know whether he's right and it was an intentional move, but he is right that the PR tends to work better if you announce something terrible and then recant and announce something merely bad which looks good in comparison to your original announcement.

Recently Google's App Engine team also announced major pricing changes which looked like a disaster for many of us, and then changed their mind about some of the details to make them more favorable. I think the overall sentiment in the community is now actually better than it would have been if they had simply announced the latest (better) details in the first place and then not made any changes.

Doing this sort of thing intentionally is probably a bit unethical, but I don't doubt that it works.

Wednesday, April 27, 2011

Closure Compiler: Removing console log statements

This thread has a great solution for stripping console.log() calls using the closure compiler:
  /** @define {boolean} */ DEBUG = true,

console_log = function(var_args) {
  if (DEBUG) {
    console.log.apply(console, arguments);

(Now you must use console_log calls instead of console.log.)

I tested it and it works great. There are some other ideas on stackoverflow but many of them don't work. I think this is the best way.

Incidentally @define will allow you to pass a flag to the compiler to enable/disable the log statements but I prefer to just change the constant in the JavaScript file since make will then automatically detect the change and rebuild without needing any changes to the Makefile.

Tuesday, April 26, 2011

The Future of Firefox

Recently I tried to create my first Firefox addon. It was not a pleasant experience. Chromium's API makes such things as adding an item to the browser context menu simpler and the debugging process is easier to understand.

This is not to knock on the fine folks working at Mozilla; I read somewhere that Firefox's codebase has been touched by so many people that it is truly a cruel and unusual form of torture trying to understand and modify it. I believe it.

Still, it makes me sad. I like the idea of contributing to the software ecosystem around a browser made by a non-profit organization. I am just finding it hard to justify it on practical grounds.

If I am having these doubts, maybe other people are too.

Thursday, April 21, 2011

CLRS bibtex entry (3rd edition)

Following on the work of Timmie Smith, here is some bibtex for the 3rd edition of CLRS.
  author = "T.~H. Cormen and C.~E. Leiserson and R.~L. Rivest and C.~Stein",
  edition = "3rd",
  publisher = "The MIT Press",
  title = "Introduction to Algorithms",
  year = "2009",
  isbn = "978-0-262-03384-8"

Tuesday, April 12, 2011

Changing Ubuntu Lucid's Boot Drive with Grub 2

I recently bought an SSD for my workstation and wanted to boot from the SSD instead of spinning disk. It used to be you could do this simply by copying all the files from your current / partition over to the new partiton, mucking about in /etc/fstab a bit, (optionally) installing grub in the MBR of the new disk, and then changing the boot order in the BIOS to put the disk with your new partition at the top of the list. This would cause grub to see your new disk as hd0 and maybe you didn't even need to change /etc/fstab either if the number of the new partition was the same as the old one.

It turns out that with Grub 2, changing the boot order of disks in the BIOS is no longer sufficient because Grub 2 uses UUIDs for all entries! So even after you change the boot order it will still find the old disk.

The good news is that you can easily solve this by search/replace of the UUIDs in the /boot/grub/grub.cfg file you are not supposed to edit. After changing the UUIDs you will boot into the new partition, and from then on update-grub seems to detect the correct UUID since it probably just finds the UUID of the partition mounted as /.

I haven't seen this procedure written about anywhere; I just guessed that it would work and it did. Your mileage may vary!

The SSD reduced boot time a bit, but not as much as I was expecting. Down from about 15 seconds (post-bios) to about 9. (Ubuntu did a lot of work reducing boot times on spinning disks in the 9.x series, I think.) My bios really sucks and takes like 20 seconds just to get to grub, so the reduction is not really noticeable.

Boot time was a wash, but the real win is that package management operations are much faster: running updates and starting Synaptic is no longer unbearably slow.

Overall, using a UUID to reference disks in GRUB is a win even if it does make the procedure for rare changes like this more confusing. My old motherboard has 2 different SATA chipsets and doesn't like some devices on some ports, so it's nice not to have to worry about screwing up your boot procedure when rearranging devices.

Tuesday, March 29, 2011

Cloud Storage: Expensive

Amazon's recently-announced Cloud Drive service shows why Internet storage still doesn't make economic sense unless you don't have much data.

The biggest storage tier, 1000 GB (1TB), is $1000/year. My cheapo FreeBSD server cost about $550 (plus my time) to build and stores 6000 GB in a reasonably redundant manner. It uses about 1kWh/day of electricity and thus costs about $26/year to operate.

Storing your own data is more risky. Probably. If the house catches fire my data will be destroyed. If I kick the box over while it is running, more than 2 drives will probably crash and the data will be destroyed. Then again, if you read the terms of service for Cloud Drive, Amazon is not going to compensate you for data loss if they screw up and delete your data, either. Maybe they make tape backups like Google does, maybe not. You don't really know.

Convenience is probably better (for some folks) with a service like Cloud Drive: you can use it with your phone or whatever though a nice interface. Me, I have to setup ssh and deal with people trying to break in. (Then again, I can also use my server to run IRC and host a website and do other useful work.)

Even if you have FiOS or some ridiculous Internet connection, transfer to local storage is probably going to be faster. I can transfer data to and from my NAS box at gigabit speeds which makes it trivial to back up my workstation every hour. I suspect that would be painful with a service like Cloud Drive.

I know they are targeting mostly non-technical folks with this service. Still, it is interesting to observe the costs from a power user's perspective.

Thursday, March 3, 2011

Yet another HTML5 canvas demo

I made a canvas demo that takes advantage of the new GPU acceleration in the beta Chrome builds. It sorta works in Firefox, too, but good god it's slow. I get 60fps at 1680x1050 in Chrome.

Friday, January 21, 2011

Python: Remove Duplicate Spaces

>>> from functools import partial
>>> import re
>>> undup_spaces = partial(re.compile(r' +').sub, ' ')
>>> undup_spaces('  foo bar  foobar    foo bar    bar fu   ')
' foo bar foobar foo bar bar fu '

Monday, January 17, 2011

Python itertools grouper with truncation

I wanted something like a "grouper" (see docs for itertools) that truncated the final list instead of filling with a fillvalue. This solution is not very efficient since the loop is in Python code and the intermediate lists must fit into memory, but it does work:
def grouper(n, iterable):
  "grouper(3, 'ABCDEFG') --> ABC DEF G"
  buf = []
  for i, c in enumerate(iterable):
    if (i+1) % n == 0:
      yield buf
      buf = []
  if buf:
    yield buf
Is there a faster way to do this using the itertools primitives?

Tuesday, January 11, 2011

Quick Python zlib vs bz2 benchmark

I use the zlib module a lot on Google App Engine; often the tiny CPU time for decompression is a good tradeoff to save disk space. I was curious how bz2 compares so I ran this short benchmark.

The test file was this plaintext book, a highly-compressible source. Columns are: level, time, bytes uncompressed, bytes compressed, ratio.

% ./bench.zsh

zlib compress
0   6.98ms 640599 640700 1.000
1  21.22ms 640599 274195 2.336
2  25.08ms 640599 261638 2.448
3  34.24ms 640599 249649 2.566
4  36.41ms 640599 241500 2.653
5  54.24ms 640599 232545 2.755
6  77.22ms 640599 228621 2.802
7  87.94ms 640599 228032 2.809
8 112.49ms 640599 227622 2.814
9 113.03ms 640599 227622 2.814

zlib decompress
0   1.54ms
1   6.39ms
2   6.13ms
3   6.02ms
4   6.22ms
5   5.96ms
6   5.94ms
7   5.90ms
8   5.89ms
9   5.94ms

bz2 compress
1 105.30ms 640599 196752 3.256
2 103.42ms 640599 186082 3.443
3 105.40ms 640599 180905 3.541
4 104.95ms 640599 177642 3.606
5 113.12ms 640599 176232 3.635
6 110.45ms 640599 173153 3.700
7 113.06ms 640599 169634 3.776
8 110.27ms 640599 169634 3.776
9 111.43ms 640599 169634 3.776

bz2 decompress
1  36.40ms
2  35.79ms
3  36.35ms
4  36.81ms
5  41.18ms
6  44.86ms
7  48.96ms
8  48.45ms
9  47.95ms

Conclusion: probably not worth it. bz2 at level=4 takes about 7 times longer to decompress than gzip at level=9 for only a modest improvement in the compression ratio from 2.8 to 3.6.

Interestingly for write-heavy workloads bz2 may actually be the better choice since compression time is not much worse than gzip at level=9.

I think it's better not to use the timeit module for this kind of benchmark since in typical usage you will just be compressing/decompressing some given data once. If the operations speed up in repeat runs due to caching (and they do), that doesn't reflect typical usage. Starting a new python process for each test seems to reduce cache effects.

Anyway, here is the code.

import zlib
import bz2
import time
import sys

level = int(sys.argv[1])
mod = zlib if int(sys.argv[2]) else bz2
is_decompress = int(sys.argv[3])

with open("pg4238.txt") as f:
  data =

if is_decompress:
  c_data = mod.compress(data, level)

t = time.time()
if is_decompress:
  data = mod.decompress(c_data)
  c_data = mod.compress(data, level)

print level, "%6.02fms" % (1000*(time.time() - t)),
if not is_decompress:
  print len(data), len(c_data), "%.03f" % (float(len(data))/len(c_data))
echo 'zlib compress'
for level in {0..9}; do python $level 1 0; done
echo '\nzlib decompress'
for level in {0..9}; do python $level 1 1; done
echo '\nbz2 compress'
for level in {1..9}; do python $level 0 0; done
echo '\nbz2 decompress'
for level in {1..9}; do python $level 0 1; done

Thursday, December 30, 2010

Python Kakuro Solver Now Pretty Fast

Ahh, winter holidays. Finally got around to working on my Python Kakuro Tools again. I've had the top spot for "python kakuro" in Google for a while, but the code was embarrassingly bad and slow. Now it is solving the fairly complex puzzle below in less than 200ms. Not bad for an interpreted language.

Still a lot of work to do before it's ready for a 1.0 release, but at least the solving engine is pretty much done. I wonder how much slower this code is rather than something written in a constraint programming language (like the process described in this paper) or a native C implementation. C implementations can get away with using a platform word to store the possible values for each cell which enables fast bitwise operations; I'm sure that helps a lot. I used Python's sets which are a more generic solution since they're not limited to 32 or 64 cell values (depending on the platform), but also much slower.

  0  | 0,24| 0,28|  0  | 0,14| 0,33| 0,13| 0,8 | 0,39|  0  | 0,27| 0,45| 0,16| 0,8 |  0  | 0,20| 0,32| 0,21|
 12,0|  1  |  1  |16,43|  1  |  1  |  1  |  1  |  1  | 30,0|  1  |  1  |  1  |  1  |11,10|  1  |  1  |  1  |
 43,0|  1  |  1  |  1  |  1  |  1  |  1  |  1  |  1  | 36,6|  1  |  1  |  1  |  1  |  1  |  1  |  1  |  1  |
 32,0|  1  |  1  |  1  |  1  |  1  |22,19|  1  |  1  |  1  |  1  |  1  |  1  |11,11|  1  |  1  |  1  |  1  |
  0  |33,13|  1  |  1  |  1  |  1  |  1  |42,45|  1  |  1  |  1  |  1  |  1  |  1  |  1  |12,40|  1  |  1  |
 15,0|  1  |  1  |  1  | 29,8|  1  |  1  |  1  |  1  |  1  |  1  |  1  |18,30|  1  |  1  |  1  |  1  |  1  |
 11,0|  1  |  1  |  1  |  1  |24,10|  1  |  1  |  1  | 18,0|  1  |  1  |  1  |  1  |22,12|  1  |  1  |  1  |
  0  | 0,24|34,36|  1  |  1  |  1  |  1  |  1  | 0,35|  0  |23,26|  1  |  1  |  1  |  1  |  1  | 0,21| 0,13|
 24,0|  1  |  1  |  1  |19,26|  1  |  1  |  1  |  1  | 6,6 |  1  |  1  |  1  |10,27|  1  |  1  |  1  |  1  |
 20,0|  1  |  1  |  1  |  1  |  1  |42,27|  1  |  1  |  1  |  1  |  1  |  1  |  1  |23,28|  1  |  1  |  1  |
 12,0|  1  |  1  | 33,7|  1  |  1  |  1  |  1  |  1  |  1  |  1  | 33,7|  1  |  1  |  1  |  1  |  1  | 0,22|
 27,0|  1  |  1  |  1  |  1  |23,12|  1  |  1  |  1  |  1  |  1  |  1  |24,12|  1  |  1  |  1  |  1  |  1  |
 37,0|  1  |  1  |  1  |  1  |  1  |  1  |  1  |  1  | 36,0|  1  |  1  |  1  |  1  |  1  |  1  |  1  |  1  |
 7,0 |  1  |  1  |  1  | 10,0|  1  |  1  |  1  |  1  | 16,0|  1  |  1  |  1  |  1  |  1  | 8,0 |  1  |  1  |
  0  | 0,24| 0,28|  0  | 0,14| 0,33| 0,13| 0,8 | 0,39|  0  | 0,27| 0,45| 0,16| 0,8 |  0  | 0,20| 0,32| 0,21|
 12,0|  7  |  5  |16,43|  1  |  3  |  4  |  2  |  6  | 30,0|  8  |  9  |  6  |  7  |11,10|  8  |  1  |  2  |
 43,0|  8  |  4  |  5  |  3  |  6  |  9  |  1  |  7  | 36,6|  5  |  6  |  2  |  1  |  3  |  7  |  8  |  4  |
 32,0|  9  |  8  |  6  |  2  |  7  |22,19|  5  |  4  |  2  |  3  |  7  |  1  |11,11|  2  |  5  |  3  |  1  |
  0  |33,13|  6  |  7  |  8  |  9  |  3  |42,45|  9  |  3  |  6  |  8  |  7  |  5  |  4  |12,40|  9  |  3  |
 15,0|  8  |  3  |  4  | 29,8|  8  |  2  |  6  |  5  |  1  |  4  |  3  |18,30|  2  |  1  |  6  |  4  |  5  |
 11,0|  5  |  2  |  3  |  1  |24,10|  7  |  9  |  8  | 18,0|  1  |  5  |  9  |  3  |22,12|  9  |  7  |  6  |
  0  | 0,24|34,36|  9  |  7  |  4  |  6  |  8  | 0,35|  0  |23,26|  2  |  8  |  1  |  9  |  3  | 0,21| 0,13|
 24,0|  7  |  9  |  8  |19,26|  2  |  1  |  7  |  9  | 6,6 |  2  |  1  |  3  |10,27|  3  |  1  |  2  |  4  |
 20,0|  2  |  6  |  1  |  8  |  3  |42,27|  5  |  8  |  3  |  7  |  4  |  6  |  9  |23,28|  8  |  6  |  9  |
 12,0|  5  |  7  | 33,7|  4  |  1  |  9  |  3  |  6  |  2  |  8  | 33,7|  4  |  8  |  9  |  7  |  5  | 0,22|
 27,0|  6  |  8  |  4  |  9  |23,12|  8  |  2  |  3  |  1  |  5  |  4  |24,12|  1  |  8  |  2  |  4  |  9  |
 37,0|  3  |  2  |  1  |  5  |  9  |  6  |  4  |  7  | 36,0|  1  |  2  |  8  |  7  |  5  |  4  |  3  |  6  |
 7,0 |  1  |  4  |  2  | 10,0|  3  |  4  |  1  |  2  | 16,0|  3  |  1  |  4  |  2  |  6  | 8,0 |  1  |  7  |

Wednesday, December 29, 2010

Weather Alerts for XMobar

By default xmobar's weather module only fetches temperature, humidity, and some other basic weather data for your location.

This script fetches weather alerts (like "Winter Storm Warning") for your county in an xmobar-friendly format. Only for US users since it queries the National Weather Service.

I like it for keeping track of winter weather alerts especially.

Thursday, December 23, 2010

Backup Strategies: Linux to FreeBSD ZFS

I use FreeBSD on my home server mainly for the ZFS support, but I run Linux on my workstation and laptop.

I've been trying to figure out the best backup strategy for this environment. If I was running FreeBSD on my client machines and they had local ZFS filesystems, I could use snapshots with ZFS send to propagate changes to the server. This would be really great but is impractical until a fast, stable version of ZFS arrives in Linux (probably never).

For now I've settled on mounting an NFS share from the server on each machine and then using rsync to copy changes from the local volume to the NFS share via cron job. Using NFS is faster than tunneling over rsync over ssh since the server is slow. I run rsync in the mode that propagates deletions to the destination volume and rely on zfs snapshots on the server to recover any accidentally deleted files.

I guess it's not perfect, but probably the best I can do without switching my client machines to FreeBSD.

Saturday, December 18, 2010

Ubuntu Slow After Hibernate Resume

I use hibernate probably more often than I should. It's not totally stable on my hardware and sometimes it crashes on resume, or does weird things like crashes on the first resume attempt but works fine on the second. And God help you if you install some system update and then accidentally hibernate instead of restarting. But even with all the weird idioscyncracies it's still faster than rebooting every morning, so I use it.

I've already had to write a custom hibernate/resume script to unmount/remount my NFS share, lest it hang on resume and make the system unusable by locking up every shell when it tries to read my home directory (still haven't figured out a way out of that pickle). I'm thinking about also adding swapoff -a; swapon -a to the resume script. For whatever reason linux seems to leave everything on the swap disk by default and only page stuff back in as it's accessed, which means the system is slow for a few hours after hibernate resume every time you access some program you haven't accessed yet. I rarely use even half my RAM so it makes more sense to page everything back in right after resume so the system is snappy again.

Isn't that true for most people? I wonder why this isn't done by default.

Wednesday, November 3, 2010

Bit Rot, or Why Hard Drives Suck

The scary thing about hard disks is that failure can be silent. The whole device doesn't have to go bad for 10 or 50 or 1000 sectors to suddenly become unreadable. And unless you are actively checking the drive periodically with a SMART surface scan or a zfs scrub, there could bit-rot you don't know about.

Here is an example of what bit rot looks like, from my home storage NAS:

[root@bsdnas /home/bthomson]# smartctl -l selftest /dev/ada0

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed: read failure       90%     22587         624048385

"Sorry, sir, I'd like to read your data at 624048385 but I just can't!"

Some further inspection reveals this disk has 26 sectors it is concerned about and 17 which are fully hosed and cannot be read:

  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       -       26
198 Offline_Uncorrectable   0x0010   200   200   000    Old_age   Offline      -       17

This drive is only 2 years old. If it was out of warranty I would replace it or rewrite it with zeros so it would reallocate the Pending_Sectors and hope for the best, but since a warranty exchange is only $6 and these sectors have gone bad recently it's probably not a bad idea to exchange it.

Of course the drive you get on exchange is refurbished and could be in even worse shape than the one you exchanged, so it's somewhat of a crapshoot.

Wednesday, October 27, 2010

Linux: Dual 2209WA at 75Hz

75hz is a nice refresh rate for LCDs since 23.976 divides more evenly into 75hz than it does into the more-standard 60hz. That makes the delay between individual frames in progressive cinema film more consistent so pans appear less jerky without requiring the material to be telecined. 29.97 and 59.94fps video will look worse at 75hz than 60hz, but both are less common than 23.976fps material.

20% extra framerate is also great for games and 3d graphics too.

The modeline has to be specified manually since Sony didn't include a 75hz timing in the monitor's EDID data because it is a little bit outside of the spec. 1680x1050 @ 75hz with normal pixel timings exceeds the bandwidth of a single DVI-D connection, but by trimming the sync timings down below spec you can make it work. This should not harm the monitor although it might make it more likely to desync occasionally.

This Xorg.conf will enable dual 2209WA monitors at 75hz for nvidia cards. I'm on Ubuntu but it shouldn't matter.

# Xorg.conf
# Modeline configuration info:
# vim :set ft=xf86conf:

Section "Monitor"
  Identifier "Monitor0"
  ModelName  "DFP-0"
  # 1680x1050 @ 60.00 Hz (GTF) hsync: 65.22 kHz; pclk: 147.14 MHz
  #Modeline "1680x1050_60"  147.14  1680 1784 1968 2256  1050 1051 1054 1087  -HSync +Vsync

  # From
  Modeline "1680x1050_75" 153.290 1680 1712 1744 1842 1050 1053 1059 1089 -HSync +Vsync

Section "Screen"
  Identifier     "Default Screen"
  Monitor        "Monitor0"
  Device         "Device0"
  DefaultDepth   24
  Option         "TwinView" "1"
  Option         "TwinViewXineramaInfoOrder" "DFP-1"
  Option         "metamodes" "DFP-0: 1680x1050_75 +1680+0, DFP-1: 1680x1050_75 +0+0"
  SubSection "Display"
    Depth   24

Section "Module"
 Load "glx"

Section "Device"
  Identifier  "Device0"
  Driver      "nvidia"
  Option      "NoLogo" "True"
  Option      "UseEDID" "false"
  Option      "ModeValidation" "NoDFPNativeResolutionCheck"
  Option      "ExactModeTimingsDVI" "True"
  Option      "ModeValidation" "NoDFPNativeResolutionCheck,NoVirtualSizeCheck,NoMaxPClkCheck,NoHorizSyncCheck,NoVertRefreshCheck,NoWidthAlignmentCheck,NoExtendedGpuCapabilitiesCheck"

Saturday, October 23, 2010

Linux I/O Schedulers with NCQ-Capable Disks

An interesting post over on Enterprise Data Management Musings asks in part whether the linux I/O scheduler is even relevant for non-RAID workloads on SATA drives that support NCQ.

I've been frustrated with the performance of a particular SATA disk on my workstation which is tasked with batch operations when it's saturated with I/O, so I decided to switch it from the cfq scheduler (default for all disks in Ubuntu) over to noop. This should increase total IO throughput (and consequently reduce total time to completion of all jobs) at the risk of starving some processes for an indeterminate amount of time. That's not a concern since this disk is only used for batch jobs.

Here are some interesting performance graphs showing the differences between the schedulers.

Instructions for making permanent changes are available here.

Sunday, October 3, 2010

D510UD AHCI support

Turns out the GIGABYTE GA-D510UD does support AHCI on all four sata ports, but you need to have verison f4 of the bios.

For FreeBSD 8.0 if you add ahci_load="YES" to your /boot/loader.conf you'll be good to go. I don't think you need to actually enable AHCI in the bios, but it doesn't hurt.

Tuesday, August 3, 2010

Using Google's Closure Compiler with GNU Make

I came into the unix scene a bit late and have never really used GNU Make for anything beyond compiling a few C files. After replacing an ad-hoc python-based build system with something based on Make, though, it has been growing on me.

Here is the part of the build script used to integrate with Google's Closure Compiler, in the hopes that it might be useful. This is a long way from being general, but some bits of it might be helpful. It creates one main output file, js.js, from a bunch of libraries and any code needed from the closure library. There is a simple compilation pass and an advanced compilation pass because I use some libraries that don't work with the advanced compilation. A source map is also created.

I read Recursive Make Considered Harmful, so the file below is called which is imported into the main Makefile with:

include makefiles/

Here is

CLOSURE_COMPILER = java -jar ~/opt/compiler.jar
CALC_DEPS = std_root/dev_only/closure/bin/

JS_ADVANCED = <js files which can be compiled with advanced compilation>
JS_SIMPLE = <js files which must be compiled with simple compilation>
JS_EXTERNS = <js files containing "externs" (see closure compiler documentation)>

# Modify me to point to where your JS files are located.
vpath %.js std_root/static/:std_root/dev_only:js_extern/

# Concatenate the simple and advanced output together
js.js: simple.js advanced.js
 cat $^ > $@
 mv js.js std_root/static/

# I'm not 100% happy with the way externs are set up here, but it works! Huzzah!
advanced.js: $(JS_ADVANCED) $(JS_EXTERNS)
 $(CLOSURE_COMPILER) --compilation_level=ADVANCED_OPTIMIZATIONS --create_source_map ../../js-source-map \
  $(addprefix --js=, $(filter-out js_extern%, $^)) \
  $(addprefix --externs=, $(filter js_extern%, $^)) \

# Note about simple optimizations: Unless you are going to include jquery in the
# compilation, nothing related to jquery should go in either since the symbols
# will need to be fixed in each file manually... which is just way too much work.
# (of course if no weird symbols are used you can go ahead and put it in)
simple.js: $(JS_SIMPLE)
  $(addprefix --js , $^) \

# Inserts dependencies from closure library
cat.js: cleanse.js
 $(CALC_DEPS) -i $^ -o script > $@

# Remove any lines containing a console statement
cleanse.js: main.js
 cat $^ | grep -v "console." > $@

# nothing to do for normal js files
# note: overriden for -soy.js
%.js : ;

# Targets which don't need to be saved can go in here
.INTERMEDIATE: cleanse.js

.PHONY: clean-js
 rm -f advanced.js
 rm -f cat.js
 rm -f std_root/static/js.js
 rm -f simple.js

Wednesday, June 16, 2010

Sorry Nouveau, You Suck

You know, I'm as much for free software as the next zealot. I get it. Proprietary software creates duplicated effort, Windows XP is everywhere and cannot be eradicated, software patents suck, yada yada yada.

But come on, Ubuntu. Nouveau by default? It makes Xorg suck 75% of my CPU just to display a youtube video. And since my hardware x264 decoders are idle since, oops, that's not supported either, I need some CPU left over to decode the dang thing!

I think this should be changed to read:

Ubuntu cannot fix or improve these drivers. However, they are already so much better than the ones we can fix and improve that if you don't enable them you're just going to spew a whole boatload of extra CO2 to the atmosphere for no good reason other than to enjoy your free software righteousness. Does it feel good? And honestly, since you're not very likely to get better help or support if you run into problems with the open source drivers, it's only about the righteousness. We have principles, damnit!

(I love Ubuntu, I really do. Everyone who works on it is providing a fantastic service and I don't mean them any ill-will at all. It's just that this dialog is too ironic not to comment on.)

Friday, June 11, 2010

Mouse Addiction, or Why Accelerator Keys Suck

It's funny, even though I have been using xmonad for a while now and can move between windows faster by keeping my hands on the keyboard, I still instinctively reach for the mouse pretty often. I'd unplug the dang thing but I still need it for web browsing and general web-app usage.

This is why I think the idea of accelerator keys is ultimately a failure: once a common task has been trained into muscle memory using the mouse, it's actually harder to replace that skill with the faster way of doing it (the keyboard) than it would have been to just suck it up and learn the faster way (the keyboard) in the first place.

Learning how to play a musical instrument the wrong way leads to the same problems if my experience with the guitar is any reliable indication.

Sunday, May 16, 2010

Useful links for xmonad on Ubuntu Lucid Lynx

Switching to xmonad was definitely worth it. I'm running it in a gnome session.

Useful linkx:
Some images of what the available layouts look like
Introduction to the xmonad Tiling Window Manager
Adding a dzen2 Statusbar to xmonad
How to get it working with the gnome-panel

Interesting comment over on the tombuntu site:
I really think that tiling WM are the way to go, floating every window just wastes so much time in reorganization and resizing. As to why this is still the predominant paradigm I would venture that “tradition” has much to do with it (going back to the first WMs by Xerox) and also the fact that most tiling WMs target a keyboard-using audience which is traditionally limited to advanced computer users. - Jedai
Only thing that worries me a bit is that I read somewhere Gnome isn't going to work as well with third-party window managers in v3.0. Ah, well...

Friday, March 26, 2010

Python micro-optimization theatre: call dispatch

Python has no switch statement, so if you want to switch control flow based on the value of a variable there are two obvious ways to do it:
if v == 'a':
elif v == 'b':
elif v == 'c':
def a():

def b():

dispatch = {'a': a,
            'b': b,

How do these stack up speed-wise? It depends on the access pattern and how many branches you have. If the first branch will usually be taken (v == 'a'), you'll be better off with the first approach. If you usually go down 4 or more branches, dict dispatch is better.
# dict dispatch always takes the same amount of time
In [3]: %timeit f1('g')
1000000 loops, best of 3: 357 ns per loop


# 7 if statements before we find a match takes longer than
# dict dispatch
In [4]: %timeit f2('g')
1000000 loops, best of 3: 491 ns per loop

# 4 is the same as a dict dispatch
In [7]: %timeit f2('d')
1000000 loops, best of 3: 359 ns per loop

# Taking the first branch is always faster
In [10]: %timeit f2('a')
1000000 loops, best of 3: 215 ns per loop
Code used:
def a():

def b():

def c():

def d():

def e():

def f():

def g():

dispatch = {'a': a,
            'b': b,
            'c': c,
            'd': d,
            'e': e,
            'f': f,
            'g': g}

def f1(v):

def f2(v):
  if v == 'a':
  elif v == 'b':
  elif v == 'c':
  elif v == 'd':
  elif v == 'e':
  elif v == 'f':
  elif v == 'g':

Tuesday, March 23, 2010

ZFS NAS for Home Backup on FreeBSD 8

Update one year later: I've quite happy with this server. In retrospect, I should have gotten a board with more SATA ports or a PCI express x1 slot. Due to only 4 SATA ports I ended up having to run two disks off a PCI SATA card which gets far less than optimal throughput, but performance is still mostly acceptable for my uses.

The CPU speed is a little bit annoying: it is not normally an issue but definitely noticeable when trying to rebuild the world after a FreeBSD upgrade, for example. Still, I think I am willing to put up with occasional slowness for the ultra-low power use.

I'll probably keep running this thing until the hardware breaks since it works well enough.

Disclaimer: I am neither a sysadmin nor a data storage expert, just a guy trying to solve his problems without spending too much $$$. I have less than perfect understanding of these issues! Please leave a note if you find a mistake.


Data storage is always an interesting challenge. Most of my text data (source code, writing, etc) is mirrored to every device I own and a few in the cloud. This is my most valuable stuff and I'm pretty confident the cloud and all the devices I own aren't going to go kaput simultaneously, so it's mostly safe.

But I can't mirror my videos and digital photos and such to every device; there are way too many bits!

Remote backup/storage over the Internet is possible, but also expensive and slow: bandwidth and storage costs are high and available bandwidth itself is also pretty limited unless you are blessed with a fiber connection (I'm not). Worse, even though many backup services claim your data will be safe, if you look at the fine print most of them will only accept liability up to whatever you paid for the service in the event of lost data. That means their incentives are not really that great and you need to keep at least one reliable local mirror of everything as well. (Of course there are also annoying security implications for remote backup that are less of an issue if you can keep everything at home.)

So for now I have chosen to forego the remote option and just manage my own storage and backups locally. For three years or so I have been keeping everything in a big ext3 filesystem on software raid 51 with periodic (read: not often enough) backups via rsync to another ext3 on raid 6 (with older, less reliable disks). This has worked fine and I have not even had a dead volume in either of the arrays yet. But in addition to total drive failure there is also bitrot to worry about, and as the amount of data in my arrays grow I feel less confident in this solution. ext3 doesn't do snapshots, either, so if something gets trashed on the main array and backed up to the secondary I'm hosed. (Not to imply that's ext3's fault: there are plenty of userspace tools that could solve this problem, I just don't use any)

Neither does ext3 store checksums for files and so file integrity on the main array cannot be reverified on an automated basis. If everything is working properly there will be no errors, of course, but this assumption is not rock solid for a variety of reasons (crashes, memory corruption, kernel bugs, etc). It would be nice to be able to check all my files every week or so and have some certainty that they are still ok.

Enter ZFS. ZFS presently has many advantages over ext3/4 with software raid2 including deduplication, elimination of the raid-5 write hole, awareness of the RAID-like redundant data at the filesystem layer, end-to-end checksumming, and better error reporting. As many users experiences' show there can often be unexpected problems somewhere in the data pipeline that cause bit errors to show up. The checksums are great here because they enable these problems to be detected and then corrected if there is enough redundant data. That's a killer feature, especially for my cheap commodity hardware which is (probably) prone to memory errors.3

Of course ZFS has disadvantages too. It was designed to run on reliable enterprise machines with lots of spare resources that never corrupt bits, so it sucks memory and there isn't even a fsck tool. There has been some discussion about that last issue, but it doesn't look like Sun's interested in adding a fsck. That means having backups of data stored in ZFS is very important becausee if the filesystem metadata gets screwed up you are basically hosed. Editing metadata by hand? No thanks.

Despite these issues, I think ZFS is a win overall if you keep backups, so I decided to migrate my data.

Unless you want to use ZFS on FUSE your choices for OS are basically FreeBSD or OpenSolaris. OpenSolaris has ZFS v21 with deduplication and some other nice features, but FreeBSD 8 only has ZFS v14. Despite the newer version I am a bit soured on the idea of running an OpenSolaris machine since apparently Sun doesn't even supply security patches. I wouldn't want to use an OpenSolaris machine as anything but a NAS behind my firewall, then, and that seems like a waste of hardware. Not to mention the uncertain prospects of OpenSolaris in light of the Oracle acquisition.

FreeBSD 8 it is, then. I've never used FreeBSD before, but getting it installed was a breeze and it even emails me daily status and security reports. Awesome.

Low power consumption is more important to me than speed so I chose the GIGABYTE GA-D510UD with Intel Atom D510. It has 4 on-board SATA ports and I have 4 disks in my main array.

ZFS scrubbing uses all 4 logical cores available on the Atom which is good because its single-threaded performance is pretty bad. Bad, as in, my five-year-old laptop beats the pants off it. Scrubbing appears to be CPU-bound rather than IO-bound, which is a bummer, but at least it doesn't use hardly any power. I don't have a Kill-a-Watt but with the high-efficiency PSU I'd guess it's under 45W at idle even with 5 disks spinning.

To actually get to the data I exported an NFS share which I can use from my Ubuntu clients. One problem I ran into is that FreeBSD assigns the first user UID and GID 1001 while Ubuntu assigns them 1000 (the UID and GID need to be the same on both sides for NFS to work properly). Rather than try to change files everywhere on the Ubuntu systems I changed both ids to 1000 on the FreeBSD server and all seems well. Hopefully this won't cause any problems down the road.

Performance over the network isn't great; I get about 25MB/s sustained read/write for single files but generally much slower performance when lots of files are involved. I think this might have something to do with NFS's sync semantics, but I'm not very familiar with it. There's plenty more bandwidth on the gigE so I could keep tweaking on it, but I don't really need it to be faster.

So now the only things left to do are to set up weekly scrubbing of the array, set up periodic zfs snapshots, and to convert my backup server over to FreeBSD and ZFS too. I might cover those in a later post if I have time.

I'm hoping that with main array replicated to a secondary via zfs send/receive I will basically not have to worry about metadata corruption (short of a bug in zfs), but I'm not 100% sure if this is the case. I guess the super-paranoid thing to do would be to make tape backups too, but I really don't want to go there.


[1] One problem with using RAID 5 on cheap drives is that the chances of having an unrecoverable error on one of the good drives after a whole disk fails are actually quite high. (some comments on this issue) So even if you don't expect 2 disks to fail simultaneously it may not be a bad idea to go with RAID 6.

[2] For various reasons Linux is now playing catch-up when it comes to filesystems. Btrfs promises many of the advantages of ZFS when it's ready for production, but it's not there yet. Since Oracle bought Sun and now is responsible for ZFS and Btrfs, the situation is a bit weird at the moment.

[3] None of that fancy raid stuff does you any good if you are writing garbage to the disk. For almost 2 months I had my array running in a machine with bad ram! Who knows what files might have been corrupted in that time... without checksums it's hard to tell.

ECC memory adds some protection from this kind of thing but it really adds to the cost, too: the cheapest motherboard on Newegg that supports ECC is a year and a half old and costs $124. (Non-ECC boards start at $40) And, to really be safe, all your workstations and laptops that will access the NAS should have ECC memory as well, right? So much for keeping it cheap... Still, if statistics are to be believed, ECC memory is more important than commonly believed.

If the market really cared about ECC I think it would be a relatively cheap addition, but probably most users of PCs (corporate and home alike) are so much more overwhelmingly likely to have crashes and data corruption due to their operating system software or operator error that any failures due to bit flips just get lost in the noise. A shame, 'cos I really wish it wasn't such an expensive feature.

Thursday, March 4, 2010

Simulate Slow Network in Firefox, Chrome, etc

I assumed it would be difficult to set up network delays at the kernel level so I was looking around at browser extensions. Firefox has Tamper Data and Firefox Throttle, but the former only delays HTTP requests and the latter only works on Windows. Not much available for Chrome yet.

But actually, it is very easy to simulate delays at the kernel level:

$ sudo tc qdisc change dev lo root netem delay 400ms
$ ping localhost
PING violet ( 56(84) bytes of data.
64 bytes from violet ( icmp_seq=1 ttl=64 time=800 ms

Increasing latency is the easiest way to see results, but netem can also simulate throttling, packet loss, and other kinds of problems. Way cool!

Sunday, February 28, 2010

This is why we never have enough memory

It's always fun to casually notice a minor extension used to make Gmail your default email client is using more ram than several of your earliest computers had, combined. When I think of the future, I envision each checkbox and radio button requiring 1 gigabyte of ram or more. Now that's progress!

Saturday, February 27, 2010

Page Fault Hell: The Worst Thing About Linux On The Desktop

Pagefaulting. Endlessly. You know, thrashing.

In theory, having 12,582,912 times as many bits of memory as the Apollo Guidance Computer would mean that I could run 12 million tasks roughly as complex as navigating a rocket to the moon and back simultaneously without breaking a sweat. In practice, things aren't quite so rosy.

It's happened to me maybe four times in the last year. I'll be minding my own business cranking out some code or whatever, when suddenly the whole system gets a little slow. I look at the memory usage indicator and it's just gone to solid dark green, indicating 100% usage. Crap. Some buggy background process has suddenly decided it desperately needs 4.5GB of ram for god knows what. Once it was vlc, once some lame-o toolbar widget, once it was a random python process... Ugh.

By now the GUI is totally locked. I desperatly mash ctrl-alt-f1, knowing that if I can just get to the console in time I can kill the process before everything I'm working on has been paged out to disk. If the kernel is in a particularly bad mood it may be some number of minutes before I can kill the evil process.

Apparently there is an OOM killer in the kernel which is sometimes activated in these situations, which aims to:

 * 1) lose the minimum amount of work done
 * 2) recover a large amount of memory
 * 3) don't kill anything innocent of eating tons of memory
 * 4) kill the minimum amount of processes (one)
 * 5) try to kill the process the user expects us to kill

Of course, a lot of the server guys don't like it. But it sounds like the perfect solution for a desktop system. If only the UI wouldn't block so much before it had a chance to work, the user might not even notice there had been a problem.

I found a bug in the Ubuntu repository about this, but it hasn't gotten much attention. My guess is most Ubuntu users don't abuse their machines enough to see this very often. The kernel maintainers argue that this isn't a bug, that the kernel is just doing the best it can with what it's told. Well, OK. Maybe a desktop system is a special use-case that deserves special behavior. But what exactly are we supposed to tell the kernel? Ideally, I'd say something like "If a process suddenly (within 1 minute) eats all the free memory and is consuming enough that there's pressure on the rest of the system, don't page out anything I'm working on that has been resident for days. Reduce this new process's priority for at least 5 minutes or so to give me time to analyze the situation and decide if it needs to be killed.

Now, there's ulimit and the various sysctl parameters like vm.swappiness that can all be tweaked and adjusted. Some of these look promising, but I'm reluctant to change any of them for fear I will make the problem worse or add a new, unrelated problem.

At any rate, this is a place where Ubuntu really falls down compared the commercial competition. I can't speak for OS X, but I know the Windows 7 UI, which you can use to shut down misbehaving processes, remains responsive even in the face of extreme memory pressure. Sure, it might take longer to finish the memory-hungry (and in all likelyhood, buggy and broken) process than a linux box would, but having the UI remain responsive at the cost of slower background tasks is just a good design choice.

Tuesday, February 16, 2010

Interesting old post mostly about Lisp/C at JPL

Interesting old post mostly about Lisp/C at JPL. A timeless story, except nowadays it's more commonly people arguing futilely against Java in favor of Ruby or Python or ...

Python: Iterating over a list vs. iterating over dict.items()

While cleaning up some code today, I was wondering: how efficient is Python's dict.items()? If we're usually iterating over a list of items but don't particularly care about their ordering, is it worth it to use a dict instead so we can benefit from an occasional key-based lookup? Here's the code:
# Build a long list and copy each member
[x for x in a]

# Build a list, put each member into a dict,
# and copy each member of the dict
b=dict((x,x) for x in a)
[x for x in c]
Iteration over the dict's items is just a little bit slower:
$ -s 'a=range(30000)' '[x for x in a]'
1000 loops, best of 3: 1.35 msec per loop
$ -s 'a=range(30000)'\
            -s 'b=dict((x,x) for x in a)'\
            -s 'c=b.values()'\
               '[x for x in c]'
1000 loops, best of 3: 1.37 msec per loop
But obviously lookups in the dict will be much faster. There's extra memory usage too, of course. Still, not a bad deal. Of course, you don't want to put dict_name.values() inside the loop or the extra lookup on each pass will make performance much worse:
$ -s 'a=range(30000)'\
            -s 'b=dict((x,x) for x in a)'\
               '[x for x in b.values()]'
100 loops, best of 3: 2.5 msec per loop

Sunday, November 15, 2009

An NES Emulator written in... wait for it... Javascript!

This is an NES emulator written in javascript. We are using dynamic languages to write emulators now? I know, I know, it's crazy.

Performance must suck, right? Well, no, it easily runs at 60fps in Chrome. But I can barely get 5fps out of it in Firefox 3.5. This agrees with my subjective impressions of Javascript performance in both browsers: although Firefox appears only perhaps twice as slow as Chrome in many synthetic benchmarks, for many real world applications it is almost an order of magnitude slower. Add to that Firefox's ridiculous garbage collector that can lock the entire browser, including the UI, for 4 seconds at a time at 60 second intervals repeatedly going over dead Javascript objects that it is never able to collect, and you've got a seriously, fundamentally broken browser.

Sorry Firefox, you are dead to me.

Wednesday, November 4, 2009

Nose: multiprocess

One of the coolest things about Nose's multiprocess plugin (aside from the fact that it can run your whole test suite in a fraction of the time on a quad core) is that it discovers, by way of causing your tests to fail, many of the areas where you didn't exactly write religiously-pure unit tests.

This is a rather ad-hoc and incomplete method of finding these problems, though. I'm reminded of a tool I played with in college that intentionally ran a program's threads in every possible order looking for deadlocks: how about running your unit tests in every possible order to automatically discover the dependency problems?

Of course there's a combinatorial explosion to worry about, but it might be worth the CPU cost every time you add a bunch of new tests.

Saturday, October 31, 2009

Self-hacks: Undistorting Your Availability Heuristic

Wikipedia will tell you the availability heuristic is a mental shortcut we use to predict the frequency of an event based on how easily an example can be brought to mind.

Modern media distorts our ability to use this built-in heuristic effectively by discussing and presenting rare events far too frequently. In fact, that's what media does almost by definition. Unfortunately, we aren't well-adapted for this scenario: you can't just turn the heuristic off in your decision-making and thought processes.

For example, if you've been paying attention to popular media lately, you will probably think about H1N1 if you think about death. (Of course, your actual risk from H1N1 is small compared to more traditional factors.) Along the same lines, particuarly earlier this decade you might have thought terrorism caused a significant loss of life globally compared to other causes. Probably more people are afraid of planes than cars even though it's clearly irrational.

We could come up with tons of other examples, but let's skip to the useful part: how can you fight back against media abuse of your own cognitive limitations? Unfortunately, merely reading this post and becoming more aware of the bias will not tend to increase the accuracy of your decisions, even though you probably think it will. But one easy way is to stop paying attention to the popular media so you can calibrate more accurately. Another way is to find accurate data (to the extent that such exists) on issues you care about and consult it frequently, such that the facts are always fresh in your memory for the heuristic to pull up.

I haven't really managed to develop a SuperMemo habit I can stick with, but the incremental reading tool is useful for self-programming. I imagine you could come up with a more efficient application if you had the goal of "undistorting the availability heuristic" instead of "maximizing retention of fact-based information". One could get a disproportionate amount of benefit from just 5 minutes of review per day.

Sadly, it's easier to just turn on the TV.

Tuesday, October 20, 2009

Default Deny Is Good

To me, the biggest failure of the major desktop operating systems is their lack of an easy, it-just-works sandbox environment for running untrusted computations. This omission has allowed browsers to do an end-run around the OS companies to the point where we now have Google attempting to make the OS itself the afterthought.

They have a point. The average user is not capable of judging whether they can trust a piece of downloaded code or not. Hell, generally the sophisticated power user with full access to an application's source code does not have time to make a sound judgement just to try out some new toy. So, people take shortcuts and mistakes happen.

The right answer is to make the initial judgement irrelevant, which is exactly what's happening as web browsers evolve into a general computing platform. As more and more software moves into the browser, the implicit change to a default-deny security model on the client side will help stem the tides of malware while letting us encourage users to try software they (or some big corporation) haven't personally vetted as safe. And in a massively multicore environment where the worst a process can do is launch an (intentional or unintentional) denial of service attack against a few of your cores, why not?

In a world with default deny, at least it becomes clear exactly who you are trusting with your data.

Sunday, October 18, 2009

Python: How long does it take to remove a key from a dict?

I try to do micro-benchmarks in python as little as possible, but it never hurts to be aware of the cost of your primitive operations. That's regardless of the language you're coding in.

So, today's experiment... Say you want to remove a key from a dict whether it exists or not. How long does it take? Depends on your method of choice...

pop: 0.24 usec/pass
try/except: 2.27 usec/pass
if in: 0.11 usec/pass
if has_key: 0.20 usec/pass

if x in d: del d[x] wins for execution time, but there are definitely times when the (almost 20x!) overhead of exception can be worth it.

import timeit
num = 10000

a = {}

setup = """
a = {}
stmt = """
a.pop("hi", None)
t = timeit.Timer(stmt=stmt, setup=setup)
print "pop: %.2f usec/pass" % (1000000 * t.timeit(number=num)/num)

stmt = """
  del a['hi']
except KeyError:
t = timeit.Timer(stmt=stmt, setup=setup)
print "try/except: %.2f usec/pass" % (1000000 * t.timeit(number=num)/num)

stmt = """
if 'hi' in {}:
  del a['hi']
t = timeit.Timer(stmt=stmt, setup=setup)
print "if in: %.2f usec/pass" % (1000000 * t.timeit(number=num)/num)

stmt = """
if {}.has_key('hi'):
  del a['hi']
t = timeit.Timer(stmt=stmt, setup=setup)
print "if has_key: %.2f usec/pass" % (1000000 * t.timeit(number=num)/num)

Tuesday, September 1, 2009

Software RAID 5 in Windows 7

A bit of Googling around reveals that MS has removed the old software RAID 5 feature from Windows 7, although it's still present in Server 2008. My experience with the stuff in Windows 2003 was that it was very reliable, but for some reason much slower than mdadm's implementation on identical hardware. I know there is some complexity around RAID 5 with write holes and potential data corruption, so I don't know if mdadm's implementation is better or just more dangerous. I certainly don't have battery backup for writing the data from cache to the disk in the event of a total power failure, and I hope mdadm doesn't think I do.

But if you are really desperate for software RAID 5 (or you happen to have an old mdadm array lying around, like me), there is one very free, very hacky solution that will let you run software RAID 5 in Windows 7: virtualize linux.

That's right: you can download the free VirtualBox, install linux, map your SATA drives, and then run mdadm just as if linux were running natively. In fact, I created the array with the machine booted into to linux, then switched to Windows and mounted the array with a virtual machine. Samba is probably the best tool for interacting with the ext3 fs from Windows.

Scrub throughput is only about 22MB/s with this setup on my 4x500GB array, but then speed probably wasn't your top concern if you really wanted software RAID 5.

For some reason the Virtualbox documentation is only available as PDF, so I can't link you to the section on mapping physical drives which unfortunately is not as easy as it is with the for-pay VMWare Workstation. Mdadm is pretty easy to use, so getting the physical drives mapped correctly is the hardest part of the process, and may involve painful things like forcing VirtualBox to run as administrator.

Update: Initially I planned for this to be a stop-gap solution while I looked for a different medium to migrate the array to, but I continue to be surprised by how well it works. With a network drive mapped to the Samba share I can even hibernate Windows while the VM is running and everything just picks up right where it left off on resume. Even media files that were playing from the array! The array still works fine from within a native linux boot as well.

Update 2: This post is ranking pretty highly in Google for "Raid 5 Windows 7" and a lot of you are undoubtedly looking for solutions. David wrote in to ask for clairification:

> So you installed VirtualBox (which I assume works like VMWare,
> roughly speaking), mounted all the SATA drives in Linux, used
> mdadm to create a RAID array. I understand that. But the next
> part you say you "switched to Windows and mounted the array
> with a virtual machine". What do you mean by this?

Yes, it is similar to VMWare Workstation.

To clarify: I originally created the mdadm array while running a linux operating system on the host. Later, I wanted to access the array while running Windows 7 as the host operating system.

The fact that I was using linux as the original host OS should be an incidental detail: I was merely trying to illustrate that the solution is not dependent on the host OS. You can switch between linux and Windows and still create, access, or destroy mdadm arrays based on physical devices from within either host OS.

So, you should be able to create an array just fine with Windows as the host OS once the drives are mapped correctly.

> Then you mention Samba -- did you share the drive via Samba in
> the Linux VM and then essentially use Windows' "Map Network
> Drive" to map the share?


> If there are any other specifics you think might be helpful,
> that would be fantastic as well.

VirtualBox doesn't offer a nice interface for setting up physical drives like VMWare does; you have to muck about with XML files and might need to run the VirtualBox process with elevated privileges. If you have a VMWare license I'd definitely try that first. But VirtualBox is free, so...

> Is there any way (VMWare or VirtualBox) to easily make the RAID
> array visible as an actual device in Windows?

Probably not unless Windows 7 has some native support for mounting devices over the network. I don't know of such features but they may exist. Samba is probably also not the ideal choice for a protocol in terms of performance, but it has the advantage that it's easy to setup and the mapped drive behaves almost like a regular drive in the windows explorer.

Wednesday, July 1, 2009

Isn't it time for a high-dynamic range computing or web standard?

Luminance and contrast versus viewing directio...Image via Wikipedia
I sometimes bemoan the fact that there's no widespread support for high dynamic range on the web or in computing in general.

The white that most new LCD panels are capable of producing is way, way too bright in most lighting environments to be used as a background color. It's like staring into the sun (ok, not really). And yet, what color is the background of 75% or more of all web sites?

The only practical solution is to run your panel at a brightness close to 0 and sacrifice that great contrast ratio to save your eyes from burning destruction.

The closest thing I've seen to a general-purpose answer to this problem is Compiz's support for dynamically adjusting the brightness and contrast of particular windows. So you could set, say, every instance of a Firefox window to have reduced maximum brightness and use a dark desktop theme. Now you can turn your panel brightness up so you get some real contrast.

This is not a perfect solution of course because it affects the entire window unilaterally. Ideally, you'd have a way to say "any idiot who makes the background color of the website white should have it changed to a medium grey, but darker photographs should be allowed to have brighter than medium grey highlights." Otherwise you are just back to wasting the dynamic range the panel is capable of.

You might be able to get close to this ideal with stylesheet overrides, but they wouldn't work in every situation unless there was some sort of a standard. And of course you're messing with the designers' intentions which is less than ideal.

You could think of an ideal solution as a ReplayGain for the web. But instead of adjusting the maximum amplitude of the sound, we're adjusting the maximum amplitude of your site's luminance.

I doubt anything like this will become popular unless the LCD panel manufacturers get the ball rolling. Until then, we'll just keep chugging along with them turned down to the minimum possible brightness. Except for the people who don't even realize this is a problem, and stare into a bright lamp all day in their dark office. But unfortunately there is no hope for those folks.

Monday, June 29, 2009

Advantages of Dual Widescreen Monitors In Portrait Orientation

Poor side-viewing image quality of an LCD moni...Image of typical portrait display distortion via Wikipedia

For a few weeks now I have been using two Dell 24" 2209WA panels in portrait mode. These panels are unique in that they are both cheap (about $220 each) and use IPS technology, which means they can be used in portrait mode without terrible, awful view-angle distortion.

One of the major benefits to portrait mode is the increased vertical viewing area for websites. Very few websites are wider than 1050 pixels, but many are very tall.

Furthermore, since Firebug attaches itself to the bottom of the browser window by default, it is much more comfortable to view Firebug and a website at the same time in this orientation.

1050 pixels is wide enough for two 80-column text files side by side, with some extra for line numbering. Vertically I get about 147 lines, although this is really more than is necessary for context when programming so I tend to use some space at the top of the screen for other terminals. Since I often need more than 80 columns in my terminals, this setup works better than trying to put a terminal beside a double-columned vim session in the landscape orientation.

Another advantage to using two medium-sized panels instead of one large panel is that you can turn one half off to save power when you don't need it.

A few problems: Vista and XP do not support ClearType in this orientation. I believe Windows 7 does. Linux does.

NVIDIA's Linux driver seems to have some trouble with rotated displays: there is tearing that is not present when the displays are not rotated. This is probably fixable, but it's not that distracting.

Reblog this post [with Zemanta]

Friday, June 12, 2009

The Best Linux Bitmap Programming Font (Debian/Ubuntu)

After a long search that went through what seemed like just about every page in Google's index, I finally found a great linux programming font! The emacs folks have some good advice.

Before I share it, let me say that good fonts are a necessity for programming. Bitmap fonts win the day on Linux because while anti-aliasing is neat, it's just harder to design a good curve-based font than a good bitmap font. I think part of the problem is that the font rendering systems in Linux are invariably moving targets, and you really need to design curve-based fonts with the specifics of the rendering system in mind if they're going to look great at the end. Consolas on Windows is great, but they designed it specifically for ClearType.

On the other hand, if someone designs a bitmap font system that allows the author to specify the antialiased pixels and integrates it into X, that might be worth using.

Anyway, the font is Neep. You can get it here. Jim's hosting seems to be broken, but the link to the mirror download is still working.

As for loading it... Well. The current situation with bitmap fonts in Ubuntu/Debian is a bit strange: they're basically disabled to ensure that old bitmap Xfonts (like Times) don't show up in webpages and make everything ugly.

Frankly I could care less about web fonts so I just tell Firefox to force my fonts and enable bitmap fonts wholesale. Instructions are available here. Incidentally, MonteCarlo (the font mentioned on that page) is pretty good, but Neep is better. Try both and I think you'll agree.

For gVim purposes you may want one of these two lines:

set guifont=Neep\ Medium\ Semi-Condensed\ 10
set guifont=Neep\ 10

Semi-Condensed lets you cram in a ton more columns and the regular characters look great, but bold characters will look funny and run together. C'est la vie.

Wednesday, June 10, 2009

Saving Your "Sessions" with GNU Screen

I've been trying to figure out the perfect way to set up my development environment for a while, and GNU Screen has been a huge help. You can find out about GNU Screen very easily on the web (and of course the man page), but when I wanted to figure out how to "save" my screen sessions I didn't find much help. It turns out that there's not really a practical way to do this for various reasons related to the shear complexity of terminals. You might be able to save sessions that worked in 95% of cases but caused severe problems in another 5%, so no one's bothered to implelment the hacky feature.

On the other hand, what you can do is configure a bunch of screenrc files that hold the commands you want to start up your sessions with. Here is a screenrc I use to start up a screen session suitable for hacking on Conquer-on-Contact:

screen -t "bzr"
stuff "bzr diff | colordiff"
screen -t "misc"
screen -t "devsvr"
stuff "cd ~/google_appengine^M"
stuff "./ -p 8084 ~/shared/appengine/conquer-on-contact/root/"
screen -t "update"
stuff "cd ~/google_appengine^M"
stuff "./ update ~/shared/appengine/conquer-on-contact/root/"

#change the hardstatus settings to give an window list at the bottom of the
#screen, with the time and date and with the current window highlighted
hardstatus alwayslastline 
hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B} %d/%m %{W}%c %{g}]'
vbell on
defscrollback 1024
startup_message off
autodetach on
defutf8 on

Here the screen -t "xxx" commands open up a new tab with the specified names, while the stuff "some text here" commands actually put that text into the newly created tabs. One weird thing you might notice here are the ^Ms... You don't type those literally; you need to actually enter the control code. In vim you can do this by typing Control-V and then Control-M. If you add that to the end of the line a return will be sent and the command will be executed, otherwise it will just sit in the buffer and wait for you to hit enter (which is also pretty useful, actually...)

You can also use chdir command instead of this funny business with stuff and control codes, see an example. I think this method is more widely applicable, however.

Once you're done, save the file as screenrc and fire up the session with:

screen -S "SESSIONNAME" -c screenrc

If you're really lazy you could put that into a shell script so you only have to type something like ./ after every reboot, or set it to fire automatically in detached mode so your screen sessions are just waiting for you to connect.

Later I'll play around with screen-profiles. I really love the little statuslines, now I'm just trying to get the current session name in there somehow...

Saturday, June 6, 2009

Python-based HIIT workout timer

During my efforts to get back in shape I discovered HIIT, or High Intensity Interval Training. Of course, I thought! Just like the hill sprints we used to do during track and field back in school.

HIIT appeals to me over straight-up aerobic training because I enjoy pushing myself to the limit. (I was a sprinter, not a distance guy.) And hey, if the studies are to be believed, from a personal fitness perspective HIIT actually generates better results than aerobic training in less total time. Holy crap, what's not to love?

But enough of my ranting. The point of this post is that if you are a linux user and have a computer near your workout area, you might enjoy this python script to help you manage your HIIT sessions. The terminal output looks like so:

bthomson@ubuntu-wintendo:~$ python 
15 seconds to get ready!
Interval 1!
Work for 15!
Active rest for 30 seconds!
Get ready...
10 seconds!
Interval 2!
Work for 15!

Better yet, if you have espeak installed, the tts system will speak the messages so you can focus on your workout. If you use Rockbox you'll probably recognize the voice.

I suspect this is far easier to use than any of the commercial HIIT timer devices on the market, but of course the catch is that you'll need a computer in your workout area. If you're doing sprints outside it's not going to help at all.

Anyway, the script:

# See end of document for copyright notice.
# Important: eSpeak must be installed for tts capability.
# Adjust these parameters to suit your needs. All values are in seconds. Most
# important are 'work' and 'recovery', which are your work and recovery
# interval lengths.

startInterval = 1
leadin = 15 # minimum: 7 seconds
recovery = 30 # minimum: 12 seconds
work = 15 # minimum: 4 seconds


import os, sys, time
from threading import Thread

class spkThread(Thread):
  def __init__ (self):
    self._active = True
    self._speakText = None

  def run(self):
    while self._active:
      if self._speakText:
        os.system('espeak "%s"' % self._speakText)
        self._speakText = None

  def speak(self, text):
    self._speakText = text

  def terminate(self):
    self._active = False

def printNSpeak(text):
  print text

def countdownFromThree():
  print "3..."
  print "2..."
  print "1..."

spkThread = spkThread()
  interval = startInterval - 1
  printNSpeak("%d seconds to get ready!" % leadin)
  time.sleep(leadin - 6)
  while 1:
    interval += 1
    printNSpeak("Interval %d!" % interval)
    print "Work for %d!" % work
    time.sleep(work - 3)
    printNSpeak("Active rest for %d seconds!" % recovery)
    time.sleep(recovery - 13)
    printNSpeak("Get ready...")
    printNSpeak("10 seconds!")

except KeyboardInterrupt:

# Copyright (c) 2009 Brandon Thomson,
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.  THE SOFTWARE IS PROVIDED

I doubt there's enough interest in this kind of thing to justify a real open source project, but for me it gets the job done in a very stylish way. Enjoy!

Edit: Apparently Android phones support Python and TTS... That would be the ideal place to run some code like this.

Tuesday, May 12, 2009

Standing Desks For Programming

I caught on to the standing desk craze a couple weeks ago and decided I just had to have one... but nothing on craigslist really caught my fancy. Sure, someone was selling the legendary Ikea Jerker for $200, but I admit it: I'm cheap! After looking at some drafting tables and custom shelving-type solutions, I decided to just use something I already had:

Ok, ok, so it's a bookcase. So what? It works great.

I use the lower shelf as a footrest (essential, by the way), so it might need to be reinforced, but for now it's just fine.

The standing pose is nice; it really makes me feel like I'm attacking a project, rather than wallowing in it. When I'm frustrated I can walk around the room rather than just sit and stew in my own mirth. Walking away is very natural and I often do it without consciously deciding to. Even better, the slight discomfort from standing makes me less likely to be distracted by the web.

The first few days were pretty painful on my feet and lower back, but by now I've pretty much adapted. I can even pace around a little bit while I'm reading an article (but don't try that with a TN panel display!)

An unexpected benefit: much more effective digestion.

Now I worry about getting a job where I don't have the option to stand...

Sunday, April 5, 2009

Graphically Visualizing Output from the Python Profiler

Back when I was a teenager just getting my feet wet with programming, I remember sometimes spending hours looking over my code and making incremental changes, trying to figure out why it was slow. Nobody had told me about profilers! These days my first instinct when something is slower than I expect is to try and find some way to profile it, because the bottleneck is probably not where I think it is.

Python's profiler is nice, but mentally grokking the output can be a little obtuse. Thanks to José Fonseca there's a simple and easy way to visualize the output. He's written a script that converts the output from various profilers (including Python's cProfile) into a dot graph. Here's the output from a recent run that shows a single call to deepcopy killing my app's performance:

Using Gprof2Dot on a basic python project is very simple, although you might run into some trouble if you want to use it on Google App Engine. I haven't tried it yet but there's a good response on stackoverflow that might help.

To learn more about Gprof2Dot, visit the Gprof2Dot project page.

Saturday, March 21, 2009

Memcache-based circular buffer for Google App Engine

The problem: AppEngine's datastore is slow and uses a lot of cpu.

One solution: store non mission-critical data in memcache only. Your app might not have any data that you can be ok with losing, but there are a lot of good examples for games: chat messages, console messages, status updates... it's ok if these things get lost for some reason and we aren't interested in keeping them around for a long time anyway.

Here is some code to get you started with a circular buffer. Basically we have 10 memcache keys for data and one key that is used as an index (iter). The last ten items stored are available in order from most recent to least recent, and if a new item is added it replaces the oldest item while still being returned at the front of the list thanks to the sorting.

If you don't have other things you don't mind being kicked out of cache you can modify this code to just keep increasing the iter value instead of performing mod 10 such that old keys are not overwritten. This should save you some cpu time. Eventually the old items will be kicked out of cache automatically.

Anyway, here is the code:

from google.appengine.api import memcache

def addToBuffer(bufferData):
  loc = memcache.incr("myBufferIter")
  if loc == None:
    memcache.set("theIter", "0")
    loc = 0
  memcache.set(key="myBufferData%d" % (loc % 10), value=text)

def getBufferData():
  keys = []
    iter = int(memcache.get(key="myBufferIter"))
    for i in range (0, 10):
      keys.append('%d' % i)
    results = memcache.get_multi(keys, "myBufferData")

    # You can omit this part if you don't care about the order of the data
    sortedResults = []
    for i in range (iter%10, 0, -1) + range(9, iter%10, -1):
    return sortedResults
    return None

Sunday, March 1, 2009

Does music increase or decrease productivity?

A Google query for "music work decreases productivity" mostly turns up hits that suggest listening to music while you work will increase your productivity.

My experience is mostly the opposite. I suppose if I were doing something I found very boring that music would help keep me entertained and perhaps more productive, but most of the work I do requires 100% of my focus. Distractions of any kind—yes, even music—pollute my ability to concentrate and slow me down. In noisy environments, earplugs are far better at reducing the impact of the distractions than covering them with music is.

On the other hand, working with music is clearly more pleasant. I like to sing along when songs have vocals and it's enjoyable for me. But I don't confuse enjoyment with productivity.