Kurt Roeckx's journal

I recently bought myself a Canon EOS 500D also known as EOS Rebel T1i, and EOS Kiss Digital X3. I also got myself a Canon EF 50mm f/1.4 USM lens.

One of the new things about the 500D, compared to my previous 400D, is that it has live view. So I wanted to see if that's useful or not. I'm not that happy with it. If you want to let it auto focus normally you would half press the release button but instead you need to press the AE lock (*) button, half pressing the release button doesn't have any effect. The live view auto focus is also very slow, takes about 5 seconds on average. It's also has a habit of not focusing it all, or going in the wrong direction to find the focus. I sometimes suddenly feel the focus ring vibrate on the 50 f/1.4 USM while it normally never moves. There is also some weird clicking noise on my zoom lens, it appears to keep trying to send it beyond the smallest focus range while the object is meters away. There is also a "quick" focus mode, so that if you press the AE lock button the mirror goes down, temporary turning live view off, doing the focus the normal way, and then the mirror goes up again. In that mode the auto focus takes about 2 seconds instead. The only thing live view seems useful for is that you can zoom in to see if something is in focus.

I'm also seeing some weird behavior when using the flash. When using the 50mm f/1.4 it really seems to prefer using f/2.8, sometimes f/1.4 or f/4.0, and you can't select anything else in the auto, CA or P mode. Sometimes the depth of field is just too small, so I tried to set it to Av mode and manually select the one I want. What I didn't expect in that case is that it's calculating the shutter time like it won't be using the flash, and you suddenly get shutter times of 5 seconds. Other modes have similar problems, only setting everything manual seems to work properly.

Like Machtelt Garrels points out, the source code for the software used on the voting computers for this year is not yet available. It only is made available to the general public the day of the election after the voting has closed. The government has a FAQ about it here (available in Dutch, French and German).

As it says, software from previous years is available, but you need to be able to find it. So here are the links: 2003 2004 2007

There are 2 pieces of software that can be used: Jites or Digivote. They're not compatible with each other, so it depends on the municipality you have to vote in which one is being used.

Here you can find a study of Belgian voting system and a comparison with other countries. It's available in Dutch, French and English.

Update: The sources is available before the vote, just not to the general public. It's available to the politcal parties and the parlement can appoint a specialist to inspect the source code.

Thanks to Thomas Vander Stichele I've found out that when ripping an audio CD you get a different file depending on the CD drive you use because they don't all start reading at the same position and that for most drives this is a constant offset. So if you know what offset your drive has, you can compare rips done with a different drive.

I was under the impression that this offset was always a multiple of the sector size, because audio CDs do not have a reliable way to detect the position. But it seems this offset can in the middle of a sector too.

It looks like they agreed on some standard to compare the drive offsets. You can find a list of drives here and here. Cdparanoia has an option you can use to correct for that offset.

But I started wondering where this error comes from. There is no real reason why all CD drives shouldn't be able to return the same thing. There is very little real info about it out there, and I basically get the impression that someone just decided to use something as base value. It isn't really a problem, since this is about very small time.

To fully understand this, I think it's important to understand the basic layout of an audio CD. A sector (frame) of a CD contains 2352 bytes (588 samples) main data and 96 byte of sub-channel data. You have 75 such sectors per second, giving you a total of 44100 samples per second.

A sector is made up of 98 sub-frames. Each such sub-frame has 24 bytes (6 samples) of main data, 4 bytes of C1 error correction and 4 bytes of C2 error correction, and 1 "byte" of sub-channel data.

The first 2 sub-frames are used to mark the beginning of the sector. It contains a special pattern at the location of the sub-channel data. The first sub-frame contains S0 and the second S1. So you are left with 96 byte of sub-channel data. The highest bit of the sub-channel byte is used for the P sub-channel, bit 6 for the Q sub-channel, and so on until the W sub-channel. So the sub-channel has 96 bits (12 byte) per sector and each bit is in a different sub-frame.

If you go look at the offsets, you'll notice there are a lot of drives that have an offset that's a multiple of 6 samples, but there are also a lot that aren't a multiple of 6.

In the P and Q sub-channels you can see at what place you're reading. The P channel is a "pause" flag. At least 2 seconds before the track starts the P flag is turned on and the second sector of the track it's turned off again. All bits of P sub-channel should be the same for the whole sector, it can only change after an S0/S1.

In the Q channel you can have different information depending on what is in the first byte. The most useful one is "mode 1", which contain things like the track number, index in the track, and time indication for both the current track and the whole disc, and a CRC. Mode 1 should be in at least 9 out of 10 sectors. So you known perfectly at which location you are. There is just 1 small problem, the sub-channels do not have any error correction, but does have a CRC. And there are lots of sectors with the information in it.

So I wondered if the drives just didn't look at S0/S1 and that the sub channel data was shifted too. So I changed cdparanoia to log the sub-channel data and dumped it. I've tried this with a total of 6 drives, and also look at what the drive offset is. This are the drives I've used and the offsets they have:

Drive	offset
SONY DVD RW DRU-700A VY03	+12
PLEXTOR CD-R PX-W4012A 1.00	+98
PLEXTOR CD-R PX-R412C 1.06	+355
TSSTcorp CD/DVDW TS-L532A TI5a1	+116
HL-DT-ST DVD-ROM GDR-H30N 1.00i	+6
ASUS CD-S520/A 1.7L	+1858

Some notes:

• They all start exactly at the sector boundary, sub-channel data is not shifted. So they all perfectly know where a sector starts.
• Except for the ASUS, they all start at the correct sector. The ASUS starts 2 sectors too fast, which is probably why it has that 1858 samples offset, which is more than 3 sectors.
• Both plextors add 588 to the offset when reading the main data + the sub-channel data. So they start reading 1 sector earlier if you request the sub-channel data.
• The PLEXTOR CD-R PX-R412C sometimes also has an offset of +354

I can perfectly understand that it might be off by 1 or more sector, like the ASUS. I could also understand that it might be off by 1 or more sub-frames, but then I wouldn't expect to get proper values for the sub-channel data. Having an offset not be a multiple of the sub-frame makes no sense to me. So I can only come to the conclusion that this are all firmware issues.

One of the discussion that always seems to come back is what to do with firmware. It's mostly about freedom versus usability. I think the main question is where do we want to put non-free firmware in our archive.

The opinion seems to range from that it should be in the non-free archive to that it should be in the main archive, with many opinions in between, and they all have good arguments.

We have 3 options that are related to it in our current vote that seem to come down to:

• Move all firmware that doesn't comply with the DFSG to non-free
• Change the DFSG to not require source for firmware so that it can be in main
• Assume that the blobs in the kernel are source so that it can be in main

I don't see the point of that second option, and really have to wonder if there is any firmware under a license that would comply with the DFSG except that the source isn't available.

The third option can make sense for some of the blobs in the kernel that are basically settings that are written to the device. You could write an editor for it, assuming there is documentation for it. But I doubt that they're all settings, and it just seems to ignore the problem.

I find Theodore Ts'o' suggestion about creating a new section for firmware the best idea so far, specially if we can agree that our official CD/DVD images will have that section on it.

It has the advantage that all software in our main archive can comply with the DFSG, that you don't need to add the non-free section just to be able to use hardware that requires firmware and that you can just take a CD/DVD that works.

Today the voting period starts for this general resolution about firmware and the Lenny release.

As many have pointed about before this is very complex vote with many orthogonal issues.

The issues and some of the options:

• Where do we want to ship firmware that doesn't comply with the Debian Free Software Guidelines (DFSG)?
  • If it doesn't comply with the DFSG it should go to non-free
  • Change the DFSG so that we don't require source for firmware
  • Create a separate section for it.
• Is a DFSG violation a release critical bug or can we make a release with known DFSG violations?
  • DFSG violation are release critical
  • Sourceless firmware can be part of Lenny as long as we are legally allowed to do so.
  • Sourceless firmware can be part of Lenny
• It's not clear whether some blobs in the kernel comply with their license and the DFSG
  • Remove those parts that are not clear or don't comply with the DFSG
  • Assume that blobs in the kernel are source
• Does the release team have the right to decide if something is release critical or not?
  • Override them and say that DFSG violations are release critical
  • Explicitly give the release team the power to decide whether a bug is release critical or not

And then we're going to decide all of that in 1 single vote. I interpret the options we have as:

Choice 1:

• non-free firmware should go to non-free.
• It is a release critical bug
• remove unclear parts
• override the release team

Choice 2:

• We're not saying anything about where non-free firmware should go
• sourceless firmware can be part of Lenny as long as we are legally allowed to do so.
• We're not saying anything about the unclear parts
• Don't override the release team

Choice 3:

• We're not saying anything about where non-free firmware should go
• sourceless firmware can be part of Lenny
• We're not saying anything about the unclear parts
• Don't override the release team

Choice 4:

• We're not saying anything about where non-free firmware should go
• We're not saying anything about whether it's a release critical bug or not
• We're not saying anything about the unclear parts
• Release team decides if something is release critical

Choice 5:

• We're not saying anything about where non-free firmware should go
• We're not saying anything about whether it's a release critical bug or not
• Assume that blobs in the kernel are source
• Don't override the release team

Choice 6:

• Change the DFSG to not require source for firmware
• We're not saying anything about whether it's a release critical bug or not
• We're not saying anything about the unclear parts
• Don't override the release team

Choice 7:

• We're not saying anything about where non-free firmware should go
• We're not saying anything about whether it's a release critical bug or not
• We're not saying anything about the unclear parts
• Don't override the release team

So in my opinion, options 2, 3, 4, 5 and 6 all decide on 1 of the 4 questions, and only 2 of them about the same question.

I also don't see any bugs filed against the linux-2.6 package about any issues the kernel might have with firmware, so I have to wonder why we're voting about this.

My current vote is "2222221". I'm not sure it's the best options, but I currently don't see a better one.

It's been a long we've had snow around here this time of the year. Lately we've been lucky to get some snow somewhere in January. But yesterday it snowed a little and it was gone after an hour. Today it started snowing again and it actually looks like it might stay a few days. There is only like 2 cm so far.

Some pictures:

I got this nice message (and some others) in my log today:
smartd: Device: /dev/sdd, 2 Currently unreadable (pending) sectors

After running a smartctl -t long /dev/sdd smartctl now reports:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   089   089   000    Old_age   Always       -       8719
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       -       2
198 Offline_Uncorrectable   0x0010   200   200   000    Old_age   Offline      -       0

[...]
SMART Error Log Version: 1
ATA Error Count: 10 (device log contains only the most recent five errors)
[...]
Error 10 occurred at disk power-on lifetime: 8692 hours (362 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 78 e5 89 e0  Error: UNC 8 sectors at LBA = 0x0089e578 = 9037176

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 08 78 e5 89 2f 00  45d+06:36:27.880  READ DMA EXT
  25 00 08 70 e5 89 2f 00  45d+06:36:27.798  READ DMA EXT
  25 00 08 68 e5 89 2f 00  45d+06:36:27.308  READ DMA EXT
  25 00 08 e0 77 e5 35 00  45d+06:36:27.295  READ DMA EXT
  25 00 08 60 e5 89 2f 00  45d+06:36:26.653  READ DMA EXT

[...]
Error 9 occurred at disk power-on lifetime: 8692 hours (362 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 60 e5 89 e0  Error: UNC 8 sectors at LBA = 0x0089e560 = 9037152

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%      8715         797566305
# 2  Extended offline    Completed without error       00%      8198         -

Notice that 197 Current_Pending_Sector indicates that there are 2 sectors with a problem.

Note that the SMART error log shows LBA sectors 9037176 (0x89e578) and 9037152 (0x89e560) but it's limited to 0xffffff. The self test logs shows us the proper LBA error at 797566305 (0x2f89e561) which has a 0x2f in front, and also shows us that it's the second sector in the block of 8 we tried to read.

The kernel log also shows:

ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata4.00: (BMDMA2 stat 0xc0009)
ata4.00: cmd 25/00:08:60:e5:89/00:00:2f:00:00/e0 tag 0 cdb 0x0 data 4096 in
         res 51/40:00:61:e5:89/00:00:2f:00:00/f0 Emask 0x9 (media error)

So I've tried to reproduce the error with:

# dd if=/dev/sdd of=797566305 skip=797566305 bs=512 count=1 iflag=direct
1+0 records in
1+0 records out
512 bytes (512 B) copied, 1.27727 s, 0.4 kB/s

Notice that it can actually read that block, it just seems to take a while.

And the error log now contains:

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 01 61 e5 89 e0  Error: UNC 1 sectors at LBA = 0x0089e561 = 9037153

Because it's part of a RAID device I could just let the whole disk resync, but I was a little curious which files were in it. So I found this document that explains on how to find which files has the problem. But it only has examples covering ext2/ext3 on a partition and LVM and nothing about RAID device. So it looks like someone using RAID 0 will have a hard time finding the file that's having a problem.

So following the document we start with:

# fdisk -lu /dev/sdd

Disk /dev/sdd: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x000578f9

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1   *          63      385559      192748+  fd  Linux raid autodetect
/dev/sdd2       972864270   976768064     1951897+  fd  Linux raid autodetect
/dev/sdd3          385560   972864269   486239355   fd  Linux raid autodetect

Partition table entries are not in disk order

In /proc/mdstat we see:

md1 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]
      1458717696 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

So sdd3 is part of md1 and I have an ext3 filesytsem on that. Tune2fs -l |grep Block gives:

Block count:              364679424
Block size:               4096
Blocks per group:         32768

mdadm tells me:

# mdadm -QE /dev/sdd3
/dev/sdd3:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 747b071a:a70f723b:c3a9ed44:15845cbf
  Creation Time : Mon Nov 12 20:13:51 2007
     Raid Level : raid5
  Used Dev Size : 486239232 (463.71 GiB 497.91 GB)
     Array Size : 1458717696 (1391.14 GiB 1493.73 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 1

    Update Time : Mon Nov 10 10:57:28 2008
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 4b3147f4 - correct
         Events : 78

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8       51        3      active sync   /dev/sdd3

   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       19        1      active sync   /dev/sdb3
   2     2       8       35        2      active sync   /dev/sdc3
   3     3       8       51        3      active sync   /dev/sdd3

I've been told that 0.90 stores the superblock at the end of the device, so that makes it a little easier. And this seems to agree:

# cat /sys/block/md1/md/dev-sdd3/offset
0

We have a left-symmetric layout, so in raid5_compute_sector() we see

	stripe = chunk_number / data_disks;
	*dd_idx = chunk_number % data_disks;
[...]
	case ALGORITHM_LEFT_SYMMETRIC:
		*pd_idx = data_disks - stripe % raid_disks;
		*dd_idx = (*pd_idx + 1 + *dd_idx) % raid_disks;

So it looks like the chunks are located on the disk like:

stripe	sda	sdb	sdc	sdd
0	D 0	D 1	D 2	P 0-2
1	D 4	D 5	P 3-5	D 3
2	D 8	P 6-8	D 6	D 7
3	P 9-11	D 9	D 10	D 11
4	D 12	D 13	D 14	P 12-14

With each of those 64K.

The problem is at 797566305 and the partition started at 385560. So that's at sector 797180745 inside partition sdd3. Each sector is 512 bytes and a chunk is 65536 bytes which gets us 128 sectors per chunk. If we want to know which the stride the sector is on we do: 797180745/128=6227974.5703125

This means that the stripe is number 6227974 and that it's sector 0.5703125*128=73 on sdd3 within that chunk.

The parity is on disk: pd_idx = 3 - 6227974 % 4 = 1, being sdb.

To get the chunk we need to multiply with the number of data disks (3) and since it's the forth disk and the parity is on the second disk we need to add 1 and get: 6227974*3+1=18683923

So calculating that back to sectors we multiply it with the number of sectors per chunk again and add our offset in the chunk: 18683923*128+73=2391542217

To test that our math is any good we try:

# dd if=/dev/md1 of=797566305.2 skip=2391542217 bs=512 count=1 iflag=direct
1+0 records in
1+0 records out
512 bytes (512 B) copied, 5.99103 s, 0.1 kB/s

We see an error message in the kernel log, there are 2 new errors in the error log, so it seems that everything is calculated correctly. The files are also identical.

The file system is in blocks of 4096 bytes, so for the file system this is block 2391542217*512/4096=298942777.125

Then we use debugfs:

# debugfs
debugfs 1.41.2 (02-Oct-2008)
debugfs:  open /dev/md1
debugfs:  icheck 298942777
Block   Inode number
298942777       <block not found>

Which isn't making any sense to me, since I get that error about every 2 hour when a certain cronjob runs. 298942777 / 32768 (Blocks per group) = 9123.00955. So I wonder if ext3 is keeping some meta data there and debugfs doesn't tell that?

Anyway, at some point I was running a smartctl -t long /dev/sdd again because I couldn't reproduce the error anymore, the cronjob started, and I then found alot of errors in my log again and now it included:

raid5:md1: read error corrected (8 sectors at 797180744 on sdd3)

It's so much fun that disks always behaves the same trying to read the same sector.

Smartctl now reports:

  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   200   200   000    Old_age   Offline      -       0

I assume the kernel removed the pending sector by writing to it. The other pending error was already gone for some time, I assume new data got written to that one.

I've been looking at DomainKeys Identified Mail (DKIM) to see if it's useful. My main interest is reducing bounced mail that I didn't send and other people receiving less spam that appears to come from me. It seems there isn't a lot of documentation about it around, and I had to go and read some of the RFCs and drafts to understand things. So I hope this is useful for someone.

According to the wikipedia article it can be used to verify that the message comes from the domain that it claims to have come from. On the other hand the DKIM FAQ says it's all about the reputation of the organization the messages passed by. There is a project that creates a reputation database that then can be used in anti-spam software like spamassassin.

So I at first sight this didn't look useful at all to me. It basically seems like a way to verify the Received lines in the message header. And it seems that that is what most of the software is doing. But there also seems to be an Author Domain Signing Practices (ADSP). It allows you to say that if the message isn't signed it can be dropped because it's spam. There is nothing else in DKIM other than reputation on which you can base that something might be spam or not.

So I started looking in software that supported that. It seems that dkim-milter is the only software available in Debian that implements it. But it seems to be using the _asp._domainkey DNS entry from an old draft instead of the _adsp._domainkey as used in latest draft. Older drafts even used _ssp._domainkey.

So it seems to me that it's not useful yet and I'll wait until the ADSP RFC is actually published and there is software available that supports it.

Update:
It seems that dkim-milter from version 2.7.0 now uses _adsp._domainkey, but we currently only have 2.6.0 in Debian. But the wizard of sendmail.org still generates the _asp._domainkey. I've filed enhancement requests for spamassassin and dkimproxy / Mail::DKIM.