Corrupted Data

No, no, not mine. Someone elses. Someone I know was given a corrupt database file and told to retrieve as much data as possible from it. If she tried to use the database file with its usual application (called Exist), then it would refuse to read much of the data as the file was corrupt.

Thankfully the software is open source. She downloaded the code to her personal machine, used a debugger to find the lines that were detecting file corruption, changed them so that they silently failed (permitting the application to continue reading from the file), and suddenly she was able to read all the data!

Very proud of her for what she achieved. :) First, she saved some people a lot of work to regenerate that data. Second, she used sophisticated debuggers to locate and create a solution. Third, this method of handling corrupt files in Exist is now one of her contributions to other developers like herself.

Conducting Experiments

The following are the basic steps for conducting thesis experiments:

1. Write the code for your algorithm (about 1000 lines)
2. Try a few simple tests (about 200 tests)
3. Assemble a larger number of tests (about 20000 tests)
4. Run all those tests (about 2 weeks)
5. Generate average performance graphs/tables (a few days)
6. Write your analysis and make some conclusions (about a week)
7. Verify that your code is correct (about a day)

Hmm, something isn't right.

One of these steps is not like the other.
One of these steps is way out of sequence.
Can you tell me which should be done earlier?
Do you think I'll ever finish this thesis?

:P

Thesis Code

Most computer science or mathematics theses are written in a language called LaTeX. This isn't a regular "markup" language like HTML. It is actually a horrible programming language based on TeX that, when executed, normally is used to output DVI, PostScript, or PDF files. It can also output other bits of information which it will then reuse on subsequent runs to generate bibliographies, table of contents, etc.

Most Linux users are acquainted with the shell. This is normally seen as a text environment to execute programs. It is actually a horrible programming language called "sh" that, when executed, can do pretty much anything, including generate files, run LaTeX, or prompt the user.

Most people are not acquainted with sed. This is normally seen as a utility program to manipulate strings. It pretty much is just that. Technically, it too is a programming language. Noone ever uses it like that.

All the experiment results for my thesis are stored in SQLite databases, which provides me a simple but fast relational storage. SQLite has a command line interface and can produce simple text tables. I can then take those tables and run them through sed, which produces LaTeX tables. I can then copy-paste those generated tables into my thesis. The shell code to automate most of this looks like:

echo '\\begin{tabular}{|c|ccc|ccc|c|}'
echo '\\hline'
echo '& \\multicolumn{3}{c|}{\\textbf{SBDS}} & \\multicolumn{3}{c|}{{\\textbf{AWCS}}} & \\\\'
echo '& Nogoods & Isgoods & Values & Nogoods & Nogoods\\footnotemark[1] & Values&$f$\\\\'
echo '\\hline'
for w in "instance>0" "constraints=420" "constraints=460" "constraints=500"
do
sqlite3 -separator '&' results.sqlite "select '$w',round(avg(a.nogoodssent)), round(avg(a.isgoodssent)), round(avg(a.valuessent)), round(avg(b.nogoodssent)), round(avg(b.nogoodsmade)), round(avg(b.valuessent)), round(avg(a.violations=0)*100)||'\\\\' from results as a,results as b where a.algorithm='coop' and b.algorithm='awcs' and a.instance=b.instance and a.$w";
done | sed -e 's/\.0//g' -e 's/violations=0/Feasible/' -e 's/violations>0/Infeasible/' -e 's/instance>0/Average/' -e 's/constraints=\([^\&]*\)/$|\\constraints|=\1$/' -e 's/tightness=\([^\&]*\)/$|t|=\1$/' -e "s/'values'=\([^\&]*\)/$|\domain|=\1$/"
echo '\\hline'
echo '\\end{tabular}'

So: I've written a program (in shell) that dynamically executes a query (in SQLite) that is piped through a dynamically created program (in sed) that generates another program (in TeX) to be executed. The execution of this TeX can produce another program (in PostScript) that can be executed to display my thesis.

Be afraid.

Vancouver Sun Run

Start of spring, Sunday morning, lots of clapping... Must by the Vancouver Sun Run again! Near identical to last year. Sometime I should join in.

What is Missing from Baraka

I'm an unashamed fan of the "experimental documentary" Baraka. It's an unusual film, having no dialogue and no plot, but amazing footage. Many people approach it as a slideshow, and will tell me "hey, I saw Baraka; really pretty pictures". As I was falling asleep I had a thought and, in the dark, scribbled it down:

what is missing from Baraka?

I've heard Baraka described by another fan as "a movie that tries to capture everything" or "what God would see if he was looking at Creation". Lofty statements, I admit. But I put down the challenge: what does Beraka fail to show? What does it show?

space, creation, heat, cold, evolution, contemplation, introspection, time, worship, god, rituals, heritage, passing information between generations, the body used for expression, reenactment, our acknowledgement of nature's power, our belief in a higher power, industry, scale, individuality, ...

And that's all before the first strange man in white paint screams!

After that comes death, massacres, war, childhood, play, ... The list goes on. So far it seems to be missing things like love, passion, and nurturing which, for rating reasons, can never be shown in full detail anyway. :)

Watch it closely. What else is missing from Baraka? The challenge is there.

Wicked Witch

At work we have a little IRC (chat) channel that we use to quietly communicate amongst ouselves. Our working environment is a large, open office, so this provides a practical means to notify others of your actions without bothering too many people. As we are technology workers, it also serves as a common area for dumping text fragments (website addresses, pieces of code, etc) and is extremely useful.

Sometimes we want to capture a conversation or conclusion that we reach, and the IRC channel is perfect for that. We have our own (ro)bot that listens to our IRC messages, logging everything that we say. This bot is called the "WickedWitch", and we are her code monkeys. What is most fun about the WickedWitch, is what she thinks of us:

WickedWitch says: You are nothing but an elf-skinned mound of imp-bladdereddle-headed rat.

Yes, our bot throws random insults at us, or, more accurately, our work. It is programmed to insult our current project once every hour. Every day, we are given a fresh serving:

WickedWitch says: You are nothing but a bootless ooze of spongy spit. You are nothing but a lumpish puddle of goatish craptacular. You are nothing but a yeasty petrification of low-quality cold.

Now, some of us leave our computers on overnight, and get to watch the Witch overnight. And a curious thing was noticed. She would only insult us with squirrels when we weren't watching. This led to the "squirrel conspiracy".

Developer: a watched witch never insults squirrels

For example, at midnight: "You are nothing but a sausage-snorfling petrification of clapper-clawed squirrel" and four hours later: "You are nothing but a villainous petrification of full-gorged squirrel". So we did some quick statistical analysis; midnight and 4am are quite popular for squirrel insults, with 3 times more squirrel insults in those hours than the average! She would insult squirrels in the day time, but mostly on the weekends, when noone was looking. It was as if she was trying to trick us...

The Squirrel Conspiracy continues now, if only in our minds. Though we took a second to reprogram the Witch. She doesn't insult us anymore. Now she always insults squirrels.

WickedWitch says: Squirrels! You are nothing but a weedy half-mouthful of contemptible number-9.

Hawaii Day 2 (Honolulu)

My laptop is grinding away on experiments, so I'm taking some time out here and blogging about that holiday in Hawaii. I'll try to get it all written down before it's a distant memory...

Second day was fairly busy. We visited the Honolulu Zoo, primarily because it was within walking distance and it has been a long, long time since I went to a zoo. I thoroughly enjoyed it, even if the animals were often in hiding and parts of the zoo were under construction. There was a particular group of monkeys that were extremely loud and active - it was a hoot! (Pun partially intended). I found it fun just to watch them casually swing around their "play pen".

After the zoo was lunch, and then time at the beach. As always, the water at the beach is a little bit cooler than you expect (we spoil ourselves in Canada with really warm showers and baths), but the sun and pleasant air temperature helps warm you while swimming. Waikiki Beach has a large area protected by a wall that is great for just paddling around in; even if you can't swim you can still enjoy yourself as much of it is only waist deep.

Dinner was a bit of a mess. If you find a place that you like the look of, then I recommend lining up and waiting for it. We found a few places that were decent, and tried each of them in turn (looking for the shortest line). By the time we'd finished doing all that, we could gotten into our first pick. After dinner it was a quick walk back to the Royal Grove to sleep.

Note that, for the entire day, we didn't cover more than about 8 blocks. Finding a hotel near Waikiki Beach means that you won't have to walk very far. Many of the regular hotels are a fair walk from the beach; I'd recommend trying to get something cheap and as close to the beach as possible.

P.S. In those 8 blocks we found about 9 ABC stores... One of which was advertising the opening of a new store nearby...

P.P.S. Sore throat evolves into a small cough on this day.

Birthdays

Looks like I missed a few birthdays in the last couple of months. Sorry everyone! I'll probably miss a few more soon. Sorry about that too! I'm hoping to squeeze out this thesis in the next month. After that my life should be back to normal.

Chocolate Yoghurt

Coming back from lunch, a few of us were talking about my penchant for Mars Bars, and the possibility of having it blended into a milkshake. We then decided to drop into the food court and get a frozen yoghurt. Normally it's wild-berries or similar, but they also had chocolate pieces in the display, and so I (of course) asked for a chocolate frozen yoghurt. Turns out, this had never been attempted at that store. Has a nice chocolate-fruity after taste to it, presumably from remnants of the wild-berries still in the machine. Not bad. :)

Dreams

I've been having weird dreams in the last few weeks. Just a few are:

• Snorkelling in Hawaii.
• Sky-diving by walking backwards from a jet plane, while holding a conversation, and noting to myself "try to avoid those jet engines" while falling.
• Writing a message after the apocalypse, explaining to future civilisations what not to do next time, though I can't remember what it was that we shouldn't be doing.
• A workmate Dan and his evil twin Stan were giving me problems, and eventually they went into a room and came out as one merged person, proving that you don't have to be strictly good or evil (Dan is not the workmates real name, but that was his name in the dream, go figure).

Seems like there's a new dream each few days. Haven't found any pattern yet, except the occasional focus on "stress".

Too Tired

I've been finding myself tired a lot recently. Hence the utter failure to actually blog about the holiday I had some 2-3 weeks ago. Instead, I need to say that "Sobe Arush (Sugar Free)" energy drink actually isn't that bad. Unlike other "sugar free" drinks they didn't replace the sugar with aspartame either - it almost tastes like they removed all the sweetness. In reality, they used sucralose (aka Splenda), but I can't taste it.

Speaking of sweeteners, you might be interested in this fruit that, for about an hour after eating it, makes all other sour foods taste sweet.