Sunday, October 25, 2009

Bedtime stories and lullabies

When we are kids, our mothers tell us bedtime stories and sing us lullabies before we go to sleep. But what when we are grown up and living alone? I love music a lot and I listen to music almost all the time; at work, at home, when I am traveling and even when I am about to fall asleep. When I go to bed, I play music on my computer. I use Debian GNU/Linux at home and I play music using Rhythmbox with 'Repeat' enabled for the album or artist I am listening to on the day.

When I am falling asleep with the music playing on my computer, gradually the music becomes disturbing and prevents me from going into deep sleep. Decreasing the volume helps but it helps only for the next ten minutes or so and then it feels too loud again. So, after sometime I need to stop playing the music and go to sleep peacefully without any music to disturb me. Decreasing the volume every five minutes or so and finally taking care to stop the player before sleeping is not a nice way to fall asleep.

I thought of writing a shell script to automate this so that I do not have to worry about the volume and the player. It occurred to me that I can make the computer tell me bedtime stories too. I needed two things to do it: a way to pause Rhythmbox from the shell and a way to decrease the volume as the night progresses. I found rhythmbox-client command to control Rhythmbox from the shell and aumix command to adjust the volume of the audio mixer. I used espeak command to read text files aloud.

So far, so good. How about asking the computer to wake me up in the morning? If the script can start the music player in the morning and increase the volume gradually, I'll have a morning alarm too.

Here is the script:
#!/bin/bash

# Bedtime story and lullaby cum morning alarm script
# Author: Susam Pal

# Falling asleep (0.5 hour)
espeak -a 100 -s 150 -f story.txt &
for volume in {80..51}
do
aumix -v $volume
echo Reduced volume to $volume.
sleep 1m
done

# Deep sleep (5 hours)
pkill espeak
rhythmbox-client --pause
echo Sleeping at `date`
sleep 5h

# Morning alarm (1 hour)
echo Waking up at `date`
rhythmbox-client --play
for volume in {41..100}
do
aumix -v $volume
echo Increased volume to $volume.
sleep 1m
done

Sunday, September 06, 2009

Formatting my new Seagate External Portal Drive using FAT32 file system

I bought a Seagate External Portable Drive a few days back as I have a lot of movie and music files now that require hundreds of gigabytes of disk space. When I connected it to my laptop running Debian GNU/Linux, I found that Debian was unable to mount it automatically as the file system was NTFS. I wanted to convert it to FAT32 so that I could use it across various operating systems without any hassles. I thought that I would do it next day at office where I had a laptop running Windows XP. But next day, at office, I found that Windows XP wouldn't give me an option to format the 320 GB portable drive as FAT32. The only option that appeared was NTFS. On searching the web, it seemed that I needed to use some third party tool to format a volume greater than 32 GB using the FAT32 file system on Windows. There are plenty of them available on the web. However, I decided to avoid the hassle of finding an appropriate tool (I was later told by a friend that 'fat32format' is a good tool for Windows). So, I decided to do the formatting using my laptop that was running Linux at home. I am documenting the commands I had to run to do it in this blog post.

I ran the fdisk command to list the partition table and selected the relevant lines by piping the output through grep.
nifty:/home/susam# fdisk -l | grep ^Disk
Disk /dev/sda: 160.0 GB, 160041885696 bytes
Disk identifier: 0x0002ade8
Disk /dev/sdb: 320.1 GB, 320072933376 bytes
Disk identifier: 0x00225e47
nifty:/home/susam# fdisk -l | grep ^/dev/sdb
/dev/sdb1 1 38913 312568641 7 HPFS/NTFS
So, the output showed that /dev/sdb represents the 320 GB portable drive, /dev/sdb1 represents the file system on it and the file system is NTFS. So, I changed the file system to FAT32 using the fdisk command again.
nifty:/home/susam# fdisk /dev/sdb

The number of cylinders for this disk is set to 38913.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): b
Changed system type of partition 1 to b (W95 FAT32)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: If you have created or modified any DOS 6.x
partitions, please see the fdisk manual page for additional
information.
Syncing disks.
nifty:/home/susam#
Note that this time fdisk was invoked with the device file representing the portable drive as an argument. The t command at the fdisk prompt is used to change a partition's system ID. The system ID was entered as b which is the code for FAT32 file system. The w command is used to write the partition table to disk and exit.

Next, the disk was formatted with the FAT32 file system.
nifty:/home/susam# mkdosfs -v -F 32 -n Susam320GB /dev/sdb1
mkdosfs 3.0.5 (27 Jul 2009)
/dev/sdb1 has 255 heads and 63 sectors per track,
logical sector size is 512,
using 0xf8 media descriptor, with 625137282 sectors;
file system has 2 32-bit FATs and 32 sectors per cluster.
FAT size is 152547 sectors, and provides 19526004 clusters.
Volume ID is aa58f967, volume label Susam320GB.
nifty:/home/susam#
The string following the -n option is the volume label for the volume. The volume label can of course be changed later using the mlabel command. This command comes with the 'mtools' package. The command can be used as shown below.
nifty:/home/susam# mlabel -i /dev/sdb1 -s ::
Volume label is Susam320GB
nifty:/home/susam# mlabel -i /dev/sdb1 ::Foo
The first command prints the existing volume label. The second one, changes the volume label to 'Foo'.

Friday, July 03, 2009

A kid who could read my mind

Last night, I was with a group of people I didn't know. In the group, there was a kid who claimed that he could read the minds of others. I did not believe it. So I approached him. He covered both my ears with his palms and asked me to think of something. I thought of the number '5' and he immediately said aloud, "5". I was shocked. How could the kid know that I thought of that number? To make sure he did not simply get lucky, I thought of another number, '7' and he immediately declared that I thought of '7'. I thought of a few more numbers and he correctly announced all the numbers I thought of. I was completely befuddled. The people around were equally perplexed. They felt that both of us were working together to hoodwink others. An old man in the group who seemed intrigued by all this approached us and asked us to perform more experiments in a structured manner. He proposed that the numbers be generated using a pseudorandom number generator of a scientific calculator and written down on a paper. He asked me to read the paper and then think of the numbers written there and see whether the kid could then read my mind.

While he was preparing a list of random numbers, I was trying to guess how the kid was reading my mind. He couldn't be correct every time just by luck. He seemed to have direct access to my mind. However, that would be a miracle. I kept on thinking, coming up with various hypotheses and discarding them. By sheer luck, I thought of a model involving one random number generator (RNG) connected to two displays. It provided me some clues. In the model, the output of the RNG appeared immediately on the first display, but only after a delay of 1 second on the second display. Now, it would seem that the first display was predicting the output of the second display. I started thinking more in that direction. What could possibly be the source of the thoughts that was accessible to both?

After a little thought, it occurred to me that it could be my own mind. Probably the kid and the people I could see were simply creations of my mind. Perhaps, I was asleep and dreaming. I wanted to test the hypothesis. If it was only a dream and my mind was creating all the experience, I should have been able to read the minds of others. I had no clue how the experience of reading others' minds would feel like but I went ahead to test the hypothesis. I asked a person to help me. He agreed. I covered his ears with my palms and asked him to think of a number and suddenly the number '17' echoed in my mind. I was thrilled. I asked him whether '17' was the number he thought of. He confirmed that it was indeed the number he had thought of.

Unfortunately, I could not stay in this state of lucid dream for long. I woke soon after I tested my hypothesis a few times. In case, you find lucid dreaming intriguing you should watch Waking Life, a very thought provoking movie that shows the experience of a man trapped in a persistent lucid dream state.

Saturday, May 16, 2009

ADAC and HE puzzles from GEB

I have been reading Gödel, Escher, Bach: An Eternal Golden Braid by Douglas R. Hofstadter since the last Monday. The book alternates between chapters and dialogues. In the words of the author:
The long and the short of it is that I eventually decided - but this took many months - that the optimal structure would be a strict alternation between chapters and dialogues. Once that was clear, then I had the joyous task of trying to pinpoint the most crucial ideas that I wanted to get across to my readers and then somehow embodying them in both the form and the content of fanciful, often punning dialogues between Achilles and the Tortoise (plus a few new friends).
After the second chapter (Chapter II: Meaning and Form in Mathematics) there is a dialogue between Achilles and the Tortoise on telephone. The title of the dialogue is Sonata for Unaccompanied Achilles. Achilles is the only speaker, since it is a transcript of one end of a telephone call. The Tortoise is at the far end of the call. The sentences spoken by the Tortoise at the other end are not present. This makes it very interesting as we keep guessing what the Tortoise might have spoken.

It starts in this manner.
Achilles: Hello, this is Achilles.
Achilles: Oh, hello, Mr. T. How are you?
Achilles: A torticollis? Oh, I'm sorry to hear it. Do you have any idea what caused it?
As the dialogue proceeds, they share a few puzzles. Here is the first one from the Tortoise.
Achilles: A word with the letters 'A', 'D', 'A', 'C' consecutively inside it ... Hmm ... What about "abracadabra"?
Achilles: True, "ADAC" occurs backwards, not forwards, in that word.
Achilles: Hours and hours? It sounds like I'm in for a long puzzle, then. Where did you hear this infernal riddle?
Here is the second one from Achilles.
Achilles: Say, I once heard a word puzzle a little bit like this one. Do you want to hear it? Or would it just drive you further into distraction?
Achilles: I agree - can't do any harm. here it is: What's a word that begins with the letters "HE" and also ends with "HE"?
Achilles: Very ingenious - but that's almost cheating. It's certainly not what I meant!
Achilles: Of course you're right - it fulfills the conditions, but it's a sort of "degenerate" solution. There's another solution which I had in mind.
Achilles: That's exactly it! How did you come up with it so fast?
Achilles: So here's a case where having a headache actually might have helped you, rather than hindering you. Excellent! But I'm still in the dark on your "ADAC" puzzle.
SPOILER ALERT: If you want to think on these puzzles, don't read further as there are spoilers below.

It didn't take much time for me to solve the puzzle because I cheated with the word list file available in Debian 5.0.

Here is the output of my cheating.
susam@nifty:~$ grep adac /usr/share/dict/words
headache
headache's
headaches
susam@nifty:~$ grep ^he.*he$ /usr/share/dict/words
headache
heartache
So, the answers to both puzzles seem to be 'HEADACHE'. Read the last sentence, in the dialogue I have shown above, again. It makes sense now as Achilles says that having a headache might have helped the Tortoise.

Later in the dialogue the Tortoise offers 'figure' and 'ground' as hints to the 'ADAC' puzzle.
Achilles: Well, normally I don't like hints, but all right. What's your hint?
Achilles: I don't know what you mean by "figure" and "ground" in this case.
Achilles: Certainly I know Mosaic II! I know ALL of Escher's works. After all, he's my favorite artist. In any case, I've got a print of Mosaic II hanging on my wall, in plain view from here.
Achilles: Yes, I see all the black animals.
Achilles: Yes, I also see how their "negative" space - what's left out - defines the white animals.
Achilles: So THAT's what you mean by "figure" and "ground". But what does that have to do with the "ADAC" puzzle?
Achilles: Oh, this is too tricky to me. I think I'M starting to get a headache.
The famous painting discussed in the dialogue can be found here: http://www.worldofescher.com/gallery/A30L.html. It can be seen how the black animals form the figure or the positive space and how the background or ground or negative space beautifully fits all the white animals.

I was unable to use this hint to solve the puzzle. But after cheating and finding the answer I could make sense of the hint and understand how 'figure' and 'ground' lead to 'HEADACHE'. The first puzzle has 'ADAC' in the question. Let us consider 'ADAC' as the figure or the positive space. Now, if we remove 'ADAC' from 'HEADACHE', we are left with the ground or negative space, which consists of 'HE' in the beginning of the word and 'HE' in the end of the word. The figure is used to make the question in the first puzzle. The ground is used to make the question in the second puzzle.

An interesting question is: What was the first answer from the Tortoise that Achilles found very ingenious but degenerate. I believe, it is 'HE' as this word begins with 'HE' and also ends with 'HE'.

The funny thing is that both of them asked two puzzles to each other without knowing that the answers to them were same. This is exactly what happened when a colleague of mine and I challenged each other with combinatorics puzzles. I wrote a blog post on this here: Combinatorial coincidence.

Saturday, April 11, 2009

Butter and Mashed Banana

Today, I went to Ranga Shankara to watch a play called Butter and Mashed Banana. Here is the description of the play from the Ranga Shankara website.
Butter and Mashed Banana is a hilariously funny take on freedom of speech in our country. It goes from three actors in real life to three characters on stage caught up in a frenzy of activity ranging from International best sellers that are made into Oscar winning films to instant stardom and conflicting ideologies. The story revolves around a boy who goes on to become a world-famous writer, but faces rabid opposition to anything he tries to say or do back home in India. Interspersed with music and movement the play tangentially takes digs at ideology and censorship, among other things. Although its intent is to entertain, the play is a result of dissatisfaction over how poorly the freedom of speech is protected. Like it says in the play, there is always someone in this country waiting to tell you… "Don't you dare say that!" Created as a satirical farce, Butter and Mashed Banana uses dance, movement and live music to heighten the funny elements.
I have spent my entire childhood at a small town called Rajgangpur where my father works in OCL India Limited. There used to be plays, cultural performances and pravachans in the company auditorium where I used to go and enjoy. Also, cultural activities in my school, Dalmia Vidya Mandir, almost always included an English play, a Hindi play, a solo song, a group song and a classical dance from each house (We were divided into three houses: Vivek, Vikas and Vinay). Having lived in such an environment, I still retain an interest in plays as well as classical songs and dances. When, I came to Bangalore, I knew about Ranga Shankara and many times I have looked up the various plays performed there on the web. However, I was always too lazy to make an effort of booking a ticket for myself and going there alone.

Luckily, today, Wesley asked me to join him and other friends to the play. So, we arrived at Ranga Shankara at 7:20 PM, 10 minutes before the play would start and found some seats for the five of us. We realized that we should reach there at least half an hour before the scheduled time to get better seats, the ones that are aligned parallely to the stage and provide the best view. The play started almost on time and I enjoyed most of it. There were three actors and one guitarist. I found it very different from the plays I have watched before. Each actor was not playing just one character. The actors danced and moved around the stage assuming different characters in different situations. When the play began, one actor played the role of a father and the other played the role of a mother. The third actor played the role of a new born baby. He was also the narrator. Half an hour later, when the new born baby had grown up to be an Oscar winner, the first two actors played the role of journalists asking questions to the third actor after the Oscar ceremony was over. Five minutes later, one of the first two actors was making threatening calls to the Oscar winner.

The play made a lot of sense to me. These days, when I am with Wesley, most of the funny things I say are satirical reflections on how some people try to define what our culture is and how these people harass those who do not agree with or follow their definition. The play was trying to express a similar idea and so, I enjoyed it a lot.

From the Ranga Shankara calendar it seems that they have a play on each day of the week except Monday. The one on the weekends are mostly in English. So, I will try to go there more often, probably every weekend if I get some company from friends, colleagues or anyone else.

Sunday, March 29, 2009

Apache Nutch 1.0 Released

Today, we received an announcement from the Nutch committer, Sami Siren that Apache Nutch 1.0 has been released. An extract from the announcement:
Apache Nutch, a subproject of Apache Lucene, is open source web-search software. It builds on Lucene Java, adding web-specifics, such as a crawler, a link-graph database, parsers for HTML and other document formats.

Apache Nutch 1.0 contains a number of bug fixes and improvements such as Solr Integration, new indexing framework and new scoring framework just to mention a few. Details can be found in the changes file:

http://svn.apache.org/repos/asf/lucene/nutch/tags/release-1.0/CHANGES.txt

Apache Nutch is available for download from the following download page:
http://www.apache.org/dyn/closer.cgi/lucene/nutch/nutch-1.0.tar.gz

I have been waiting for this release for a long time as I made some contributions to this project. These contributions were also my first contributions to an open source project. In 2007, when I was playing with the search engine in Infosys (my previous employer), I found a few things that could be fixed and improved. I submitted these fixes and enhancements to the Nutch community and they were committed to the subversion repository. Let me list my contributions from the CHANGES.txt file.
62. NUTCH-559 - NTLM, Basic and Digest Authentication schemes for web/proxy
server. (Susam Pal via dogacan)

77. NUTCH-44 - Too many search results, limits max results returned from a
single search. (Emilijan Mirceski and Susam Pal via kubes)

80. NUTCH-612 - URL filtering was disabled in Generator when invoked
from Crawl (Susam Pal via ab)

81. NUTCH-601 - Recrawling on existing crawl directory (Susam Pal via ab)
NUTCH-559 was my major contribution. I did this when I found that Nutch was unable to authenticate itself to the intranet sites which were protected with NTLM authentication scheme. I modified the module that deals with the HTTP protocol so that it could authenticate itself with configured credentials when challenged with authentication. While developing this, I also developed support for Basic and Digest authentication schemes. More details on this can be found in NUTCH-559 (JIRA) and the Nutch wiki entry on HTTP authentication schemes.

NUTCH-44 and NUTCH-612 were bug fixes. NUTCH-601 involved the removal of a minor irritant. In the days of Nutch 0.9, the crawler complained if a directory with the name 'crawl' already existed in the current directory. As a result, before beginning a re-crawl using the bin/nutch crawl command, we had to move the existing crawl directory to another location. After a discussion in the community, we agreed that it was better to avoid shuffling the crawl directories by allowing re-crawls on the same directory. The change was made and committed.

Nutch users' mailing list has often received mails from users who wanted to know how they can enable support for authentication schemes in Nutch 0.9 by applying the patch in NUTCH-559. Patching Nutch 0.9 was a little cumbersome as the patch was generated against the trunk. With this release, the users can simply download Nutch 1.0 and configure the authentication schemes.

Tuesday, February 03, 2009

AUTH CRAM-MD5

I spent some time last weekend trying to understand the various SMTP authentication mechanisms. I found CRAM-MD5 authentication mechanism particularly interesting as it is a challenge-response authentication and involves HMAC-MD5. Let me first show a complete session with an ESMTP server before coming to CRAM-MD5. In the following session, I have selected the PLAIN authentication mechanism. Whatever I have entered appears in italic font.
susam@nifty:~$ telnet mail.susam.in 25
Trying 64.62.254.117...
Connected to sdclinux2.rdsindia.com.
Escape character is '^]'.
220 sdclinux2.rdsindia.com ESMTP
EHLO
250-sdclinux2.rdsindia.com
250-PIPELINING
250-8BITMIME
250-SIZE 11534336
250 AUTH LOGIN PLAIN CRAM-MD5
AUTH PLAIN AGZvb0BzdXNhbS5pbgBkcm93c3NhcA==
235 ok, go ahead (#2.0.0)
MAIL FROM:<foo@susam.in>
250 ok
RCPT TO:<example.recipient@gmail.com>
250 ok
DATA
354 go ahead
Date: Mon, 2 Feb 2009 10:28:00 +0530
From: Foo <foo@susam.in>
To: Example Recepient <example.recipient@gmail.com>
Subject: Test Mail

This is a test mail.
.

250 ok 1233593978 qp 15929
QUIT
221 sdclinux2.rdsindia.com
Connection closed by foreign host.
susam@nifty:~$
In the AUTH PLAIN line I have sent the base64 encoding of the string \x00foo@susam.in\x00drowssap. \x00 indicates a null character, foo@susam.in is the sender's user name and drowssap is the sender's password. Note that the user name here is in email address format. For most servers, the user name would only be 'foo' instead. If an eavesdropper intercepts this network traffic, he can easily find the user's password by simply decoding the base64 response sent by the client. This is also susceptible to replay attacks as the eavesdropper can use the AUTH PLAIN line containing the base64 encoded credentials, in future, to log into the server.

Now, let me quickly show only the relevant lines for an authentication using the LOGIN mechanism.
250 AUTH LOGIN PLAIN CRAM-MD5
AUTH LOGIN
334 VXNlcm5hbWU6
Zm9vQHN1c2FtLmlu
334 UGFzc3dvcmQ6
ZHJvd3NzYXA=
235 ok, go ahead (#2.0.0)
What is happening here becomes clear by decoding the base64 encoded messages. The following statements in Python 2.5.2 decode these messages.
>>> 'VXNlcm5hbWU6'.decode('base64')
'Username:'
>>> 'Zm9vQHN1c2FtLmlu'.decode('base64')
'foo@susam.in'
>>> 'UGFzc3dvcmQ6'.decode('base64')
'Password:'
>>> 'ZHJvd3NzYXA='.decode('base64')
'drowssap'
LOGIN authentication mechanism is susceptible to the same problems that PLAIN authentication mechanism is susceptible to. Now, let me describe the CRAM-MD5 authentication mechanism. When the client selects the CRAM-MD5 authentication mechanism, the server sends a base64 encoded challenge as shown below.
250 AUTH LOGIN PLAIN CRAM-MD5
AUTH CRAM-MD5
334 PDc0NTYuMTIzMzU5ODUzM0BzZGNsaW51eDIucmRzaW5kaWEuY29tPg==
An HMAC is calculated for this challenge with the password as the key and MD5 function. A string is formed by concatenating the user name, a space and the hex representation of the HMAC. The base64 encoding of this string is sent as the response by the client. The following statements I tried in Python 2.5.2 show how a response can be formed for the above challenge.
>>> 'PDc0NTYuMTIzMzU5ODUzM0BzZGNsaW51eDIucmRzaW5kaWEuY29tPg=='.decode('base64')
'<7456.1233598533@sdclinux2.rdsindia.com>'
>>> import hmac, hashlib
>>> hmac.new('drowssap', '<7456.1233598533@sdclinux2.rdsindia.com>', hashlib.md5).hexdigest()
'667e9fa4470dff3da9d621fe40676722'
>>> 'foo@susam.in 667e9fa4470dff3da9d621fe40676722'.encode('base64')
'Zm9vQHN1c2FtLmluIDY2N2U5ZmE0NDcwZGZmM2RhOWQ2MjFmZTQwNjc2NzIy\n'
Of course, this can be written as a small function:
import hmac, hashlib
def cram_md5_response(username, password, base64challenge):
return (username + ' ' +
hmac.new(password,
base64challenge.decode('base64'),
hashlib.md5).hexdigest()).encode('base64')
The following snippet shows the ESMTP server accepting the client-response.
250 AUTH LOGIN PLAIN CRAM-MD5
AUTH CRAM-MD5
334 PDc0NTYuMTIzMzU5ODUzM0BzZGNsaW51eDIucmRzaW5kaWEuY29tPg==
Zm9vQHN1c2FtLmluIDY2N2U5ZmE0NDcwZGZmM2RhOWQ2MjFmZTQwNjc2NzIy
235 ok, go ahead (#2.0.0)
CRAM-MD5 authentication mechanism is relatively more secure than the other two mechanisms because the password can not be retrieved by decoding the base64 encoded client-response. The password is used as the key to calculate the HMAC but the password is not present anywhere in the response. It prevents replay attacks too because the server sends an unpredictable challenge for every authentication. The client-response computed for a certain challenge is invalid for further authentications which will involve different unpredictable challenges.

References:
  1. RFC 4954 : SMTP Service Extension for Authentication
  2. RFC 4616 : The PLAIN Simple Authentication and Security Layer (SASL) Mechanism
  3. RFC 2195 : IMAP/POP AUTHorize Extension for Simple Challenge/Response
  4. RFC 2104 : HMAC: Keyed-Hashing for Message Authentication