Task: List the programming languages you consider important or interesting. Describe each of them in one sentence.

Posting this from the new N810 using its built-in keyboard. Typing those tags was painful, and as a result, more languages will be added to the table tomorrow — from the workstation.

Spent last night getting a crash course in using OCaml to do non-functional things (hash tables, file I/O, regular expressions) and the result is now up.

The JoCaml version does the file-partitioning trick used in the C++ implementation, with each finder workers being run inside a JoCaml channel; the channels share a single lock so they can update the hash table serially.

Interestingly, current implementation does not get a speed-up from the input file being cached (Ilmari’s wf.ml does). Will have to peruse his to see what’s slowing things down.

Lesson: not all techniques for processing a file line-wise are equally good!

Talked with a colleague about the slow single-threaded performance of my Wide Finder implementation, and we narrowed it down to two possibilities:

• Boost regular expression is not compiled?
• C++ strings have higher overhead than null-terminated c_str

First point can be ruled out: Boost compiles regular expressions when you assign them. Second point — well, reading in the file using std::getline turns out to consume the bulk of time.

I’ve reorganized the code a bit, using a multimap rather than a vector to rank the URLs by count, with no effect on speed. With two and four threads on a dual-core Intel notebook, the performance is at least on par with Ruby.

Alastair Rankine has a C++ implementation that is slightly faster, but uses Boost memory-mapped IO that I avoided for the same reason he put as caveat: that it will not scale to files that are too large. Which Tim’s log file might well be. Again, that is not significantly faster than the Ruby code.

Moral of the question: Perl and Ruby can be faster than C++! The C implementations out there are blindingly fast, but the way they do regular expression handling are really painful.

Will turn my (limited) spare time to doing a clean JoCaml implementation — it might not be faster but it definitely will look cleaner!

After turning in the C/C++ monster (cleanest C code I reckon it is possible to write, thus the total lack of memory-mapped I/O and other optimizations), I turned my attention to picking a better implementation language.

Requirements:

• Functional
• Good support for threading
• If possible, support for distributed computing

As it turns out, JoCaml fits the bill perfectly. It’s an extension of Ocaml, so it combines a rich library with a familiar syntax (not to me, but having used both Scheme and Haskell, how different can it be) — and a very nice process calculus!

Example: this is a concurrent stack that blocks if there is no input available

let new_stack () =
def state (s) & push (v) = state (v::s) & reply to push
or state (x::s) & pop () = state s & reply x to pop in
spawn state([]);
pop, push
;;

This defines a private state channel, and then export the pop and push synchronous channels (that to the user behave just like ordinary functions)

and this is how you use it:

let pop, push = new_stack ();;
spawn echo(pop());;
push(1);;

Note that the echo channel will block, since pop can’t return a value until the stack contains something! This value is then pushed into the stack and ‘1′ printed.

More of this at the JoCaml site. And, as it turns out, there already is a JoCaml implementation of the Wide Finder, by Ilmari Heikkinen. Will have to grok the finer details from him.

Several weeks ago, Tim Bray posted his Wide Finder project: take the Ruby script that parses an Apache log file and report the top 10 hits, and parallelize it in your language of choice.

It occurred to me a while back that this is a perfect job for a C multi-process program, taking advantage of Linux’s cheap copy-on-write fork, if not for the need to merge the result. So it would probably be easier to write it using the pthreads library instead. One would want to reduce inter-thread communication as much as possible, though.

I did not have time to touch the code until today, but now it’s done. Two key insights:

• The input file can be partitioned cleanly into multiple chunks of roughly identical sizes without communication, as long as each thread follows the same protocol. i.e. a common chunk size is used, and all but the first thread needs to check if the character before their starting offset is the end-of-line. If not, they need to skip until the end of line, yielding that line to the previous thread.
• When merging the hash maps, the first thread to acquire the lock does not need to insert its items one by one, but can just set the main hash map to point to its own

On a data set size of 200MB, with a single thread, performs slightly slower as Ruby by wall clock:

Ruby timing

2.99user 1.88system 0:05.26elapsed 92%CPU (0avgtext+0avgdata 0maxresident)k
152inputs+0outputs (3major+853minor)pagefaults 0swaps
3.04user 1.84system 0:05.33elapsed 91%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+856minor)pagefaults 0swaps
2.84user 2.01system 0:05.19elapsed 93%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+856minor)pagefaults 0swaps


C++ timing, one thread

2.70user 5.21system 0:07.95elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+655minor)pagefaults 0swaps
2.78user 5.14system 0:07.96elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+655minor)pagefaults 0swaps
2.72user 5.12system 0:07.90elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+654minor)pagefaults 0swaps


With two threads, though, it runs faster, almost at Ruby speed.

C++ timing, two threads

3.33user 5.40system 0:05.51elapsed 158%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+675minor)pagefaults 0swaps
3.40user 5.33system 0:05.86elapsed 148%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+672minor)pagefaults 0swaps
3.41user 5.36system 0:05.60elapsed 156%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+674minor)pagefaults 0swaps


So, as Tim observed, the problem is not entirely IO-bound. More testing is needed, but will probably need to be done on a machine with faster IO (and more CPUs). Like Tim’s new Niagara T2 testbed.

Will edit this post later — need to run now, and wanted to get this out as soon as possible. Code is available here… now I just need to find Tim’s email, since his blog is down. Argh!

Update: switched to the new std::tr1::unsorted_map. Performance seems identical with a single thread, but slightly higher (within margin of error) on two threads — which makes sense: a multithreaded run exercises the map more, because each thread’s map has to be merged into the final map.

“Be not afraid of greatness: some are born great, some achieve greatness and some have greatness thrust upon them”.
- (Shakespeare, W., Twelfth Night, Quote Act II, Scene V).

“Some men are born mediocre, some men achieve mediocrity, and some men have mediocrity thrust upon them. With Major Major it had been all three”.
- (Heller, J., Catch-22, Quote Page 94 (paperback)).

“Some languages are born lazy, some languages achieve laziness, and some languages have laziness thrust upon them”.
- (Salim, M., unpublished)

So you can probably guess what this is about. A stream library for Lua, that allows you to do things like running the Sieve of Eratosthenes without specifying an upper bound, lazily filtering the set of all positive integers as you go along.

Witness the use of the library:

require("stream")

function ints(n)
   return {
      head = function () return n end,
      tail = function () return ints(n+1) end }
end

function primes (sints)
   local curprime = false
   local tailthunk = false
   local fhead =
      function ()
	 if curprime then return curprime
	 else
	    curprime = sints.head()
	    tailthunk = sints.tail
	    return curprime
	 end
      end
   local ftail =
      function ()
	 if (not tailthunk) then
	    fhead()
	 end
	 return primes(
		 stream.filter(function(n) return not (n%curprime == 0) end,
			       tailthunk() ))
      end
   return {head=fhead, tail=ftail}
end

-- print the first 100 primes
stream.print(100, primes(ints(2)))

A stream is a table containing a head and a tail; both are thunks, so you call them with no arguments to obtain the value. Note the caching that avoids recomputation, similar caching is done in the functions provided by the stream library (like stream.filter)

Code available on hircus.org.

And some benchmarks:

	Lua	Haskell

The Haskell code was run using GHC’s ‘runhaskell’, which does not precompile the code, to be fair to Lua which is interpreted. Not bad when put up against a language that is lazy by design (and thus optimized for it)! Timing taken using the ‘time’ utility on an Intel Core 2 Duo 1.66 GHz laptop which was mostly idle.

(Thanks to Roshan for posing the task)

Given two binary trees, and a depth-first-search traversal ordering of the leaf nodes, produce two new trees with the same structures as the originals, but with the leaf nodes swapped. If there are more leaf nodes in one tree than the other, the remaining leaf nodes are copied unchanged.

So (swap-leaves ‘(1 . (2 . 3)) ‘(a . ((b . c) . d)))
=> ‘((a . (b . c)) ‘(1 . ((2 . 3) . d))

Waiving the requirement that the program be completely functional, my original thought was I could probably just modify my tree walker to do this. This treewalk procedure takes a tree and produces a stream of leaf node values. Changing it so that it returns a pair of the value, and a setter procedure that can modify the value in place, is trivial: the recursive auxiliary procedure for treewalk is given the parent node, so it can use set-car! or set-cdr! (depending on which side of the parent node this leaf node is) to update the parent:

(define treewalk
  (lambda (t)
    (if (not (pair? t))
        (stream-cons
          `(t . (lambda (v)
                  (error "Cannot modify singleton")))
          empty-stream)
        (stream-append (treewalk^ (car t) #t t)
                       (treewalk^ (cdr t) #f t)))))

(define treewalk^
  (lambda (t left? parent)
    (if (not (pair? t))
        (stream-cons `(,t . ,(if left?
                               (lambda (v)
                                 (set-car! parent v))
                               (lambda (v)
                                 (set-cdr! parent v))))
                     empty-stream)
        (stream-append (treewalk^ (car t) #t t)
                       (treewalk^ (cdr t) #f t)))))

The swapping is then just a matter of calling the setter for node1 with the value of node2, and vice-versa; this generalizes to swapping between multiple trees as well.

(define treewalk
  (lambda (t)
    (if (not (pair? t))
        (stream-cons `(t . (lambda (v)
                             (error "Cannot modify singleton")))
                     empty-stream)
        (stream-append (treewalk^ (car t) #t t)
                       (treewalk^ (cdr t) #f t)))))

(define treewalk^
  (lambda (t left? parent)
    (if (not (pair? t))
        (stream-cons `(,t . ,(if left?
                               (lambda (v)
                                 (set-car! parent v))
                               (lambda (v)
                                 (set-cdr! parent v))))
                     empty-stream)
        (stream-append (treewalk^ (car t) #t t)
                       (treewalk^ (cdr t) #f t)))))

But this modifies existing trees in place, a big no-no in functional programming land (though imperative programmers live with this every day). Is there a way to do this in, if not a fully functional way, then in a way that the side-effects are confined? (I mentioned to Roshan that this is a dual of Haskell’s monads: it behaves functionally even though it is imperative; monads gave the illusion of imperative behaviour even though they are fully functional. He pointed out that internally the state monad is optimized into imperative calls as well, so the comparison is even more interesting)

Instead of modifying existing trees in place, we want to copy the trees. But copying, naively done, would involve traversing the trees again, and that would be inefficient. The copying should be done Lazily as needed, and with streams, that means doing it while you’re walking the tree and collecting the nodes!

In short, at every internal node,
- create a new node, which has as its children the children of the original node;
- update the clone of the parent, instead of the parent, so that its car or cdr points to this new node
- the setter procedures for each leaf nodes modify these cloned nodes instead of the originals

...
(let ([copy (cons (car t) (cdr t))])
  (if left?
    (set-car! copy-parent copy)
    (set-cdr! copy-parent copy))
  (stream-append (treewalk-copy^ (car t) #t copy)
                 (treewalk-copy^ (cdr t) #f copy))
...

The main treewalk procedure now has to return the cloned tree as well, since at each subsequent stage of the walk only the direct parent is known, not the entire tree.

...
(let ([copy (cons (car t) (cdr t))])
       (if left?
           (set-car! copy-parent copy)
           (set-cdr! copy-parent copy))
  (stream-append (treewalk-copy^ (car t) #t copy)
                 (treewalk-copy^ (cdr t) #f copy))
...

Code, including stream implementation, available here

This being mid-term week for course I am a TA for, the lab sections I taught yesterday were made optional, and so apart from handling several questions on exam questions and assignments, I had some free time.

That time was spent reading Lambda the Ultimate, which was how I noticed the LiteratePrograms wiki. Based on Wikipedia’s MediaWiki, with added feature from the noweb literate programming system, it is to collaborative programming what other wikis are to collaborative writing. You can declare code blocks, and if the language is supported get syntax highlighting for free; the code is automatically packaged into a zip archive everytime someone hits the download link.

Check it out. But if you’re teaching a programming course, don’t mention it to your students

Two days ago, this tail call decorator
by Crutcher Dunnavant was mentioned on LtU. It’s a clever little trick: the decorator returns a new function that, when called, checks if its grandparent
is itself. This would mean that a function declared to be tail-call-optimized is called from another tail-called-optimized function, and so the caller’s frame in the stack can be wiped. How does it do this? The function throws an exception. This is caught by the grandparent, which then calls the saved function with the arguments passed in the exception.

This is the original implementation:

#!/usr/bin/env python2.4
# This program shows off a python decorator(
# which implements tail call optimization. It
# does this by throwing an exception if it is
# it's own grandparent, and catching such
# exceptions to recall the stack.

import sys

class TailRecurseException:
  def __init__(self, args, kwargs):
    self.args = args
    self.kwargs = kwargs

def tail_call_optimized(g):
  """
  This function decorates a function with tail call
  optimization. It does this by throwing an exception
  if it is it's own grandparent, and catching such
  exceptions to fake the tail call optimization.

  This function fails if the decorated
  function recurses in a non-tail context.
  """
  def func(*args, **kwargs):
    f = sys._getframe()
    if f.f_back and f.f_back.f_back         and f.f_back.f_back.f_code == f.f_code:
      raise TailRecurseException(args, kwargs)
    else:
      while 1:
        try:
          return g(*args, **kwargs)
        except TailRecurseException, e:
          args = e.args
          kwargs = e.kwargs
  func.__doc__ = g.__doc__
  return func

@tail_call_optimized
def factorial(n, acc=1):
  "calculate a factorial"
  if n == 0:
    return acc
  return factorial(n-1, n*acc)

print factorial(10000)
# prints a big, big number,
# but doesn't hit the recursion limit.

@tail_call_optimized
def fib(i, current = 0, next = 1):
  if i == 0:
    return current
  else:
    return fib(i - 1, next, current + next)

print fib(10000)
# also prints a big number,
# but doesn't hit the recursion limit.

There is a subtle bug: after handling the exception, func then calls its own saved version of g, not the g of the func that throws the exception. Modifying func and the TailRecurseException class so that the function to be called can be passed in the exception solves this nicely:

class TailRecurseException:
  def __init__(self, g, args, kwargs):
    self.g = g
    self.args = args
    self.kwargs = kwargs

  ...
  def func(*args, **kwargs):
    f = sys._getframe()
    if f.f_back and f.f_back.f_back         and f.f_back.f_back.f_code == f.f_code:
      raise TailRecurseException(g, args, kwargs)
    else:
      newg = g
      while 1:
        try:
          return newg(*args, **kwargs)
        except TailRecurseException, e:
          newg = e.g
          args = e.args
          kwargs = e.kwargs
  ...

Throwing an exception just to handle the required stack manipulation because the interpreter won’t handle it is grossly inefficient, granted, and requiring the programmer to declare functions as optimizable is not be ideal, but still a really neat trick.

As a side note, if a function can be called in tail and non-tail position, a non-decorated version should be made available, otherwise bad things can happen:

@tail_call_optimized
def add1(n):
  return (n+1)

@tail_call_optimized
def add2(n):
  return add1(add1(n))

This returns 41.

Modified version, and test cases, available here [IUCS] (backup copy here).

Update: added isTailCall by jorend. Now it is safe to optimize every function (though for efficiency reason you’d only optimize the ones that make tail calls).

While the aptness of the Whorfian hypothesis is still a highly-contentious issue when it comes to natural languages, the hypothesis more or less confirms itself again and again in the realms of programming languages. Certain programming styles are simply more suited to certain languages: you don’t try to use higher-order functions in C, after all; the lack of nested functions make this idea unworkable.

Which makes languages like Scala exciting. It’s fully functional, and also fully object-oriented.Integers are objects. Functions? Also objects. The List ADT familiar to functional programmer is now an Object, with methods such as fold and filter defined on it (which makes the syntax a bit confusing, but not if you come from Ruby. Or your first language is japanese. But I digress)

Best of both worlds? Well, not quite. The new Scala compiler looks promising, but it still trips up on this simple mutual-recursion example:

object RecEvenOdd {
  def rec_oddp(n: Int): Boolean =
    { if (n == 0) false else rec_evenp(n-1) }
  def rec_evenp(n: Int): Boolean =
    { if (n == 0 ) true else rec_oddp(n-1) }
  def rec_oddevenp(flag: Boolean, n: Int): Boolean = {
    if (n == 0) flag else rec_oddevenp(!flag, n-1)
  }

  def main(args: Array[String]): unit = {
    val n: Int = Integer.parseInt(args(0));
    System.out.println("recursive: " + n + ": " +       rec_oddevenp(true, n));
    System.out.println("mutual rec: " + n + ": " +       rec_evenp(n));
  }
}

The former gets detected as a tail call and optimized. The second? No such luck.. yet.

Of further note is the implementation of streams: Scala implements the “odd” style of lazy programming (Wadler et al., How to add laziness to a strict language without even being odd); the provided implementation is very similar to the reference implementation of streams for Scheme in SICP. The following program will thus unnecessarily compute the square root of -1:

object WadlerTest {
  def countdown(n: Int): Stream[Int] =
    Stream.cons(n, countdown(n-1));
  def sqrt(n: Double): Double = {
    System.out.println("sqrt: called with " + n);
    Math.sqrt(n);
  }
  def main(args: Array[String]): unit = {
    countdown(4) map { n => sqrt(n) } take(5) print
  }
}

The syntax can also be bizarre at times, a motley of Java-isms and proper FP style. For example,

{ n => sqrt(n) }

is η-reducible to sqrt, but Scala would not compile it with this definition of sqrt. Rewriting sqrt thus:

  def sqrt = { n: Double =>
  ...

now sqrt can be passed to map - but the automatic cast from Int to Double is no longer performed. Eek.

Moral of the story? I’m learning Haskell.

Eric and I were discussing scoping in Scheme and Python earlier today, our third over the past few weeks - and we finally nailed it shut. The first time he brought up dynamic scoping in Common Lisp and how Prof. Friedman dislikes it; the second was on how Python appears to have dynamic scoping (which turns out to be true, pre-Python 2.2), and now, thanks to Wikipedia, I think we have it right.

Provided Eric gets the H211 Introduction to Programming (Honors) class, which is in Python, and I get the C211 Introduction to Programming (Scheme), our discussion should stand us in good steed, though funnily today I played the Python guy and he played the Scheme one.

I’m going to show the examples, both in Scheme and Python; the first one in each section would appear to show that the language in question features dynamic scoping, which is incorrect as both actually do lexical scoping.

Scheme:
Bad:

(let ((pi 3.1415))
 (define area
  (lambda (r)
   (* pi r r)))
 (display (area 10))
 (newline)
 (set! pi 3)
 (display (area 10)))

Good:

(let ((pi 3.1415))
 (define area
  (lambda (r)
   (* pi r r)))
 (display (area 10))
 (newline)
 (let ((pi 3))
  (display (area 10))))

Python:
Bad:

pi = 3.1415
def area(r):
 return pi*r*r
area(10)
pi = 3
area(10)

Good:

pi = 3.1415
pi_holder = 10
def create_area():
 pi_holder = pi  # local pi_holder, different from pi_holder outside
 def area(r):
  return pi_holder*r*r
 return area
area = create_area()
area(10)    # 314.15
pi_holder    # Still 10
pi_holder = 3
area(10)    # Still 314.15

The above works, but is a bit problematic. I introduced pi_holder=10 to show that, (1), pi_holder inside of create_area() is a local variable; (2), that this local pi_holder is the one that is in area’s scope, and thus changing the value of pi_holder does not affect it.

Isn’t it easier to just do pi = pi ? Well, that does not work. My initial hunch was that Python reads the LHS of the expression, decided pi has been redeclared as a local variable, and thus since it’s not been initialized it got confused trying to assign it the value of itself. But it’s actually worse; this code does not work either:

x = 42
def local_var_test():
 temp = x
 print temp  # 42
 x = temp
 print x   # 42?

Surprise! Python won’t let you do that either. Take out the last two lines and the code works, though. Basically, if in the block a variable is declared anywhere, it is a local variable everywhere in that block, and trying to refer to a variable declared in the surrounding scope, even before the local declaration, will fail.

But this is where default parameters come in handy. A better way to rewrite the clunky code above is as follows:

pi = 3.1415
def create_area(pi = pi):
 def area(r):
  return pi*r*r
 return area
area = create_area()
area(10)    #314.15
pi = 3
area(10)    #314.15

So Python has static scoping after all. The thing to bear in mind is that Scheme functions are named closures, while Python functions inherit the surrounding scope, so to freeze the variables you depend on you have to wrap your function definition inside another function that copies in the values you need into its local variables.

References:

• Python programming language (Wikipedia): Closures
• Scope (Wikipedia)

And the funny thing is, I started the day trying to find good dynamic languages that run on the Java platform (platform envy, I guess, since .NET more prominently touts its language neutrality). Sun’s finally catching up, though - Tim Bray wrote a few months back about the Coyote project to support dynamic languages in Sun’s open source IDE, NetBeans, and pointed to an interesting Sun-developed scripting language, Pnuts. Which reminded me of Groovy and Boo.

Googling for groovy boo .net - Groovy being a Ruby-like scripting language for Java that received a lot of attention a few months ago, and then taken some flak over its development model, and Boo being the Python-like language for .NET - yields this very interesting Slashdot discussion that led me to such intriguing functional OO languages as Scala and Nice. .NET fans do not get to have all the fun!

Groovy, on the other hand, seems rather disappointing. Oh well. Scala looks more like Haskell, but with dynamic type inference (like Boo).. yay!

Update - 2005/06/06

Realized a few days ago, but haven’t gotten round to posting about it, that I was unfairly comparing Scheme and Python, and that Python methods are closures in themselves. Note:

pi = 3.1415
def area(r):
 return pi*r*r
print area(10)  # 314.15
def test():
 pi = 3
 print area(10) # 314.15
test()

In the earlier example, overriding the value of pi with pi = 3 is the equivalent of doing (set! pi 3) in Scheme, i.e. it will change the value of the variable that both the top-level pi, which is the one that area knows. In a dynamic scope, which uses a stack to figure out which value assignment should apply, pi = 3 would affect the call to area just after it.