Persisting Closures

A few weeks ago Bruno Vecchi submitted failing tests that serialize code references to KiokuDB. I finally got around to trying to make these tests pass on my flight to YAPC|10. Long story short, KiokuDB now serializes closures, including managing shared closure variables between several code references.

Serialization is relatively trivial. If you're not familiar with PadWalker, it's a module that lets you introspect the lexical pads of subroutines, including extracting references to all the closed over variables using the closed_over function.

The code itself is deparsed to text using B::Deparse, and the hash returned by closed_over is serialized along side it, as normal making sure that shared references are handled consistently. Any closure data that can be handled by KiokuDB will work normally.

Deserialization was slightly trickier, PadWalker needed additional functionality to actually assign to the subroutines' pads.

First KiokuDB compiles the body of the code while creating an empty lexical environment:

my $vars = join(", ", keys %$pad);

eval qq{
    my ( $vars );
    sub $body;
};

Then the new set_closed_over function added to PadWalker goes through the pad and aliases any values from the provided hash into the lexical scope of this subroutine.

This function goes through the CV's PADLIST field, and simply aliases the referents into the appropriate slots in the pad array.

On the one hand it's very reassuring that when you want to do this sort of thing in Perl you almost always can. What's slightly disappointing is that you do have to drop to C for these stuff.

My first version used the B module to introspect the structures, but this API is simply not powerful enough to allow modification of the data with aliasing semantics.

Thankfully the incredible amount of work in recent years on Perl 5 seems to be inspiring some new directions in Perl 5's core, and while the C API is pretty cumbersome, it does let you do almost anything you want if you want it hard enough.