git rebase Considered Awesome

git rebase is subject to quite a few controversial opinions, but in all my experience working with Git it seems that most people who use Git, even those who use rebase, myself included, often don't really know exactly what it does. Apparently, many people are afraid of using it because it does evil magic like changing the past and eating babies. The truth is that git rebase actually does so much more!

The critics of rebasing generally contrast it to Subversion, in which history is never modified after being recorded. However, much like the baby eating, this is just delusional superstition when you understand what Git actually does (and what subversion doesn't do).

Personally, I think git rebase should be used all the time; that is, git pull --rebase should be used instead of plain git pull (this can actually be the default).

You probably shouldn't rebase published heads, that can confuse people, but that's not a reason not to use rebase correctly.. I don't like the anti rebasing dogma:

• Just because you can do evil doesn't mean you should (or even that it's easy)
• Even if people do that anyway, if everyone uses git pull --rebase it won't actually be a problem^[1].

In my experience rebase always produced the most desirable results, creating a clean history instead of one riddled with meaningless nonsense. rebase is smart.

Explaining exactly what rebase does is a job for the documentation, but what I feel the docs are lacking on is explaining what rebase is actually for.

Forward-port local commits to the updated upstream head

Actually, I take that back. This is an excellent way of explaining what rebase is for. Assuming, of course, that you already know what that means ;-)

Broken down, we have a few fnords to sort out:

• upstream head
• updated head
• local commits
• forward porting

I don't really want to explain what these mean, just to clearly define them.

Upstream head

Git is distributed. Even if you work with a single central Git repository for your project, that's just a special, simplified case of a distributed workflow.

When you clone a repository you get your own local copies of the remote heads (branches).

When you commit your work only your own local copies of these heads are modified.

git push will overwrite the references on the remote side with the local ones, but by default only if the operation is safe (Git calls this "fast forward"), when the local revision is derived from the remote revision (the commit in the remote version is an ancestor of the local revision).

If this is confusing then you should read up on how to work with remote repositories in Git first.

Updated head

Between the point in time when you started working on an up to date local copy of a head and the the time you want to push your changes, other people may have modified the remote version.

When this happens the development has diverged, there are two chains of commits in history, one is the updated head, and one is the local head.

Local commits

The local commits is the chain of commits leading from git merge-base updated-head local-head to the local head.

These are commit objects that are not visible by walking the history of the upstream head (updated or not), but only by walking the history of the local head.

Forward porting

This is where the awesome of rebase comes into play.

You can run git merge updated-head in your local version to create a new merge commit, that will combine the changes two diverged histories leaving you with the results of both in both the commit log and the source tree.

This is what git pull does by default; it's the same as git fetch followed by git merge.

git rebase is much more clever.

The results of git merge and git rebase are the same in terms of the resulting trees: the files will end up containing the same merged changes, and you will similarly need to run git mergetool if there are any conflicts.

The difference is that git rebase will take your local commits and apply them one by one to the revision from the updated upstream head, effectively creating a brand new local head with new versions of the local commits, whereas git merge creates a single merge commit that is derived from both the local commits and the upstream ones.

A while ago I was unaware that rebase is actually very smart about how, and more importantly whether or not to apply every local commit. rebase supports patch idempotence, local changes which have identical changes upstream (even if the commit metadata is different) are simply skipped without error. This means that changes that were merged upstream, even if they were cherry picked, signed off, etc, will still be dealt with correctly.

Similarly, merge commits that are no longer meaningful will be omitted. If you run git pull followed by git pull --rebase the merge commit created by the first pull will be omitted from the results.

The new set of forward ported local commits is clean and minimal, and therefore easier to work with.

Habitual rebasing

If you always use git pull --rebase your local changes will never get out of control, resulting in a mess of branches and merges. They will simply be the most minimal set of changes needed to bring the upstream head into the changed version that you are trying to create in your development.

Correctly using rebase to create a clean local history is simple using git pull --rebase.

Furthermore, when working with local (and therefore probably unpublished) commits you can even modify them in other ways as you keep writing code. For a more managed approach to creating clean patches incrementally see Stacked Git. This sort of history (and change) rewriting is fair game. This is what Linus means when he says Keep your own history readable and Don't expose your crap.

svn rebase

Under Subversion, when you run svn commit but you aren't up to date, you need to run svn update to forward port local changes to the updated upstream head. Does that sound familiar?

The difference is that in SVN you're rebasing history that hasn't been saved anywhere yet. With git you can at least back out of it (git reset, git rebase --abort) if you run into hurdles like conflicts. Subversion is merciless, it will simply modify your working copy with conflict markers, and you have no choice but to clean up the mess.

So in a sense Subversion does just as much "evil history rewriting" as Git does, except that in Git every step of the process is recorded as explicit versioned data, that you can compare with and revert to freely, whereas in Subversion it's done as one monolithic operation to your uncomitted working changes.

Breaking idempotent patches

Git is not omniscient.

There are a number of things that won't be automatically cleaned up by git pull --rebase (and rightly so):

• commit --amend
• rebase --interactive's edit and squash
• git merge --squash

Applying such changes to a published branch is not just reorganization of history. The resulting commit history will have different changes when the deltas are actually computed.

Trying to rebase on top of something that had these operations applied to it will cause conflicts to the person running rebase, leading to confusion, which brings us back to the reason people are so afraid of rebase.

As long as you're aware of the implications of such operations, and make sure never to publish them where people won't expect them to happen, you can use them as much as you like.

[1] as long as there are no conflicts or other things that actually modify the deltas in a way that rebase can't automatically pick up

Reducing Scope

Jay Kuri recently made the distinction between external and internal dependencies. He makes the case that when you choose to internalize a dependency the implementation of that dependency usually suffers as a result.

Whenever you choose to internalize any part of a software component, in my opinion that's very similar to hard coding a value.

My code is often criticized for being too componentized or abstract. To use it you usually have to use an OO API (even for simple things), provide configuration values explicitly (even for obvious values), it will have many CPAN dependencies, etc, but there is a reason I code this way.

FAIL

Today I was trying to generate a password string from a large integer. Specifically I wanted something I can paste into a password input box, that contains alphanumeric and punctuation characters, and is the direct result of the output of Digest::HMAC.

There are many modules on the CPAN that generate passwords (Crypt::RandPasswd, String::MkPasswd, Crypt::PassGen). However, none of them can be used to solve my problem.

Generating random passwords involves two steps:

1. Generate a random seed
2. Encode a password based on that seed

All of these modules provide a new approach to the second step, implementing clever pronounceable passwords, interesting encoding schemes, and so on. This is the code I would like to reuse.

Unfortunately they all also internalize the dependency of generating random numbers, and the closest thing to an API to override that is this gem from Crypt::RandPasswd's documentation:

{
    local $^W; # squelch sub redef warning.
    *Crypt::RandPasswd::rng = \&my_rng;
}

So to adapt Crypt::RandPasswd to my requirements I'd have to recreate the rand API that takes an upper bound and returns a floating point number between 0 and that number.

This is better than all the other modules, at least it would let me use something like Crypt::Random::Source to make sure the passwords are truly random, and it's documented so it's unlikely to break if I rely on it, but this is hardly a solution.

I'm left with two choices, monkey patch or reinvent.

Object Oriented design

If Crypt::RandPasswd was an OO module I could perhaps subclass it instead of monkey patching, but that's not necessarily enough to be reusable.

If these modules had OO APIs that used two delegates, one to generate random data and one to encode random data into a string representation optimized for human consumption, I probably wouldn't have this problem.

Like Jay said, by externalizing the algorithmic dependencies here (even if they still end up in the same CPAN distribution), we're increasing the likelyhood that a clean, well thought API for each delegate would emerge. This is what makes polymorphism so powerful when used right.

My critics would argue that this would be bloated and overengineered, but if the problem of encoding passwords were separated into a standalone class, at this point the work is done.

There is no need for more features, or an extensible API. Another encoding scheme could be implemented using the same API and dropped into where this fits. it wouldn't need to worry about details like generating cryptographically strong random numbers, or providing additional customizability.

This scope reduction is in my opinion fundamental to writing reusable and maintainable code. It seems like a PITA until you have a working system, and it may seem like more work when you're writing such code or installing modules that are written in this way.

Old monolithic code can obviously be cannibalized or refactored into new code during the maintenance cycle, but if its scope was properly reduced to begin with the flawed component could be replaced a lot more easily, either by dropping in a replacement if the API can remain intact, or also adjusting the dependent code to use a better API.

An example of how to not write a proper solution to a self contained problem, take a look at Catalyst::Plugin::Session.

When I rewrote the system that predated it, I made the distinction between preserving state in HTTP and storing associated data on the server. However, I unnecessarily intertwined the implementations of these two sub problems, creating an overly complex hook based API that is still monolithic.

This code cannot be saved. At best, snippets can be copied and pasted into a new implementation, but since the design is flawed and the implementation is so large and complex, there is no hope of reuse or even a backwards compatible plugin API. At best the same API (or a subset) can be provided for applications, so that user code will not have to be rewritten.

Monoliths aren't always evil

Off the top of my head I can enumerate at least 6 different components of a session management system, with no overlap and very little interaction between them. This does not mean that you would have to configure at least 6 different Catalyst components/plugins just to get $c->session. Instead, a wrapper component that configures them for you (and does nothing but configure) would be written on top of them, to provide DWIM.

This is the role of a sugar layer. I've previously stated the advantage of Moose lies in its underlying componentization. However, Moose's success lies in its appearance of a monolithic system; the sugar is concise, friendly and works out of the box. When you look closer the real extensibility becomes apparent.

In this way Moose is also very future proof. We know that the sugar layer exported by Moose.pm is flawed, we've learned many new things since it was written, and we can provide a cleaner, friendlier and more powerful syntax in the future.

Reducing burdens

Some time ago I commented on a blog post asking how some of the more prolific CPAN authors manage to keep up. Scope reduction is the most fundamental part of doing that, they key is that I don't actually maintain a lot of that code, there is nothing left to do except fix minor bugs. Most of my modules will never get a new feature, instead that new feature would be written as a module that depends on the existing one.

It's a little more work up front, hard coding seems easy when you're doing it, but it's a very near sighted optimization. I suspect this is what makes the quality of code in Haskell's hackage generally higher than other languages, the type system makes it hard to be lazy that way on the small scale, serving as a constant reminder to keep things cleanly separated.

Thanks

Catalyst::Plugin::Session makes me cringe. I am very grateful to t0m for maintaining that piece of shit. Thankfully at least it taught me a valuable lesson in software design.

SV *

In my last internals post I mentioned that most of the data in Perl revolves around the SV struct.

The excellent PerlGuts Illustrated describes these structures in great detail, and of course there's the two main documents in Perl's own reference, perlguts which tells you about them and perlapi which tells you what you can do with them.

This post is sort of like PerlGuts for dummies, intended to lightly touch on the SV struct and then explain some typical usage but not much more.

Unfortunately this is going to a hard read. In early revisions I tried to show how to write a very simple XS function, but there are just too many details, so I will save that for a later time.

What's in an SV

I've stolen this diagram of an SV from PerlGuts Illustrated, but just this is only the tip of the iceberg. Click the link if you haven't already. Really. JFDI. I'll wait.

SV stands for scalar value. This C struct represents every singular value available in Perl, numbers, strings, references. The SV struct is technically what's called a tagged union, it contains a field that denotes that value's type, and the meaning of its 4 fields can change depending on that type.

The first field is called ANY. This contains a pointer to the actual data. We'll get back to this one in a second.

The second field is the reference count. This is just a simple integer. You can increment it by calling SvREFCNT_inc, and decrement it using SvREFCNT_dec. New SVs have a reference count of 1, and if you SvREFCNT_dec to 0 then the SV is destroyed.

The third and forth fields are the flags and the type. Again, for details refer to illguts, but the gist of it is that the type and the flags tell the various macros what structure the ANY field points to.

The simplest value is an unintialized SV. In this case the type is SVt_NULL and the ANY field is a null pointer.

So what about an SV containing a number?

In the Perl source code the type IV is an alias to the native integer type. It's pretty much the same as long or long long depending on the architecture and the arguments to Configure.

If you call sv_setiv(sv, 42) then the SV's ANY field will be set up such that it points at an an IV whose value is 42, and the SvTYPE will be SVt_IV, and set the IOK flag saying that the IVX slot has a valid value (this is actually not entirely accurate, so again, see illguts for more details).

The ANY field actually is set by the SvUPGRADE macro. If necessary this macro will allocate additional structures require for the target type, and changes the SvTYPE. In this case it will ensure storage area for the IV is available.

When the IV is actually stored in that structure the IOK flag will be enabled, signifying that there is a valid integer stored in this SV.

To extract an IV value from SV you use the SvIV macro. This macro resolves to an expression that will return a native integer type by from the structure pointed to by the ANY field.

So what happens when you call SvIV(sv) but sv actually contains a string? What SvIV actually does is check the IOK flag, which means that the ANY field points to a structure with a valid IV in it. If the flag is not set, then an IV will be generated (and stored) by numifying the value.

In the case of a string like "42" (whose type is SVt_PV), the value will be parsed using the grok_number function. The SVt_PV will then be upgraded to an SVt_PVIV, which has slots for for a string as well as an integer.

The next time SvIV is called on this SV the IOK flag will be set from the previous invocation, so it will just return the IV without computing anything.

So in effect, there is an SvTYPE for every kind of value value, and also for every sensible combination of values (in our second example we make an SV that is both an integer and a string at the same time).

As is the case everywhere in Perl, there is a rich set of macros to manipulate the structures, so that you generally don't have to think about the various flags and types, they are an implementation detail. You tell the macros what you want to get out of an SV and it does the hard work for you.

If you haven't already, please take a few minutes to at least skim illguts. It's not important to know all the different variations and wha they mean, but you should know what a reference looks like in C land, and how things like arrays are represented.

Stacks

Perl 5 is a stack based virtual machine. This means that in order to pass data around, pointers to SVs are pushed onto a stack by the caller, and taken off the stack by the code they are called. The called code does its thing, and then overwrites its parameters with pointers to the SVs that are the result of the computation.

The two main stacks that pass data around are "the" stack and the mark stack. SVs are pushed onto the stack, while the mark stack contains pointers to interesting places on the stack.

In order to push a value, you use the XPUSHs macro or a variant. X stands for "extend", it makes sure the stack has enough room for the value before adding it.

So what's the mark stack for? When you call a subroutine, like foo($x, $y) a new mark is pushed, then $x, then $y. The body of foo gets a variable sized list of parameters by looking from TOPMARK to the head of the stack.

To get values you can use the SP macro. $x is available as SP(0) (though there are many convenience macros that are usually quicker than SP).

As a side note, recall that values pushed to the stack are passed as aliases. This is because a pointer to the SV is copied to the head of stack (as opposed to a copy of the SV itself).

So what about reference counting? Pushing parameters is fine (there's still a counted reference in the call site), but when you're pushing a return value you've got a problem: if you SvREFCNT_dec the value then it will be destroyed prematurely, but if you don't then it will be leak.

Fortunately Perl has yet another stack to solve this, the tmps stack. When you call sv_2mortal(sv) a pointer to sv is saved on the tmps stack, and on the next call to the FREETMPS macro all of these mortal SVs will have their reference count decremented.

Assignment and copies

In order to assign the value of SV to another, use the SvSetMagicSV to copy the data over. It doesn't copy the pointer in ANY, but the values themselves.

One important thing to keep in mind is that when assign copy a reference, like:

my $x = \%hash;
my $y = $x;

$y is actually copied, it doesn't share any data with $x. The two SVs are both pointing at the same HV (%hash).

This copying is also done when returning values from a subroutine.

It's important to keep in mind that most operations in Perl involve copying the SV structure, because assigning an SV pointer actually creates an alias. Accidentally creating aliases is typical issue I have as an XS n00b.

In XS we usually use newSVsv to create a copy, because often the target SV does not yet exist. This is the same as creating a new empty SV and calling SvSetMagicSV on it.

If you've ever seen the error Bizarre copy of %s? That's what happens when sv_setsv is called on a value that isn't a simple SV, such as a number, string or reference. Correctly copying arrays and hashes involves making copies of all the SVs inside them and reinserting those copies into a new AV or HV.

References

So as I just mentioned, there are 3 valid things to copy around using setsv: undef, simple values (strings and numbers), and references. Complex data structures are passed by reference, or by evaluating in list context in a way that simply puts every element SV on the stack.

A reference is an SV whose ANY field points to an SV * which poinsts at another SV.

Calling the SvRV macro on an SV with ROK set returns the target SV.

In the case of e.g. \@array the SV will return true for SvROK(sv), and SvRV returns the AV of @array.

Casting

When working with references casting becomes important. Structs like AV, HV etc have the same fields as SV but the operations you can perform on them are different. The various API functions operating on these types require pointers of the correct type, which means you often need to cast points from one type to another.

For intsance, if we've received a reference to an array in ref_sv, we could do something like this:

SV *ref_sv; /* assume ROK is set */

SV *array_sv = SvRV(ref_sv);
assert( SvTYPE(array_sv) == SVt_PVAV );

AV *array = (AV *)array_sv;
SV **elem = av_fetch(array, i, FALSE);

Fin

This post is getting very long so I will STFU now. In the next post I'll review the basics of XS boilerplate, and that plus this post should be enough to get started on reading/writing simple XS code.

Bionic eyes to go with your MacBook

So I was just informed that my MacBook probably doesn't have millions of colors. I know, shocking. What amuses me is that the guy repeatedly admits to not being able to tell the difference with his eyes, he tries to confirm with Apple over the phone instead, and fails to get a real answer.

There are two messages here:

1. Apple sucks
2. Millions of colors matter

I agree with the first point. Apple is evil and it's getting worse.

The second point is bullshit. He justifies it by saying that as a designer color is important to him. However, the whole post implies that he is not actually able to see the difference, which is why he is calling Apple.

If he can tell the difference why does he need to call/write/confirm? If it makes his design work suffer, can't he tell the difference while he's working?

Furthermore, the people for whom he is designing things probably can't see the difference either, they don't buy first class displays and they they are not professional designers.

I find this lack of perspective a little sad.

jrockway conjectured that this is because bit depth is a measurable spec that you can compare, like LCD refresh rates. People pay lots of money for 1ms refresh rates (1000 FPS), whereas displays with 5ms refresh rates (200 FPS) are cheaper. I don't think either of these solves the wagon wheel effect, there is still aliasing happenning, and anything over 30 frames per second is smooth enough for non pathological cases). And yet people still pay good money for this. Even when their video card has a maximal refresh rate of 120 Hz.

Compare this with a much more meaningful value like the dynamic range of a display (how deep the black is and how bright the white is) matters a lot more. We don't really have a measurement system for that.

What we have is vendor made up values like the contrast ratio, which you can "improve" by jacking up the output of the backlight, while giving your display shitty looking blacks. I bet my MacBook's display really sucks in terms of dynamic range, but guess what, I can't tell. This is despite the fact that top of the line scanners give slightly over 4 f-stop equivalents of range, whereas the human eye can see about 60.

Even though most people will easily be able to tell the difference between a display with a high dynamic range and one without, things like dust, glare and ambient light make a much bigger difference. Blended HDR images give the same dramatic effect while faking dynamic range, and we're able to enjoy them just fine in non optimal conditions. Is it really worth all that extra money?

IMHO there are much better reasons to hate Apple, like the iPhone jail and the awful app store policies, supporting DRM, backwards incompatible changes (10.5's language switching is still driving me nuts after all these months). As a long time Mac user (since 3rd grade), for the last 10 years I've been feeling more and more like I'm being shafted by apple, to the point where I know I want to switch (probably to Ubuntu), but I'm just too lazy at the moment.

I agree that their policy of treating their customers like children is annoying and negative, but we have bigger fish to fry.

So lastly, If you agree and you're a programmer, please keep this in mind the next time you are benchmarking or optimizing your code, decide whether it makes an actual difference, or whether the numbers just look better on paper =)

Crashing OSCON

Unfortunately Dave Rolsky can't make it to OSCON. Coincidentally I'm in the area visiting John Wang, so I'll be standing in for Dave. This means I get all the rock star perks without actually writing any slides ;-)

As a meta note, I'd like to say that open source collaboration really gives me a warm fuzzy feeling. Dave has generously picked up the slack of releasing Moose and Class::MOP over the past year, and when Dave couldn't make it to YAPC either, Shawn and Jon gave his tutorial there. Many of the MooseX:: modules are released by whoever patched them last.

I think this is compelling evidence that the Perl community in general and the Moose community in particular are two of the warmest and fuzziest of them all. Clay Shirky gave a short but fascinating talk on this topic:

Optimizing Moose's Startup

It's well known that while Moose competes well with hand written code at runtime. However, Moose code still needs to pay a lot at compile time, due to the comprehensive extensibility of the metamodel. We've previously made fairly successful attempts to optimize startup but there's always room for improvement.

Thankfully Goro Fuji has received sponsorship to optimize Moose's startup time, and even though he only started working recently his efforts are already paying off. I expect to see many exciting developments by the time August is through!

Ongoing performance improvement projects have been slow going due to their difficult nature, and lack of volunteer time. Because of this I am very pleased to have such a talented programmer working towards solving this problem.

If you would like to see possible improvements to your Moose based code's startup time, please consider contributing NYTProf profiles, so that we can more easily identify the slow path for real world scenarios. Different apps have different needs, and the more data we have the clearer the picture gets.

Holy crap, I suck at coding

Every so often I look back at old code and think to myself WTF was I on when I was writing it. Just take a look at my Perl module releases in chronological order^[1].

Usually it's small things I find annoying, when just looking at the code it with fresh eyes is enough to unveil small idiocies. It's easy to judge because I don't need to sugar coat any criticism I have when I'm the one under the lens. This stuff is usually the result of refactoring too quickly, and not rounding off all the corners, especially if the code I was refactoring was new code.

More embarrassing is when I find flaws in my assumptions or thought process: things that are more complicated than they should be, or inability to separate an idea from the engineering context it emerged from. This is usually the result of rushing through something (often towards the end of a project) and not thinking things through, being ignorant of a better approach, or just plain old lack of insight.

This process used to make me feel bad, but now it makes me feel good. I used to be concerned with how fast I learned things, eager to understand ideas that would propel me towards finding the ultimate paradigm [singular. ugh!] of the future.

When your goals are as laughable as that failure makes you feel bad.

I think now I'm much more relaxed about everything. I'm not looking for a silver bullet at all anymore. I'm looking for fun ideas to "spice up my routine" as they say. If I remain interested I usually produce dramatically better results than if I need to force myself to work.

Leisure has replaced ambition as the primary purpose of my learning efforts.

However, without ambition to drive me forward, I occasionally feel like I'm not trying hard enough to better myself. Seeing past failures (especially recent ones) is an great way of reassuring myself that my brain has not in fact crawled under a cupboard and died.

Recognition of past failure was always evidence that I am still learning, but in the past I thought it meant that I was not yet any good. Now it just means that I'm still having enough fun to stay curious, and I'm also learning what I should be unlearning.

So now that I am old and wise, I know that when I grow up I want to still be having fun, or at least to realize why this blog post is full of shit.

[1] Unfortunately the backpan directory listing doesn't seem to have the dates right, so the linked listing doesn't contain mistakes that have been superseded with new module uploads, only ones big enough to completely give up on, or that have been fixed by other people ;-)

Bootstrapping KiokuDB::Cmd

I just closed a ticket in KiokuDB's RT queue that asked for simpler dependencies. The fix was splitting the KiokuDB::Cmd modules into their own distribution, making the core more easily installable on Windows.

The problem is that I depended on Proc::InvokeEditor for the kioku edit command. Unfortunately that module does not install cleanly on Windows, preventing KiokuDB itself from installing, even though it's not required for normal use.

The problem is that KiokuDB's command line tool requires KiokuDB::Cmd to run. What I ended up doing is having the kioku script (which still ships with KiokuDB) allow bootstraping KiokuDB::Cmd by using Module::AutoInstall (if it's available and the user grants permission to do so).

If kioku can't require KiokuDB::Cmd cleanly, then it calls try_auto_install function:

sub try_auto_install {
    if (eval {
        require Module::AutoInstall;
        require ExtUtils::MakeMaker;
        1;
    }) {
        print "\n";
        if (
            ExtUtils::MakeMaker::prompt(
                "Module::AutoInstall is available. " .
                "Should I use it to install KiokuDB::Cmd?",
                "N/y",
            ) =~ /^y/i
        ) {
            my @installed = Module::AutoInstall->install(
                [],
                "KiokuDB::Cmd" => $KiokuDB::REQUIRED_CMD_VERSION,
            );
            return @installed > 0;
        }
    }

    return;
}

If installation seems to have succeeded then we delete $INC{"KiokuDB/Cmd.pm"} and try to load it again.

Otherwise the script exits with an error asking the user to install KiokuDB::Cmd manually.

I think this is a fair tradeoff, especially if you don't need the command line tools in production deployments (you can still make backups by programmatically scanning the database yourself).

Degradable Gists (update)

The good people at GitHub have implemented a JSONP api for Gists, so I have updated my graceful degarding script accordingly (see previous post).

Now the code no longer needs to hijack document.write and has better load time characteristics as well. If you use traditional <script> tags then each gist will block the loading of the page at that point, causing jittery page renderings, and preventing the rest of the content from appearing. This script styles the <pre> tags when the DOM becomes ready, and only fetches the syntax highlighted HTML after the onload event, when all other page elements have finished loading.

The second new bit is the Perl scripts I've added to streamline working with these degradable Gists.

The first script is very simple, given a Gist ID it prints out a <pre> tag suitable for inclusion into your HTML document. For instance make_pre_tag.pl 115368 outputs:

<pre id="fake-gist-115368" class="fake-gist">use Moose;
has fun =&gt; ( isa =&gt; &quot;Constant&quot; );</pre>

The second script, update_gists.pl is much more interesting. It uses XML::LibXML to find all the pre tags with the class fake-gist in a document. If these tags have no ID then App::Nopaste is used to create new Gists. Otherwise AnyEvent::HTTP is used to concurrently fetch the updated versions from http://gist.github.com/.

I start by editing a plain XHTML document in my editor, adding code in the <pre> tags, and then I post the code blocks to github all at once. I use CDATA sections to escape the source code in the XHTML document.

<pre lang="perl" class="fake-gist"><![CDATA[
$object->method(
    foo => $bar, 
);
]]></pre>

The script re-escapes that using HTML entities so that it is valid HTML (which doesn't support CDATA). This way the <pre> tags render fine under quirks mode, allowing jQuery to append HTML strings to the document.

PL_runops

I'm going to try to do a series of posts about learning Perl internals. I am still a beginner, I have trouble remembering the many macros, or keeping everything in my head all at once, so hopefully I will be able to make some sense of this stuff in a way that is accessible to other beginners.

While I'm definitely diving right in to the deep end, I think PL_runops is a good place to start as any, there's not a lot you need to learn to see how it works. I don't think Perl has a shallow end.

PL_runops is a variable in the interpreter containing a function pointer, which in most cases will be Perl_runops_standard.

Perl_runops_standard is the function that executes opcodes in a loop. Here's how it's defined:

int
Perl_runops_standard(pTHX)
{
    dVAR;
    while ((PL_op = CALL_FPTR(PL_op->op_ppaddr)(aTHX))) {
        PERL_ASYNC_CHECK();
    }

    TAINT_NOT;
    return 0;
}

Perl makes extensive use of macros, which can sometimes be confusing, but in this instance it's not too daunting. This loop will essentially keep calling PL_op->op_ppaddr, assigning the result to PL_op. As long as a valid value is returned it will keep executing code.

Just to get it out of the way, PERL_ASYNC_CHEK is a macro that checks to see if any signals were delivered to the process, and invokes the handlers in %SIG if necessary.

So what's PL_op? It's essentially Perl's instruction pointer, it refers to the opcode currently being executed. When Perl compiles source code it produces an optree. This is probably one of the more complicated structure in Perl internals, but right now we're only concerned with one small part, the op_ppaddr field of a single node in the tree. The op_ppaddr field contains a pointer to the function that implements the op.

PP stands for push/pop, which means that it's a function that operates in the context of the Perl stack, pushing and popping items as necessary to do its work, and returns the next opcode to execute.

pp.h defines a PP macro, which sets up a signature for a function that returns an op pointer. Let's have a look at two simple PP functions. First is pp_const:

PP(pp_const)
{
    dVAR;
    dSP;
    XPUSHs(cSVOP_sv);
    RETURN;
}

This is an implementation of the const op, which pushes the value of a literal constant to the stack. The cSVOP_sv macro is used to get the actual SV (scalar value structure) of the constant's value from the optree. SVOP is an OP structure that contains an SV value. The c stands for "current".

Let's rewrite the body of the macro using some temporary values:

/* the current opcode is op_const, so the op structure is an SVOP
 * Instead of using the long chain of macros we'll put it in a variable */
SVOP *svop = (SVOP *)PL_op; 

/* and then we can simply use the op_sv field, which
 * contains a pointer to a scalar value structure */
SV *sv = svop->op_sv;

This SV is then pushed onto the stack using the XPUSHs macro. The X in XPUSHs means that the stack will be extended if necessary, and the s denotes SV. To read the documentation of these macros, refer to perlapi.

The next thing that executes is the RETURN macro. This macro is defined in pp.h, along with a few others:

#define RETURN          return (PUTBACK, NORMAL)

#define PUTBACK         PL_stack_sp = sp

/* normal means no special control flow */
#define NORMAL          PL_op->op_next

/* we'll use this one later: */
#define RETURNOP(o)     return (PUTBACK, o)

Recall that there are no lists in C. The comma operator in C executes its left side, then its right, and returns that value (unfortunately we have that in Perl, too). The RETURN macro therefore desugars to something like:

PL_stack_sp = sp;
return(PL_op->op_next);

The XPUSHs macro manipulated the local copy of the pointer to the stack, sp as it was adding our SV. This change is not immediately written to the actual stack pointer, PL_stack_sp . The PUTBACK macro sets the "real" stack to the version we've manipulated in the body of our opcode.

Then the opcode simply returns PL_op->op_next. The op_next field in the op contains a pointer to the next op that should be executed. In this case if the code being executed was:

my $x = 42;

then the const op compiled to handle the 42 literal would have pushed an SV containing the integer 42 onto the stack, and the op_next in this case is the assignment operator, which will actually use the value.

So, to recap, when PL_op contains a pointer to this const op, PL_op->op_ppaddr will contain a pointer to pp_const. PL_runops will call that function, which in turn will push ((SVOP *)PL_op)->op_sv onto the stack, update PL_stack_sp, and return PL_op->op_next.

At this point runops_standard will assign that value to PL_op, and then invoke the op_ppaddr of the next opcode (the assignment op).

So far so good?

To spice things up a bit, here's the implementation of logical or (||):

PP(pp_or)
{
    dVAR; dSP;
    if (SvTRUE(TOPs))
        RETURN;
    else {
        if (PL_op->op_type == OP_OR)
            --SP;
        RETURNOP(cLOGOP->op_other);
    }
}

The or op is of a different type than the const op. Instead of SVOP it's a LOGOP, and it doesn't have an op_sv but instead it has an op_other which contains a pointer to a different branch in the optree.

When pp_or is executed it will look at the value at the top of the stack using the TOPs macro, and check if it evaluates to a true value using the SvTRUE macro.

If that's the case it short circuits to op_next using the RETURN macro, but if it's false it needs to evaluate its right argument.

--SP is used to throw away the argument (so that $a || $b doesn't end up returning both $a and $b).

Then the RETURNOP macro is used to call PUTBACK, and to return PL_op's op_other. RETURN is essentialy the same as RETURNOP(NORMAL). op_other contains a pointer to the op implementing the right branch of the ||, whereas op_next is the op that will use the value of the || expression.

This is one of the most basic parts of the Perl 5 virtual machine. It's a stack based machine that roughly follows the threaded code model for its intermediate code structures.

The data types mostly revolve around the SV data structure, and moving pointers to SVs from op to op using the stack.

For me the hardest part to learn in Perl is definitely the rich set of macros, which are almost a language in their own right.

If you search for runops on the CPAN you will find a number of interesting modules that assign to PL_runops at compile time, overriding the way opcodes are dispatched.

For more information on Perl internals, the best place to start is perlguts, and the wonderful perlguts illustrated.