Writing a perl REPL part 3 - lexical environments

(this is a continuation of the series started in this post, so you may want to start there)

New tool for today: Rocco Caputo's amazingly handy Lexical::Persistence module. We're going to use this to persist lexical variables - i.e. those declared with 'my $var' - between invocations. But before we can do that, we need to break Devel::REPL out a bit and move it from doing a simple string eval on the supplied line to building a subroutine reference from it. So, -now- we change $self->execute($line) in run_once to call $self->eval($line) instead, and expand the eval code as follows -

sub eval {
  my ($self, $line) = @_;
  my ($to_exec, @rest) = $self->compile($line);
  return @rest unless defined($to_exec);
  my @ret = $self->execute($to_exec);
  return @ret;
}

sub compile {
  my $_REPL = shift;
  my $compiled = eval $_REPL->wrap_as_sub($_[0]);
  return (undef, $_REPL->error_return("Compile error", $@)) if $@;
  return $compiled;
}

sub wrap_as_sub {
  my ($self, $line) = @_;
  return qq!sub {\n!.$self->mangle_line($line).qq!\n}\n!;
}

sub mangle_line {
  my ($self, $line) = @_;
  return $line;
}

sub execute {
  my ($self, $to_exec, @args) = @_;
  my @ret = eval { $to_exec->(@args) };
  return $self->error_return("Runtime error", $@) if $@;
  return @ret;
}

sub error_return {
  my ($self, $type, $error) = @_;
  return "${type}: ${error}";
}

The end result here is that we end up with '1 + 1;' in the REPL becoming "sub {\n1 +1;\n}\n" before execution; since perl treats the final expression in a sub as an implicit return if no explicit one is present, everything continues to work as before - but we now have a bunch of extra hooks with which to modify the execution flow (we'll need the hook offered by mangle_line today, and the error_return one will come in handy for printing backtraces and similar should we desire those later).

The reason for the odd setup in 'sub compile' is to ensure only the $_REPL variable is in scope when the compilation takes place; this means that we have any other variable name to ourself in the code being executed (and accidental specification of any other variable without declaring it will cause a compile-time error).

So, main code refactored, on with Devel::REPL::Plugin::LexEnv -

package Devel::REPL::Plugin::LexEnv;

use Moose::Role;
use namespace::clean -except => [ 'meta' ];
use Lexical::Persistence;

has 'lexical_environment' => (
  isa => 'Lexical::Persistence',
  is => 'rw',
  required => 1,
  lazy => 1,
  default => sub { Lexical::Persistence->new }
);

The -except in namespace::clean is new, and actually indicates a mistake I made in the first article - fortunately, since Devel::REPL itself inherits a meta method from Moose::Object it didn't actually break anything, but in the case of a plugin we can't unimport it since a role isn't a class and can't have a superclass. I'm using a slightly more verbose formatting for the attribute for clarity; anybody who prefers one or t'other should leave a comment to that effect. Also, note that the isa type is actually referencing a type constraint (see Moose::Util::TypeConstraints for the list of standard ones and the functions to create custom ones), which since it doesn't exist Moose automatically creates for us as a subtype of Object which requires that the value passes ->isa('Lexical::Persistence'). Now for the meat -

around 'mangle_line' => sub {
  my $orig = shift;
  my ($self, @rest) = @_;
  my $line = $self->$orig(@rest);
  my $lp = $self->lexical_environment;
  return join('', map { "my $_;\n" } keys %{$lp->get_context('_')}).$line;
};

around 'execute' => sub {
  my $orig = shift;
  my ($self, $to_exec, @rest) = @_;
  my $wrapped = $self->lexical_environment->wrap($to_exec);
  return $self->$orig($wrapped, @rest);
};

Ok, this is slightly involved for not many lines of code. So in order to make it clearer and ease your mind that I'm still sane, we'll examine it in reverse order.

return $self->$orig($wrapped, @rest);

Ok, by this point we've got a subroutine reference that's been wrapped by the Lexical::Persistence context, so we pass that off to the base Devel::REPL execute to deal with normally (note @rest currently doesn't do anything - passing it around all over the place is basically a politeness in case another plugin author wants to use it).

my $wrapped = $self->lexical_environment->wrap($to_exec);

This is the bit where Lexical::Persistence does its magic. It wraps the subroutine reference built by the $repl->compile process in code that fills out the lexical environment of the subroutine from the data it has stored - so if the $lp has data for a $foo variable and it sees a 'my $foo' lexical in the subref, it sets the value of that lexical to its data for it before execution - and then afterwards, it goes through the lexicals' current values and saves them away. Which means -

$ my $foo = 3;
3
$ $foo + 1;
4
$

- that the variables we've declared are now persistent between lines executed within the repl, even though each line becomes a separate subref with its own independent lexical environment and namespace.

Which begs the question, why did the second line compile? Without the plugin we'd simply get an error saying $foo isn't declared, since the code is still compiled under 'use strict'. The magic for this is in the first around, in this line:

return join('', map { "my $_;\n" } keys %{$lp->get_context('_')}).$line;

which pulls out all the keys from the default context (which currently is the only one we're using, and called '_' since other contexts in Lexical::Persistence are keyed by variable name prefix) and constructs declarations from them which are prepended onto the line during mangling and before the subref is compiled. To see what I mean by this a bit more clearly, I'll declare a couple more variables and then replicate the behaviour (getting $lp directly to save space)

$ my $foo = 3;
3
$ my @bar = (1, 2, 3);
1 2 3
$ my $baz = 'spoon!';
spoon!
$ join('', map { "my $_;\n" } keys %{$_REPL->lexical_environment->get_context('_')});
my $foo;
my $baz;
my @bar;

$

and also, here we see one other useful behaviour - variable names starting with '_' aren't persisted by default, thus avoiding us needing to worry about $_REPL being accidentally stomped on by Lexical::Persistence. You should probably have a read of the full documentation to get the big picture, though, along with PadWalker and Devel::LexAlias if you want to understand how all this is implemented under the hood. And, of course, the complete final code to the plugin.

Here endeth part 3, two days late due to a combination of extreme tiredness one night and having to wait for my laptop to dry out after being caught in heavy rain on the walk home the next. I'm not entirely sure what I'll be writing about next time round, but hopefully a few of you will turn up to find out anyway. Later ...