Three ways to shoot yourself in the head - the dangers of too much magic

So, there was some fascinating discussion on my first post that got onto the subject of some of the magic tricks that people play under the hood, notably touching on Smalltalk and Ruby. Piers Cawley summed up my real views on the subject for me, noting

• "With so many of these things it's not the capability itself that's bad, it's the of the bad uses people put it to."

and I use a hell of a lot of magic in perl sometimes, albeit carefully encapsulated magic - the principle reason I'm still using perl is that python and ruby just aren't as flexible as perl. Which means that this article is going to be perl all the way down, because of course more flexible means more ways to shoot yourself in the face - and I'm going to run through three that I've run into in production code and been driven insane by.

Headshot One: sub UNIVERSAL::foo { }

This is probably the least horrible of the things I'm going to discuss - as Piers points out, smalltalk does something fairly equivalent to this a lot to great effect. The idea of this is that you can provide a method on all objects in your program with a single definition, so for example

sub UNIVERSAL::debug_print { require Data::Dumper; warn Data::Dumper::Dumper($_[0]); }

would mean that for any object in your program, you're guaranteed to be able to call

$obj->debug_print

and get the contents of your object dumped to STDERR.

Or, at least, that's the theory. Where this can become a headshot is in its interactions with other things - for a start, now you're messing with a namespace that's shared interpreter-wide so you can get name clashes. But, of course, people are all careful about injecting things into UNIVERSAL:: because they know that, so you're pretty much safe, right?

Wrong.

Just because you don't get a clash at that level, you're effectively making the assumption that no class your program will load, ever, will define a method of that name with a different meaning! So if you have a pretty printer class -

package PrettyPrinter;

sub debug_print {
  my ($self, @to_print) = @_;
  die "no arguments passed to debug_print" unless @to_print;
  ...

then as soon as you try

$obj->debug_print

on your pretty printer ... exception time!

The place I encountered this one personally was with UNIVERSAL::moniker combined with Template::Plugin::Class - the latter returns a proxy object that catches all unknown method calls directed to it with an AUTOLOAD (which I considered mentioning for this article but is only really a footshot ... unless somebody else already put one in the global namespace). However, this proxy object only accounts for the standard UNIVERSAL methods, so moniker gets called on the proxy object and fails spectacularly to be any use. The solution, of course, was to inject an explicit moniker method into the proxy object's class, but that's an evil workaround in and of itself. However, provided you know what you're doing a sparse few extra global methods can be very powerful, so it's arguably the least scary of the techniques

Headshot rating: May take an ear off.

Headshot Two: Wrapping or replacing other people's methods

This technique, also known as monkey patching, is a common means of in-place modification of other people's code to allow tweaking without requiring everything deals with a factory. This -can- in theory be a very useful tool, although you should always try to subclass - but, sometimes (for e.g. with my Template::Plugin problem above) you either don't have time or don't have an option to integrate patches upstream or ...

So, anyway, a first attempt might be

sub MonkeyPatch::TargetClass::target_method { ... }

which is ok in theory except ... (1) your sub is declared in the current package so your class globals won't be right,and (2) if the other class ever declared a target_method method itself it'll all go boom. Now, neither matched for my problem above, and if they do perl will give you a handy 'method redefined' warning so you know you did something silly, so this is only really a shooting yourself in the foot moment (I don't think every language has this warning though,  it was added because of perl programmers shooting themselves in the head due to its absence years back ... sometimes being old and unfashionable is an advantage :)

So. Let's assume for the moment there -is- a target method and you want to wrap it. Well, so, I guess the easiest way is just to rename the old one and replace it, right?

package MonkeyPatch::TargetClass;

*old_target_method = \&target_method; # set symbol to subref
{
  no warnings 'redefine'; # only turn the warning off within the current block
  *target_method = sub {
    my $self = shift;
    ...
    $self->old_target_method(@_);
  }
}

Ok, that's fine. But what if somebody -else- tries to wrap this the same way? They'll overwrite old_target_method with your wrapped version and ... BOOM. Alright, so can we wrap it in place? Well, in perl, yes we can -

package MonkeyPatch::TargetClass;

my $old = \&target_method;
{
  no warnings 'redefine';
  *target_method = sub {
    my $self = shift;
    ...
    $self->$old(@_);
}

So now if somebody else does the same thing, they'll grab our redefined version and wrap that and there are no conflicts - well, so long as our wrappers aren't order-dependent, but at that point you're beyond what this approach can ever handle and it's time to break out multiple inheritance and C3 mro support.

But ... what if we don't know if the method's defined in TargetClass or one of its superclasses? Worse still, what if it -is- currently in TargetClass but an update to the library that comes from factors it out into a base class? You guessed it: BOOM.

So, one last tweak -

my $old = &target_method;

my $old = __PACKAGE__->can('target_method'); # look up along MRO

So, this one gets a vote of: bloody hard to get right, still dangerous even then, but sometimes useful.

Headshot rating: All fun and games until somebody loses an eye

Headshot Three: Source filtering

And finally, the most strange and fascinating of the bunch, and the one which a client of ours got shot in the face by in the wild inspiring me to write this article in the first place. Source filtering is ... well, for those of you unfamiliar with perl the implementation you're likely to have encountered is C preprocessor macros; I don't -think- any other dynamic languages have been foolish enough to copy this particular feature and if the author of your preferred one proposes doing so, please break his fingers and confiscate his keyboard until the madness passes.

So, right, source filters allow you to provide a textual preprocessor to attach to your source code which runs before the file is fed to the interpreter's parser. Which theoretically allows you to extend the language to do -anything-, for example add a completely pointless switch statement (you can produce something almost identical with other syntactic constructs - if anybody cares comment and I'll elaborate).

It also allows you to shoot yourself in the head. Spectacularly. Observe -

package ShootMeInTheHead;

use strict;
use warnings;
use Switch;

=head1 ShootMeInTheHead

=cut

sub foo {
  my ($variable) = @_;
  return $variable + 1;
}

sub bar {
  my ($variable) = @_;
  return $variab1e + 1;
}

A very simple piece of code, and one with a visible typo - the last return line has a 1 in 'variab1e' which should be an l. And lo, asking perl to compile it yields

Global symbol "$variab1e" requires explicit package name at line 18.

and thus informed of our silly error we may swiftly correct it. Except. Consider the following identically buggy but subtly different file -

package ShootMeInTheHead;

use strict;
use warnings;
use Switch;
# load switch statement

=head1 ShootMeInTheHead

=cut

sub foo {
  my ($variable) = @_;
  return $variable + 1;
}

sub bar {
  my ($variable) = @_;
  return $variab1e + 1;
}

which, while -functionally- identical (in theory), now contains the magic word 'switch'. Now, since we added a line, the compilation error should now be line nine, right?

Global symbol "$variab1e" requires explicit package name at line 14.

Erm. But that's the "return $variable + 1;" line further up. Which is -correct-.

Which is where we originally came in, with one of Shadowcat's clients contacting us in a not very happy state of mind because their core app wouldn't start and their devs swore blind the line the compiler error was coming from was correct. And, indeed, they were right - that line was. A quick comment-out of the 'use Switch' and dependent code revealed the correct line (some 200 lines off, their code was a tad more involved than this), I corrected a typo in a variable name, and lo, the app was working again.

I'll confess to not having tracked down -exactly- how Switch's tweaks to line numbering manage to mangle the error reporting, and I'll also confess to not particularly caring - my fundamental advice to anybody considering using a source filter that alters things beyond a single line is, and will ever be, don't.

Headshot rating: Right between the eyes.

So, having scared myself and perhaps a number of my readers too, I think I'm going to go for a beer and some cat macros while I recover. Adieu!