working effectively with legacy perl code

Working Effectively with Legacy Perl Code

Erik RantapaaFrozen Perl 2010

What is Legacy Code?

What is Legacy Code?

Uses old/obsolete perl and modulesNot well factored / organically grownOriginal developers goneNo one fully understands itUses outdated practicesHard to modify without breaking itLong new developer ramp-up timeNo documentation (or wrong documentation)No tests

On the other hand... Doing useful work - generating revenue

What is Legacy Code?

Michael Feathers - "Code without tests"

Ward Cunningham - "Accumulated technical debt"

Stuart Halloway - "Legacy is the degree to which code: - fails to capture intent - fails to communicate intent - captures irrelevant detail (ceremony)"

Example Legacy App

e-commerce applicationover 10 years old"rewritten" at least a couple of timesworked on by lots of different developers> 1000 .pm files> 150 database tables, >2000 columns3 templating systems, 100's of template filesused perl 5.8.0 until recentlylots of old versions of CPAN modules (some customized)didn't compile with -cw until recentlyrm -rf not an option

Overview

Unit Testing from a Perl perspectiveWorking with the Code BaseInstrumentationFuture Directions

The Case for Testing

"... With tests we can change our code quickly and verifiably. Without them we really don't know if our code is getting better or worse. ..." - Michael Feathers

Cost of Fixing Defects

The Feedback Cycle

Testing vs. Debugging

Debugging:manual set-up (set breakpoints, step code, etc.)temporarily modifies code (printf debugging)manual verificationpay for it every time

Testing:no source code modificationautomated set-up and verificationpay once when you create the testreap the rewards every time you run it

Your Application

A Unit Test

Unit Testing

isolates a small piece of functionalityeliminates dependencies on other components through mocking / substitutionideally runs quicklyprovides automated verification

Benefits:

safety net during refactoring after refactoring becomes a regression testspeeds up the Edit-Compile-Run-Debug cycle

Impediments to Unit Testing

Main impediment: the way the application is glued together.

use LWP;...sub notify_user { my ($user, $message) = @_; ... my $ua = LWP::UserAgent->new; ... $ua->request(...); }

Strongly Coupled Concerns

sub emit_html {

}

Dependency Breaking TechniquesAdapt ParameterBreak Out Method ObjectDefinition CompletionEncapsulate Global ReferencesExtract and Override CallExtract and Override Factory MethodExtract and Override GetterExtract ImplementerExtract InterfaceIntroduce Instance DelegatorIntroduce Static SetterLink Substitution

Parameterize ConstructorParameterize MethodPrimitive ParameterPull Up FeaturePush Down DependencyReplace Function with Function PointerReplace Global Reference with GetterSubclass and Override MethodSupersede Instance VariableTemplate RedefinitionText Redefinition...

Sprouting

replace a block of code with a subroutine/method call (even for new code)

Benefits:

simple and safe code transformationcode block can be run independentlypermits the code to be redefined

Sprouting Example - DBI calls

Any DBI call is a good candidate for sprouting. DBI call = an action on a business conceptSprouting permits:

testing of SQL syntaxtesting of query operationremoval of interaction with the database

Dependency Injection

# use LWP;...sub notify_user { my ($user, $message, $ua ) = @_; ... # my $ua = LWP::UserAgent->new; ... $ua->request(...); } "Ask for - Don't create"

Preserving Signatures

use LWP;...sub notify_user { my ($user, $message, $ua ) = @_; ... $ua ||= LWP::UserAgent->new; ... $ua->request(...); }

Perl-Specific Mocking/Isolation Techniques

Replacing a Package

alter @INC to load alternate code

alter %INC to omit unneeded code

Replacing Subroutines

Monkey patch the symbol table:

no warnings 'redefine'; *Product::saveToDatabase = sub { $_[0]->{saved} = 1 }; *CORE::GLOBAL::time = sub { ... }

Create accessors for package lexicals

Modifying Object Instances

Re-bless to a subclass:

package MockTemplateEngine;our @ISA = qw(RealTemplateEngine);sub process { ... new behavior ...}...$template = RealTemplateEngine->new(...);bless $template, 'MockTemplateEngine';

Use the Stack

Define behavior based on caller(). Use in conjunction with monkey patching subs.

Exploiting Perl's Dynamicism

Use the dynamic capabilities of Perl to help you isolate code and create mock objects.Not the cleanest techniques, but they can greatly simplify getting dependency-laden code under test.

Manual Testing

Not always bad Some times it's the best option:

automated verification is difficult / impossibleautomated test to costly to create

Manual Testing Example

Using an Wrapper Layer

Real-World Experience

Writing the first test is very difficultIt gets easier!Biggest win: automated verificationNext biggest win: efficient testsManual testing is ok (save what you did!)

More Real-World Experience

Isolating code will help you understand the dependency structure of your applicationLet your tests guide refactoringTestable code ~ clean code ~ modular code

Working with the Code Base

Source Control

Get everthing under source control:perl itselfall CPAN modulesshared librariesApache, mod_perl, etc.

Reliable Builds

Automate building and deployment Break dependencies on absolute paths, port numbersEnable multiple deployments to co-exist on the same machine

Logging

make logs easy to accessimport logs into a databaseautomate a daily synopsis

SELECT count(*), substr(message, 0, 50) as "key"FROM log_messageWHERE log_time BETWEEN ... AND ...GROUP BY keyORDER BY count(*) DESC;

Wrappers

Create deployment-specific wrappers for perl and perldoc:

#!/bin/shPATH=...PERL5LIB=...LD_LIBRARY_PATH=...ORACLE_HOME=......

/path/to/app/perl "$@"

Create Tools

$ compile-check Formatter.pm

Runs app-perl -cw -MApp::PaymentService::Email::Formatter

Full module name determined from current working directory.

Automated Testing

Continuous IntegrationSmoke TestsRegression Tests

Instrumentation

Devel::NYTProf

Pros:full instrumentation of your codetiming statistics on a per-line basiscall-graphs, heat mapuseful for

Cons:big performance hit (not suitable for production)you need to drive it

Also see Aspect::Library::NYTProf for targeted profiling

Devel::Leak::Object

Original purpose: find leakage of objects due to cyclic references

Modifications:

Record how (the stack) objects get createdCount the number of each type of object createdDetermine where objects of a certain class are createdRobust facility to instrument bless and DESTROY.

dtrace

Pros:instrument events across the entire systemruns at nearly full speed - usable in production

Cons:only available for Mac OS and Solarisperl probes still under development

Future Directions

More static analysismanually added soft type assertionsmore help in editors/IDEsmore refactoring toolsbetter compilation diagnosticscatch mistyped subroutine namescatch errors in calling methods/subscatch other type errors

Future Directions

More dynamic instrumentionusable in productionlow performance hitmainly interested in:

the call chain (stack)types of variables / subroutine parameters / return values

dtrace? modified perl interpreter?

Future Directions

Automated Instrumentationbuild a database of the runtime behavior of your programcollect design details not expressed in the sourcemine the database to infer internal rules:

call graphssignature of subroutines

use rules to aid static analysis

A Refactoring Problem

Want to change $self->{PRICE} to $self->getPrice:

package Product;

sub foo { my ($self, ...) = @_; ... ... $self->{PRICE} ... ...}

A Refactoring Problem

Within package Product → easy!

package Product;

sub getPrice { ... }

sub foo { my ($self, ...) = @_; ... ... $self->getPrice ... # know type of $self ...}

A Refactoring Problem

Can we change this?

package SomeOtherClass;

sub bar { ... ... $product->{PRICE} ...}

Need to know if $product is from package Product (or a subclass)

A Refactoring Problem

Look at context for help.

package SomeOtherClass;

sub bar { ... my $product = Product->new(...); ... ... $product->{PRICE} ...}

A Refactoring Problem

Now what?

package SomeOtherClass;

sub bar { my ($self, $product, ...) = @_; ... ... $product->{PRICE} ...}

Need to figure out where SomeOtherClass::bar() gets called.

Instrumentation DB → just look up the answer

References

"Working Effectively with Legacy Code" - Michael Feathers "Clean Code Talks" - googletechtalks channel on YouTubeWriting Testable Code - Misko Hevery http://misko.hevery.com/code_reviewers_guide"Clean Code: A Handbook of Agile Software Craftsmanship," Robert C. Martin, ed.

Thanks!

Feedback, suggestions, experiences?

erantapaa@gmail.com