Porting A Perl/Sybase Application to Model 204

Chuck GaitCSC

Agenda

• Goal of the Task

• Perl

• The Process

• Lessons Learned

Goal of the Task

• Move the existing Sybase database part of the application to M-204, AND

• Maintain the current Sybase code base, AND

• Change as little of the code as possible.

Goal, cont'd

• Why?– Task broken up into 2 parts (short term and near

long term); we're in Phase 1– Phase 2 will involve a complete rewrite (to get

rid of duplicate and spaghetti code)– Access Restriction:

• Two separate sets of users, on two separate networks need access to the same data.

Goal, cont'd

• So– we need to have a single code base to be able to

access both databases.

• But, why have two separate databases since both contain the same data?– Because the security people say so, that's why.

Application Specs

• User GUI– 28,000 lines of Perl code (packages and

scripts)– Approx. 1000 lines of Java code– About 10,000 lines of shell programs– About 200 lines of Sybase code

• ISQL• Dblib

– Not much static HTML (100-200 lines)

Application Specs, cont'd

• Input Side– About 10,000 lines of Perl

• Processes incoming message traffic:– Scans the data, auto-correcting where it can– Reformats certain text portions into formatted data– Stores the messages via Sybase

– 200 or so lines of Sybase

• If we get to this prior to July, great! Or, not.

Perl

• Phil's Eclectric Rubbish Lister

• AKA CGI programming

• The workhorse behind most web processing

• Very Fast and Powerful

• Many different ways to do the same thing (we saw this all over)

• Can be obfuscated, uh, obtuse

More Perl

• @name means an array (context)– $name is a list context for above array

• $name[0], $name[1], ..., $name[n]

• %a is a hash context (i.e, an associative array)– $a{b} is the b-th element in the “a” hash

• Perl is “way cool”

The Process

• Looked at the application– Pulled out “all” of the Sybase code– Had a meeting with the M-204 shop

• Tasked them with writing a SQL parser– Our job was done!

The Process, 2

• M-204 shop revolted!

• Had another meeting

• Result:– Pseudo UL interface for M-204– 3 Perl packages on the front end:

• SYBX• DBX• SWAPI

SYBX

• A Perl package

• Replaced all ISQL with Dblib calls

• Returned data in the agreed-upon format

DBX

• A Perl package

• Software switch:– If server is SYBASE

• Send SQL request to SYBX

– Otherwise• Send SQL request to SWAPI

SWAPI

• Steve's WRANGLER API

• A Perl package– Translates SQL into Pseudo UL

• Uses regular expressions

– Sends PUL to M204– Retrieves output

• Postprocesses it, if necessary (aggregates, etc)

– Returns data in agreed-upon format

Perl and Regexp

$reSelect = "select\\s+(?:(distinct)\\s+)?"

. "($reSelColList|\\*)"

. "(?:\\s+from\\s+($reIdList))"

. "(?:\\s+where\\s+($reComplex))?"

. "(?:\\s+($reGroupBy))?"

. "(?:\\s+($reOrderBy))?";

Perl & Regexp, part 2

select\s+(?:(distinct)\s+)?((?:(?:|)\s*,\s*)*(?:|)|\*)(?:\s+from\s+((:(?:\b[A-Za-z]\w*(?:\.[A-Za-z]\w*)?\s*,\s*)*\b[A-Za-z]\w*(?:\.[A-Zaz]\w*)?)))(?:\s+where\s+((?:\s+(?:and|or)s+*(?:nt\s+)?(?:|(?:"(?:.*?[^\\])?"|'(?:.*?[^\\])?'|\b-?\d+}\b-?\d+(?:\.\d*)?|\b[A-Za-z]\w*(?:\.[A-Za-z]\w*)?|\*)\s+(?:not\s+)?between\s+(?:"(?:.*?[^\\])?"|'(?:.*?[^\\])?'|\b-?\d+}\b-?\d+(?:\.\d*)?|\b[A-Za-z]\w*(?:\.[A-Za

-z]\w*)?|\*)\s+and\s+(?:"(?:.*?[^\\])?"|'(?:.*?[^\\])?'|\b-?\d+}\b-?\d+(?:\.\d*)?|\b[A-Za-z]\w*(?:\.[A-Za-z]\w*)?|\*)|(?:"(?:.*?[^\\])?"|'(?:.*?[^\\])?'|\b-?\d+

}\b-?\d+(?:\.\d*)?|\b[A-Za-z]\w*(?:\.[A-Za-z]\w*)?|\*)(?:\s+not\s+)?\s*(?:like|<>|<|<=|!=|=|>=|>)\s)*(?:"(?:.*?[^\\])?"|'(?:.*?[^\\])?'|\b-?\d+}\b-?\d+(?:\.\d*)

?|\b[A-Za-z]\w*(?:\.[A-Za-z]\w*)?|\*)|(?:"(?:.*?[^\\])?"|'(?:.*?[^\\])?'|\b-?\d+

}\b-?\d+(?:\.\d*)?|\b[A-Za-z]\w*(?:\.[A-Za-z]\w*)?|\*)\s+(?:not\s+)?in\s*\(\s*(?

:(?:"(?:.*?[^\\])?"|'(?:.*?[^\\])?'|\b-?\d+}\b-?\d+(?:\.\d*)?|\b[A-Za-z]\w*(?:\.

[A-Za-z]\w*)?|\*)\s*,\s*)*(?:"(?:.*?[^\\])?"|'(?:.*?[^\\])?'|\b-?\d+}\b-?\d+(?:\

.\d*)?|\b[A-Za-z]\w*(?:\.[A-Za-z]\w*)?|\*)\s*\)|(?:"(?:.*?[^\\])?"|'(?:.*?[^\\])

?'|\b-?\d+}\b-?\d+(?:\.\d*)?|\b[A-Za-z]\w*(?:\.[A-Za-z]\w*)?|\*) \s+is\s+(?:not\

s+)?null)))?(?:\s+((?:\bgroup\s+by\s+(?:(?:\b[A-Za-z]\w*(?:\.[A-Za-z]\w*)?\s*,\s

*)*\b[A-Za-z]\w*(?:\.[A-Za-z]\w*)?))))?(?:\s+((?:\border\s+by\s+(?:(?:\b[A-Za-z]

\w*(?:\.[A-Za-z]\w*)?|\b-?\d+)(?:\s+(?:asc|desc))?\s*,\s*)*(?:\b[A-Za-z]\w*(?:\.

[A-Za-z]\w*)?|\b-?\d+)(?:\s+(?:asc|desc))?)))?

Pseudo User Language

• A select turns into GET

• An update turns into RPL

• A delete turns into DEL

PUL

• select fld1,fld2 from tbl1 where fld1=10– becomes

• GET~TBL1~FLD1|FLD2~FLD1=10~

• select * from tbl1,tbl2 where fld1=10 and tbl1.x=tbl2.x– Becomes

• XREF~TBL1|TBL2~FLD1|FLD2~FLD1=10~x~

The Returned Data

• Data coming back from both databases had to be identical

• That's why we converted all ISQL calls to Dblib– Some code changes required.

The Result

• No change for the user– Application looks identical for both databases– Great for the user since no retraining is needed.

Lessons Learned

• Was a lot more difficult than originally estimated.

• Missed some SQL that was generated on the fly

• Should have just bit the bullet and did the complete rewrite.