a novel approach to automatic extraction of configuration parameters from industrial code
DESCRIPTION
A novel approach to automatic extraction of configuration parameters from industrial code. Ian Davis, Ric Holt † , Ron Mraz ‡. †. ‡. OWL Computing Technologies Inc. ( www.owlcti.com ). Provides solutions that: - PowerPoint PPT PresentationTRANSCRIPT
A novel approach to automatic extraction of
configuration parameters from industrial code
Ian Davis, Ric Holt †, Ron Mraz‡
†‡
OWL Computing Technologies Inc.
( www.owlcti.com ) Provides solutions that: Facilitate ultra-secure unidirectional data transfer
across the internet (very secure one-way firewall) Potential Customers:
Governments Military and Intelligence Power plants Water management, etc.
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 2
The Idea Two Unix machines in a single box Write only network card upstream Read only network card downstream Internal cable connects two network cards
10Mb/sec and 155Mb/sec (fiber optics) supported External connections for:
Incoming and outgoing network cable Local access to each Unix machine USB Sockets for each Unix machine
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 3
The Product Line
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 4
For VehiclesFor Personnel
An Application
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 5
The Unix Software Supports:
Transparent unidirectional TCP/UDP firewall SMTP email systems and other protocols Automatic file mirroring to/from secure machine All major historian/tracking/logging systems
Bash scripts: Facilitate software installation Initiate client/server and other executables Monitor real time behaviour of system Stop/restart services, etc
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 6
The Engineering Problem
Customers requirements vary They must change configuration variable values This results in Bash scripts being cloned
Why is this problematic Customers have to understand Bash Customers risk breaking their scripts Repairing broken scripts very costly Owl must support many variants of scripts Hard to switch from Bash to better languages
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 7
The Issues with Bash Problems
Antiquated (limited support) Cryptic language (people dislike it) Slow (Python can be 6 to 21 times faster) Memory leaks and other bugs (software failure) To configure software scripts must be edited
Solution Refactor all scripts to use external XML files Port Bash to Python/Ruby etc.
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 8
Benefits of XML configuration file Supports script migration
Massive consolidation of scripts Releasing updates to scripts much easier End users do not need to understand scripts
Intrinsic benefits: Users know exactly what they can change They don’t risk breaking the scripts Configurations documented in the XML files Can easily compare configurations XML files can be shared across scripts
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 9
Our Challenge Mission impossible:
Change ~1,000 scripts to use XML configuration The largest script ~2,000 lines long Scripts may not be copied off site You are only on site for a week
We have no clue what the scripts do: Don’t understand the contents of the scripts Don’t know what parameterizes the scripts Don’t know what parameters users may change No obvious way of reading an XML file in Bash
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 10
The First Hour Refactoring task
Manually examine first script Record how variables are used Guess which variables customer’s change Somehow change the script to read XML
Unworkable solution Very tedious work Requires constant attention to detail Task enormous Deliverable will be far from optimal
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 11
Prowl(Parameterizing Owl)
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 12
VariablePropertiesBASH
SCRIPTXMLFILE
The Solution
Hack the Bash 4.2 interpreter Open source C code is easily changed Continue to source scripts etc Change interpreter to “action” everything Remove ability to invoke external programs Augment variables with usage flags Output XML file when interpreter finishes
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 13
Dynamic Fact Extraction
x=2;
if (…test…)
then
…statements…
else
…statements…
fi
function($x,$y)
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 14
y=2; if (…test…)then returnficase $x in1) …;break;;2) …code…esac
Example XML output
y=10; x=1; y=2; z=$x+$y; exit $(($z))
<prowl>
<script1 version="default">
<default>
<x>1</x>
<y changed>2</y>
<z derived returned>1+2</z>
</default>
</script1>
</prowl>
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 15
XML Configuration File(Manually edited)
<prowl>
<script1 version="default">
<default>
<x>10</x>
<y>20</y>
</default>
</script1>
</prowl>
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 16
Look at:
Naming conventions
Documentation
Examine variable usage
Compare cloned scripts
What customers change
Prowler
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 17
VariableAssignments BASH
SCRIPTXMLFILE
Reading Configuration File
Small C program parses the XML file Optional script name Optional version name Optional external assignments (x=21 y=3)
Prowler outputs: Bash variable assignment statements
x="10";y="20"; This string then eval’d within the script
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 18
Just One Extra Line/Script
But where to place this one line ?
To early: Configuration values later get changed
To late: Configuration values are used before set
Just right How to tell ?
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 19
Validate Placement
Run prowl on the refactored Bash script Execute prowler (and only prowler) Complain about any variable named by
prowler already flagged as USED. Prowler invoked too late
Complain if any variable named by prowler is later assigned a value. Prowler invoked too early
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 20
Contributions
Automated analysis of all Bash scripts Automated construction of XML files One line solution for parsing XML files Ability to validate changes applied Removed a significant road block for OWL Existing customers were asking for this Meets new US purchasing requirements
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 21
What Was Learned
Think outside the box Necessity is the mother of invention Source code is a valuable resource
It can be exploited in novel ways Tools that make a good worker
Don’t refactor if a tool can do the work for you Getting it right saves a lot of time Getting it wrong isn’t an option
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 22
Conclusions
Software implemented in a week Only minor changes to Bash C source files Available as open source software http://www.swag.uwaterloo.ca/prowl http://cs.uwaterloo.ca/~ijdavis (CSMR paper)
Easy to port to other Bash environments [Almost] as robust as Bash
OWL thrilled Customers happy
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 23
PLE Workshop 9 Dec 2013
Fact Extraction from Bash in Support of Script Migration 24
Thank You
That is our novel approach to automatic extraction of configuration parameters
from industrial code