cvs reorganization, installation reorganization & a simple build system
DESCRIPTION
CVS Reorganization, Installation Reorganization & A Simple Build System. Steve Fischer October 24, 2002. Goals. Make GUS portable Easily build and configure runnable instances of our projects Avoid conflicting with the pre-existing structure of external sites Self-explanatory CVS structure - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/1.jpg)
CVS Reorganization,Installation Reorganization & A Simple Build System
Steve FischerOctober 24, 2002
![Page 2: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/2.jpg)
Goals
Make GUS portable Easily build and configure runnable instances of our projects Avoid conflicting with the pre-existing structure of external sites
Self-explanatory CVS structure Improve collaborators’ ability to understand our code structure And ours (particularly new folks)
Improved ability to release projects on a schedule Projects wait for other projects only when necessary
Easily maintain released versions while development continues Support development on a unified file system (new file server)
Don’t rely on local file systems having different installed versions Encourage a move away from individual ownership of projects
![Page 3: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/3.jpg)
CVS Reorganization
![Page 4: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/4.jpg)
CVS Structure
At the root there are projects, which have components, which have parts:
$CVSROOT/
Project1/
Component1/
bin/
lib/
Component2/
Project2/
Projectn/
parts
![Page 5: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/5.jpg)
Projects
Units of software development having their own release schedule Have their own release number, eg, GUS 3.1.2, Annotator 1.0.0
May depend on other projects Either on a specific release or ‘latest’ No circular dependencies
Dependency example: DoTS->GUS->CSP
![Page 6: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/6.jpg)
Proposed Projects (phase 1)
AllGenes Annotator (new) CSP (CBIL Style Police: web form assistance) DJob (distributed job controller) DoTS (build, including DoTS related plugins) GLE (grammars for transcription) GUS ParaDBs PlasmoDB RAD (plugins & website) (or should these be separate?)
![Page 7: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/7.jpg)
Projects Contain Components
For example:GUS/
Common/
DBAdmin
Model/ A component is usually formed from one or a couple of
package directories (eg, java packages). In addition to code, components have:
Documentation TODO list Configuration Required data
A component may depend on others within its project
![Page 8: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/8.jpg)
Proposed GUS/ Components
SeqMatch/ Common/ DBAdmin/ GenBank/ GOPredictor/ Model/ ObjRelJ/ ObjRelP/ Pipeline/ PluginMgr/ SchemaBrowser/ Util/ WebDevKit/
![Page 9: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/9.jpg)
Proposed GUS Components in Detail - 1
SeqMatch/ Blast, BLAST2, and BLAT object models
Common/ Commonly used schema dependent stuff, such as common plugins
DBAdmin/ Database admin scripts
GenBank/ Data model for genbank parser
GOPredictor/ Predict GO function
Model/ Schema definition files Hand edited objects (eg, GUS::Model::DoTS::Assembly.pm.man)
![Page 10: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/10.jpg)
Proposed GUS Components in Detail - 2
ObjRelJ/ Java object layer: superclasses
ObjRelP/ Perl object layer: superclasses and code generator
Pipeline/ Pipeline management API
PluginMgr/ GA and friends
SchemaBrowser/ Web based schema browser
Util/ Commonly used schema-independent utilities
WebDevKit/ Servlet and/or JSP based
![Page 11: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/11.jpg)
Components Contain Standard Parts
As needed:
Component1/
bin/
cgi-bin/
config/
data/
doc/
htdocs/
lib/
src/
test/
![Page 12: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/12.jpg)
Component Parts in Detail - 1
Bin executables
Cgi-bin/ Cgi executables
Config/ Properties files for the component Files typically named component.prop
Doc/ Specifications User guides TODO
![Page 13: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/13.jpg)
Component Parts in Detail - 2
Data/ Extra data needed by component, eg, matrices,
Htdocs/ Static web pages
Test/ Test data
![Page 14: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/14.jpg)
Component Parts in Detail - 3
Lib/ Contains linkable object files Perl includes full package path (Project::component::module)
lib/
perl/
Project/
component/
*.pm
xml/
*.xml
![Page 15: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/15.jpg)
Component Parts in Detail - 4
Src/ Contains source that needs to be compiled or otherwise
transformed before it can be installed
src/ c/ java/ edu/ cbil/ project/ component/ org/ gusdb/ component/
![Page 16: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/16.jpg)
Process to Migrate CVS Structure
origCVS/ GUS/ perl/ … www/ perl/ lib/ Fasta.pm CSP/
fromCVS/ GUS/ perl/ … www/ perl/ lib/ Fasta.pm CSP/
CVS checkout copy
newCVS/ DoTS/ Common/ lib/ perl/ Common/ Fasta.pm CSP/ GUS/
orig pruned
Transformscript
new
![Page 17: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/17.jpg)
Process to Migrate CVS Structure
Do transform Check out old to origCVS/ Copy to fromCVS/ Run script to transform from fromCVS/ to newCVS/
moves files (thus pruning fromCVS/) Validate transform by:
Examing transform script (see where YOUR stuff is going to be) Examine fromCVS/ to see what I left behind
Create new CVSROOT with newCVS/ (losing history)
How can I get everybody’s OK on the transform??
![Page 18: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/18.jpg)
Releases
A project is the unit of release When it is released, it is tagged in cvs with a release number,
eg 2.1.1 At that time, other projects can declare a dependency on that
particular release (discussed in detail later) Bug fixes are applied to the tagged branch
![Page 19: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/19.jpg)
Installation Reorganization
![Page 20: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/20.jpg)
Objectives
Install into a single relocatable location: $GUS_HOME Don’t conflict with the site’s pre-existing structure Easy to install and uninstall Avoid path and classpath conflicts
Be able to find (almost) all GUS installed resources there: Executables Libraries Documentation Configuration Some data Third party resources
Support multiple running instances on a machine, eg dev, beta Make explicit the versions of each included resource
In a file called $GUS_HOME/versions Also have installation targets for websites and appl. servers
![Page 21: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/21.jpg)
P1/ C1
C2
P2/ C1
C2
P3/ C1
C2
Installing Project1 to $GUS_HOME
$GUS_HOME/Bin/Lib/Doc/
= dependencies
![Page 22: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/22.jpg)
Understanding $GUS_HOME
The home for a single GUS related installation Contains one or more projects, and the projects they depend
on Including third party resources (such as BioJava or BioPerl) May contain projects that are running separately, as long as
they don’t have conflicting dependencies (May even contain the entire set of GUS projects) It contains a version file, eg:
DoTS 1.2.1
GUS 2.0.0
CSP 3.0.1
BioJava 1.1.1
![Page 23: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/23.jpg)
Sample $GUS_HOME
$GUS_HOME/ bin/ dotsbuild extractSeqs config/ dotsbuild.prop csp.prop gus.prop data/ matrices/ doc/ DoTS/Dotsbuild/TODO GUS/Model/uml/*.uml CSP/UserGuide.html (cont’d)…
![Page 24: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/24.jpg)
Sample $GUS_HOME (cont’d)
$GUS_HOME/
…
lib/
java/
gusmodel.jar
guswdk.jar
perl/
DoTS/DotsBuild/*.pm
GUS/Model/*.pm
GUS/ObjRelP/*.pm
GUS/WebDevKit/*.pm
CSP/*.pm
xml/
*.xml
![Page 25: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/25.jpg)
Installation Targets - Website
Contains cgi-bin and htdocs
$/world/www.allgenes/
cgi-bin/
*.pl
htdocs/
*.html
![Page 26: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/26.jpg)
Installation Targets – Application Server
EG, Tomcat Will copy .jar files and other stuff as needed
![Page 27: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/27.jpg)
Configuration
Code never refers to absolute file paths Many resources are conveniently located relative to
$GUS_HOME, so don’t need configuration Otherwise, code relies on property files for configuration We provide a perl and java PropertySet object to make this
easy Property files are found in… $GUS_HOME/config Named after the component that is using them:
gusmodel.prop Consider using properties instead of macro substitutions when
possible
![Page 28: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/28.jpg)
A Simple Ant-based Build System
![Page 29: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/29.jpg)
Objectives
Install projects to installation targets Serves two kinds of installs:
From $PROJECT_HOME to installation targets (developers) From .tar file to installation targets (external users)
Support inter-project dependencies Support dependencies on third party resources Handle object layer code generation Be easy to use Be relatively easy to maintain Be generic across projects and components, but also flexible
![Page 30: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/30.jpg)
$PROJECT_HOME -> $GUS_HOME (developers)
Install the build system Create $PROJECT_HOME Check out the install/ project Copy build to a local bin/
Check out the project of interest build DoTS install $GUS_HOME –co
Now $PROJECT_HOME is an image of CVS containing your project and all projects it depends on.
Edit away Install for testing
build DoTS install $GUS_HOME
![Page 31: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/31.jpg)
.tar -> $GUS_HOME (external users)
Untar the download file cd install/ setenv $GUS_HOME /usr/local/somewhere ./build DoTS install $GUS_HOME
Also works for installing to a website ./build GeneDB installweb /world/www.genedb
![Page 32: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/32.jpg)
The Mechanics
Uses Ant (http://jakarta.apache.org/ant/) Uses xml to declaratively specify the build Used extensively in the java community
Each project contains its own build.xml file The install/ directory contains the main build.xml file
Calls the appropriate project’s build.xml Houses “subroutines”
The project build.xml file Specifies the project’s dependencies on other projects Specifies the project’s component’s dependencies on each other Calls default project and component build routines, unless custom
ones supplied
![Page 33: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/33.jpg)
The Mechanics (cont’d)
Java compilation The build compiles java into the classes/ directory of a component It places .jar files into lib/java/ directory of a component Does compilation in dependency order
Copying from $PROJECT_HOME to target (eg, $GUS_HOME) Copies and merges all dirs, such as bin/, lib/ doc/ Can perform macro substitution on the way
Creates version file Because Ant is kind of slow, build can do just a component
instead of a whole project
![Page 34: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/34.jpg)
Special case: object layer code generation
Generates into: Perl:
$PROJECT_HOME/GUS/Model/lib/perl/GUS/Model/DoTS $PROJECT_HOME/GUS/Model/lib/perl/GUS/Model/Core Etc
Java: $PROJECT_HOME/GUS/Model/src/java/org/gusdb/model/dots $PROJECT_HOME/GUS/Model/src/java/org/gusdb/model/core Etc
Hand created files have .man suffix, eg: GUS/Model/DoTS/Assembly.pm.man
Only generates if files are older than “schema definition” file: $PROJECT_HOME/GUS/Model/schema/definition.sql
This file is generated by schema modification process (and lives in CVS)
![Page 35: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/35.jpg)
Making it All Happen
![Page 36: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/36.jpg)
Goals
Getting it done quickly With a minimum of disruption
![Page 37: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/37.jpg)
Overarching strategy
Work on projects and components one at a time Start with ones that are most central and not in development
GUS/ObjRelP GUS/Model/lib/perl/Model/DoTS GUS/PluginMgr GUS/Common DoTS DJob
Get that core working with new build system and $GUS_HOME In parallel, convert to GUS 3.0 In parallel, bring in GUS/ObjRelJ and Annotator (Dave) Upgrade WebDevKit install apparatus Bring in Web sites one at a time
![Page 38: CVS Reorganization, Installation Reorganization & A Simple Build System](https://reader035.vdocument.in/reader035/viewer/2022062315/56814e56550346895dbbed40/html5/thumbnails/38.jpg)
Inconveniences
During the time that a project is being upgraded: Everybody will need to commit their work to cvs Will need to maintain 2 copies in cvs:
Old New Will need to merge changes to both
Hopefully this will be short for most projects