september 6, 2007open source software practice overview of the software process bill lorensen ge...
Post on 26-Dec-2015
214 Views
Preview:
TRANSCRIPT
September 6, 2007Open Source Software Practice
Overview of the Software Overview of the Software Process Process
Bill LorensenBill LorensenGE Research (retired)GE Research (retired)bill.lorensen@gmail.combill.lorensen@gmail.com
September 6, 2007Open Source Software Practice
Course WikiCourse Wikihttp://public.kitware.com/OpenSourceSoftwarePractice http://public.kitware.com/OpenSourceSoftwarePractice
September 6, 2007Open Source Software Practice
OutlineOutline
MotivationMotivation The ProcessThe Process Case Study – Insight ToolkitCase Study – Insight Toolkit Lessons LearnedLessons Learned
September 6, 2007Open Source Software Practice
A Personal Look BackA Personal Look Back
1976- NASA’s Computer Software 1976- NASA’s Computer Software Management and Information System Management and Information System (COSMIC)(COSMIC)
1978 - MOVIE.BYU1978 - MOVIE.BYU 1984 - GE Research Workstation1984 - GE Research Workstation 1984 - LYMB1984 - LYMB 1994 - Visualization Toolkit1994 - Visualization Toolkit 2000 - Insight Toolkit2000 - Insight Toolkit
September 6, 2007Open Source Software Practice
Flattener #4Flattener #4Open SourcingOpen Sourcing
Self-organizing collaborative Self-organizing collaborative communitiescommunities
September 6, 2007Open Source Software Practice
Open Source Anecdote *Open Source Anecdote *
IBM Manager:IBM Manager: “So, walk me through the development “So, walk me through the development process for e-commerce. What’s the underlying web process for e-commerce. What’s the underlying web server?”server?”
Developer:Developer: “It’s built on top of Apache.” “It’s built on top of Apache.”Manager:Manager: “Apache?” “Apache?”Developer:Developer: “A shareware program for web server “A shareware program for web server
technology, produced for free by a bunch of geeks technology, produced for free by a bunch of geeks working online in some open-source chat room.”working online in some open-source chat room.”
Manager:Manager: “How do you buy it?” “How do you buy it?”Developer:Developer: “You download it off a Web site for free.” “You download it off a Web site for free.”Manager:Manager: “Who supports it if something goes wrong?” “Who supports it if something goes wrong?”Developer:Developer: “I don’t know, it just works!” “I don’t know, it just works!”
*from The World is Flat
September 6, 2007Open Source Software Practice
Successful Open Source Successful Open Source ProjectsProjects
September 6, 2007Open Source Software Practice
Visualization Toolkit - vtkVisualization Toolkit - vtk
Open source toolkit for scientific visualization, Open source toolkit for scientific visualization, computer graphics, and image processingcomputer graphics, and image processing
www.vtk.orgwww.vtk.org
September 6, 2007Open Source Software Practice
Computer Vision Library - vxlComputer Vision Library - vxl Based on Target jr and Based on Target jr and
Image Understanding Image Understanding EnvironmentEnvironment
NumericsNumerics ImagingImaging GeometryGeometry Camera modelsCamera models
vxl.sourceforge.netvxl.sourceforge.net
September 6, 2007Open Source Software Practice
SlicerSlicer
Developed by Brigham and Womens Surgical Planning Developed by Brigham and Womens Surgical Planning LabLab
User interface plus plug-ins for applicationsUser interface plus plug-ins for applications SegmentationSegmentation RegistrationRegistration Image GuidanceImage Guidance
http://www.slicer.orghttp://www.slicer.org
September 6, 2007Open Source Software Practice
SCIRunSCIRun
http://www.sci.utah.eduhttp://www.sci.utah.edu
Developed by U Utah SCI InstituteDeveloped by U Utah SCI Institute Computational WorkbenchComputational Workbench Visual ProgrammingVisual Programming Modeling, Simulation and VisualizationModeling, Simulation and Visualization
September 6, 2007Open Source Software Practice
National Library of MedicineNational Library of MedicineSegmentation and Registration ToolkitSegmentation and Registration Toolkit
$14 million over 6 years$14 million over 6 yearsLeading edge algorithmsLeading edge algorithmsOpen Source SoftwareOpen Source Software
www.itk.orgwww.itk.org
September 6, 2007Open Source Software Practice
Open Source Menu for SuccessOpen Source Menu for Success
A Community with a common visionA Community with a common vision A pool of talented and motivated A pool of talented and motivated
developers/scientistsdevelopers/scientists A mix of academic and commercialA mix of academic and commercial An organized, light weight approach to An organized, light weight approach to
software developmentsoftware development A leadership structureA leadership structure CommunicationCommunication A business modelA business model
September 6, 2007Open Source Software Practice
Project StagesProject Stages
StartupStartup EarlyEarly MiddleMiddle MatureMature
September 6, 2007Open Source Software Practice
Project Stages - StartupProject Stages - Startup
Define goals and/or requirementsDefine goals and/or requirements Select developer communitySelect developer community Define developer rolesDefine developer roles Select host serverSelect host server
Roll your ownRoll your own SourceForge, CollabNet, GForgeSourceForge, CollabNet, GForge
Select software development toolsSelect software development tools Revision control systemRevision control system CompilersCompilers Code coverageCode coverage Dynamic analysisDynamic analysis Build systemBuild system Testing systemTesting system
Select an open source licenseSelect an open source license
September 6, 2007Open Source Software Practice
Project Stages - EarlyProject Stages - Early
Refine toolsRefine tools Change ASAP if necessaryChange ASAP if necessary
Establish coding guidelinesEstablish coding guidelines Setup communication channelsSetup communication channels
Mailing listsMailing lists DevelopersDevelopers UsersUsers
Establish web presenceEstablish web presence Project home pagesProject home pages Wiki for developersWiki for developers
Establish nightly build/testEstablish nightly build/test
September 6, 2007Open Source Software Practice
Project Stages - MiddleProject Stages - Middle
Establish release mechanismsEstablish release mechanisms Establish policy to add additional developersEstablish policy to add additional developers Establish backward compatibility policyEstablish backward compatibility policy Establish procedure to add new featuresEstablish procedure to add new features MarketingMarketing
PapersPapers TutorialsTutorials Web buzzWeb buzz
September 6, 2007Open Source Software Practice
Project Stages - MatureProject Stages - Mature
Project has an installed baseProject has an installed base Everything should be hard to changeEverything should be hard to change
CodeCode ToolsTools ProcessProcess
September 6, 2007Open Source Software Practice
What is itk?What is itk?
A common Application Programmers Interface (API).A common Application Programmers Interface (API). A framework for software developmentA framework for software development A toolkit for registration and segmentationA toolkit for registration and segmentation An Open Source resource for future researchAn Open Source resource for future research
A validation model for segmentation and registration.A validation model for segmentation and registration. A framework for validation developmentA framework for validation development Assistance for algorithm designersAssistance for algorithm designers A seed repository for validation case studiesA seed repository for validation case studies
September 6, 2007Open Source Software Practice
Insight - Open Source ProductsInsight - Open Source Products
September 6, 2007Open Source Software Practice
itk by the Numbersitk by the Numbers March 2000March 2000
First code checkinFirst code checkin 16001600
# of nightly builds# of nightly builds 12001200
# tests run nightly# tests run nightly 5050
# of platforms# of platforms 730730
# of classes# of classes 20002000
# of files with code# of files with code
400K400K # of lines of code# of lines of code
100K100K # of lines of test code# of lines of test code
45K45K # of lines of examples# of lines of examples
160K160K # of lines of Applications# of lines of Applications
> 300> 300 # weekly t-cons# weekly t-cons
8585 # unique developers# unique developers
September 6, 2007Open Source Software Practice
itk by the Numbersitk by the Numbers 186186
# of subscribers to the developers mailing list# of subscribers to the developers mailing list 900900
# of subscribers to the users mailing list# of subscribers to the users mailing list 350350
# of monthly posts to users-list# of monthly posts to users-list
September 6, 2007Open Source Software Practice
A Common VisionA Common Vision
Create a Create a dynamicdynamic, , self-sustainingself-sustaining, , public public domaindomain and and extensibleextensible toolkit that will toolkit that will
empower researchers throughout the world to empower researchers throughout the world to develop new segmentation and registration develop new segmentation and registration algorithms and create new applications that algorithms and create new applications that leverage the NLM’s investment in the Visible leverage the NLM’s investment in the Visible
Human Male and Female data setsHuman Male and Female data sets
September 6, 2007Open Source Software Practice
The Original TeamThe Original Team
Six prime contractorsSix prime contractors IndustrialIndustrial
Kitware (Schroeder)Kitware (Schroeder) Insightful (Ng)Insightful (Ng)
UPenn (Gee)UPenn (Gee) GE Research (Lorensen)GE Research (Lorensen)
Brigham and Womens (Kikinis)Brigham and Womens (Kikinis) AcademicAcademic
UNC (Aylward)UNC (Aylward) Pitt (Stetton)Pitt (Stetton)
Utah (Whitaker)Utah (Whitaker) Rutgers (Metaxas)Rutgers (Metaxas)
Columbia (Imielenski)Columbia (Imielenski) UPenn (Udupa)UPenn (Udupa)
September 6, 2007Open Source Software Practice
Insight ConsortiumInsight Consortium
A competitive process selected the A competitive process selected the development teamdevelopment team
Each group was evaluated individually, Each group was evaluated individually, without regard to how they might integrate without regard to how they might integrate with each otherwith each other
Each team had strengths in software Each team had strengths in software development, validation or algorithm development, validation or algorithm development, but no one group had the development, but no one group had the necessary skills to do the entire projectnecessary skills to do the entire project
Distributed software developers with varying Distributed software developers with varying software engineering experiencesoftware engineering experience
September 6, 2007Open Source Software Practice
Insight Software ArchitectureInsight Software Architecture
Object-oriented designObject-oriented design Generic ProgrammingGeneric Programming Design PatternsDesign Patterns FrameworksFrameworks Separation of Algorithms from InterfacesSeparation of Algorithms from Interfaces
September 6, 2007Open Source Software Practice
Object-Oriented DesignObject-Oriented Design
Dominated software systems throughout the Dominated software systems throughout the 1990’s1990’s
Continues to be the accepted software Continues to be the accepted software design techniquedesign technique
Particularly useful for dealing with complexityParticularly useful for dealing with complexity Provides programmatic abstractions to deal Provides programmatic abstractions to deal
with generalization and encapsulationwith generalization and encapsulation C++ and Java have mechanisms to support C++ and Java have mechanisms to support
OODOOD
September 6, 2007Open Source Software Practice
Generic ProgrammingGeneric Programming
Organize libraries consisting of generic—Organize libraries consisting of generic—or reusable—software components. or reusable—software components.
The essential ideas of generic The essential ideas of generic programming are containers to hold data, programming are containers to hold data, iterators to access the data, and generic iterators to access the data, and generic algorithms that use containers and algorithms that use containers and iterators to create efficient, fundamental iterators to create efficient, fundamental algorithms.algorithms.
ITK uses generic programming to ITK uses generic programming to process n-dimensional “images”.process n-dimensional “images”.
September 6, 2007Open Source Software Practice
Design PatternsDesign Patterns
Good object-oriented software systems Good object-oriented software systems have recurring designs (patterns) that have recurring designs (patterns) that occur frequentlyoccur frequently
ITK employs a number of powerful design ITK employs a number of powerful design patternspatterns object factoriesobject factories command/observercommand/observer smart pointer memory managementsmart pointer memory management
September 6, 2007Open Source Software Practice
FrameworksFrameworks
Define how a group of participants can be Define how a group of participants can be put together to solve a particular task. put together to solve a particular task.
Particularly suitable for describing Particularly suitable for describing complex flows or algorithms that have a complex flows or algorithms that have a number of steps that can be variednumber of steps that can be varied
ITK FrameworksITK Frameworks A demand-driven data processing A demand-driven data processing
pipeline that connects algorithms to pipeline that connects algorithms to process n-dimensional image dataprocess n-dimensional image data
Registration frameworkRegistration framework Level-set frameworkLevel-set framework
September 6, 2007Open Source Software Practice
Separation of Algorithms from Separation of Algorithms from InterfacesInterfaces
Implement the algorithms with a clear Implement the algorithms with a clear separation from the applications and separation from the applications and especially the user interfaces.especially the user interfaces.
Uses the Command/Observer design Uses the Command/Observer design pattern that permits applications to watch pattern that permits applications to watch for significant events during the execution for significant events during the execution of an algorithmof an algorithm
ITK has no built-in visualization, but has ITK has no built-in visualization, but has been interfaced to several systems been interfaced to several systems including 3D Slicer, Analyze, SciRun and including 3D Slicer, Analyze, SciRun and Volview.Volview.
September 6, 2007Open Source Software Practice
In search of a new software In search of a new software development processdevelopment process
September 6, 2007Open Source Software Practice
A Light Weight Software A Light Weight Software Engineering ProcessEngineering Process
Based on the new Extreme Programming processBased on the new Extreme Programming process High intensity design, test, implement cycleHigh intensity design, test, implement cycle Supported with web-enabled toolsSupported with web-enabled tools Automated testing integrated with the software Automated testing integrated with the software
developmentdevelopment
September 6, 2007Open Source Software Practice
Extreme ProgrammingExtreme Programming
Compresses the standard analyze, design, Compresses the standard analyze, design, implement, test cycle into a continuous implement, test cycle into a continuous processprocess
September 6, 2007Open Source Software Practice
A process supported by a suite of A process supported by a suite of portable, open source toolsportable, open source tools
Apache, perl, phpApache, perl, php Web servicesWeb services
cvs, subversioncvs, subversion Revision controlRevision control
MediaWikiMediaWiki Collaborative content capture Collaborative content capture
DoxygenDoxygen Automated documentationAutomated documentation
CMakeCMake Cross-platform program buildCross-platform program build
DartDart Continuous and distributed test reportingContinuous and distributed test reporting
September 6, 2007Open Source Software Practice
Extreme ProgrammingExtreme Programming
The community owns the codeThe community owns the code
Although the identity of the original author is Although the identity of the original author is kept, other developers are free to correct kept, other developers are free to correct defects and enhance each other's codedefects and enhance each other's code
In the end, all of the software should appear In the end, all of the software should appear as though one author wrote itas though one author wrote it
September 6, 2007Open Source Software Practice
Extreme ProgrammingExtreme Programming
Release early, release oftenRelease early, release often
Although developers are tempted to keep their code Although developers are tempted to keep their code under wraps until it is perfect, the process under wraps until it is perfect, the process encourages them to release their code as soon as it encourages them to release their code as soon as it passes some minimum testspasses some minimum testsThe longer the code is visible to the community, the The longer the code is visible to the community, the better integrated it will bebetter integrated it will be
September 6, 2007Open Source Software Practice
Extreme ProgrammingExtreme Programming
Continuous integrationContinuous integration
There is no scheduled porting to computer platformsThere is no scheduled porting to computer platformsAll new software builds supported platforms every All new software builds supported platforms every eveningevening
September 6, 2007Open Source Software Practice
Extreme ProgrammingExtreme Programming
All developers agree to keep the software All developers agree to keep the software defect freedefect free
Although everyone is encouraged to submit their Although everyone is encouraged to submit their code early, the code must compile and pass tests code early, the code must compile and pass tests nightlynightlyA continuous build process sends e-mails to A continuous build process sends e-mails to developers who check in code that does not compiledevelopers who check in code that does not compileMore effectively, the community enforces the More effectively, the community enforces the commitment though peer pressurecommitment though peer pressure
September 6, 2007Open Source Software Practice
Insight - Development CyclesInsight - Development Cycles
Daily – dashboardDaily – dashboard Weekly – telephone conferencesWeekly – telephone conferences Periodic – architecture reviewsPeriodic – architecture reviews Quarterly – developer meetingsQuarterly – developer meetings Yearly – work assignmentsYearly – work assignments
September 6, 2007Open Source Software Practice
Extreme ProgrammingExtreme ProgrammingDaily Testing Is The KeyDaily Testing Is The Key
Testing anchors and drives the development Testing anchors and drives the development process (Dart)process (Dart)
Opens up the development process to Opens up the development process to everyoneeveryone
Developers monitor the testing dashboard Developers monitor the testing dashboard constantlyconstantly
Problems are identified and fixed immediatelyProblems are identified and fixed immediately Developers receive e-mail if theyDevelopers receive e-mail if they
“Break the Build”“Break the Build”
September 6, 2007Open Source Software Practice
How DART Enables How DART Enables CollaborationCollaboration
CVS
Results posted on web(the dashboard)
CVS maintainssource code
revisions DART compilessource code, runs tests
Developers review results
Developers check-in code
September 6, 2007Open Source Software Practice
Continuous System MonitoringContinuous System Monitoring
September 6, 2007Open Source Software Practice
Insight ToolkitInsight ToolkitLessons LearnedLessons Learned
September 6, 2007Open Source Software Practice
Lessons Learned: The GoodLessons Learned: The Good
Good mix of commercial and academicGood mix of commercial and academic Communication is criticalCommunication is critical The daily rhythm of Extreme TestingThe daily rhythm of Extreme Testing The Whole >>> Sum of the partsThe Whole >>> Sum of the parts Process automation reduces the burden on Process automation reduces the burden on
developersdevelopers The process is open and repeatableThe process is open and repeatable
September 6, 2007Open Source Software Practice
Lessons Learned: The BadLessons Learned: The BadCommunicationCommunication
Early in the project, communication was limited to the Early in the project, communication was limited to the mailing list and the Quarterly meetings. Almost a year mailing list and the Quarterly meetings. Almost a year passed before we added the weekly telephone passed before we added the weekly telephone conferences. Developers felt isolated and the impersonal conferences. Developers felt isolated and the impersonal electronic communication methods failed to build the electronic communication methods failed to build the personal relationships required for any software personal relationships required for any software development process.development process.
SolutionSolution Weekly developer t-consWeekly developer t-cons WikiWiki
September 6, 2007Open Source Software Practice
Lessons Learned: The BadLessons Learned: The BadLack of Design ReviewsLack of Design Reviews
The lightweight process has its drawbacks. The current The lightweight process has its drawbacks. The current process emphasizes coding and testing. This process is process emphasizes coding and testing. This process is well defined and benefits from automation. But, the well defined and benefits from automation. But, the design process is not well supported by the current design process is not well supported by the current mechanisms.mechanisms.
SolutionSolution Insight Journal review processInsight Journal review process
September 6, 2007Open Source Software Practice
Lessons Learned: The BadLessons Learned: The Bad
Unstable Application Programming Interfaces (API’s)Unstable Application Programming Interfaces (API’s) The nightly test/build was so effective in empowering The nightly test/build was so effective in empowering
programmers to make changes, that API changes programmers to make changes, that API changes occurred too frequently without the necessary buy-in occurred too frequently without the necessary buy-in from the development community.from the development community.
SolutionSolution Established backward compatibility policyEstablished backward compatibility policy
September 6, 2007Open Source Software Practice
Insight Observations: The BadInsight Observations: The BadInsufficient AutomationInsufficient Automation
Automation is critical to maintain consistent style Automation is critical to maintain consistent style and robust software. Although the project and robust software. Although the project provided many automatic techniques to catch provided many automatic techniques to catch developer errors, the project would benefit from developer errors, the project would benefit from more automation.more automation.
SolutionSolution KWStyle – tool to do style checkingKWStyle – tool to do style checking
September 6, 2007Open Source Software Practice
Insight Observations: The BadInsight Observations: The BadComplexity and Feature CreepComplexity and Feature Creep
Like most software development projects, this Like most software development projects, this software suffers from complexity and feature software suffers from complexity and feature creep. Early designs and implementations used creep. Early designs and implementations used many features of the C++ language to an many features of the C++ language to an “extreme”. The team realized and began removing “extreme”. The team realized and began removing the complexity.the complexity.
SolutionSolution Ongoing reviewsOngoing reviews
September 6, 2007Open Source Software Practice
Lessons Learned: The UglyLessons Learned: The Ugly Still need a stickStill need a stick
Someone needs to monitor the daily Someone needs to monitor the daily process.process.
XP is not everyone’s cup of teaXP is not everyone’s cup of tea Almost all bought into the process. A few Almost all bought into the process. A few
tolerated it.tolerated it.
September 6, 2007Open Source Software Practice
Open Source ResourcesOpen Source Resources http://www.opensource.orghttp://www.opensource.org http://sourceforge.nethttp://sourceforge.net http://gforge.orghttp://gforge.org
http://www.itk.orghttp://www.itk.org http://www.vtk.orghttp://www.vtk.org http://vxl.sourceforge.nethttp://vxl.sourceforge.net http://www.slicer.orghttp://www.slicer.org http://www.sci.utah.eduhttp://www.sci.utah.edu
http://www.google.com/codesearchhttp://www.google.com/codesearch http://ohloh.nethttp://ohloh.net
September 6, 2007Open Source Software Practice
Quiz Monday – Reading ListQuiz Monday – Reading List Free Culture, Lawrence Lessig, Free Culture, Lawrence Lessig,
Available online at: Available online at: http://www.free-culture.cc/freecontenthttp://www.free-culture.cc/freecontent// Sections to read Sections to read
Introduction Introduction Chapter Four Chapter Four
Free Software Free Society, Richard Stallman Free Software Free Society, Richard Stallman Available online at Available online at
http://www.gnu.org/doc/book13.htmlhttp://www.gnu.org/doc/book13.html (HTML) (HTML) http://www.gnu.org/philosophy/fsfs/rms-essays.pdfhttp://www.gnu.org/philosophy/fsfs/rms-essays.pdf (PDF) (PDF)
Sections to read Sections to read Chapter One: "The GNU Project" Chapter One: "The GNU Project" Chapter Two: "The GNU Manifesto" Chapter Two: "The GNU Manifesto" Chapter Three: "Free Software Definition" Chapter Three: "Free Software Definition" Chapter Four: "Why software should not have owners" Chapter Four: "Why software should not have owners"
Open Source Software Licensing, Lawrence Rosen Open Source Software Licensing, Lawrence Rosen Available online at: http://www.rosenlaw.com/oslbook.htm Available online at: http://www.rosenlaw.com/oslbook.htm Sections to read Sections to read
Chapter Two: "Intellectual Property" Chapter Two: "Intellectual Property" Chapter Four: "Taxonomy of Licenses" Chapter Four: "Taxonomy of Licenses"
Retrieved from "http://public.kitware.com/OpenSourceSoftwarePractice/index.php/Intellectual_Monopolies_Reading_Assignement_I"Retrieved from "http://public.kitware.com/OpenSourceSoftwarePractice/index.php/Intellectual_Monopolies_Reading_Assignement_I"
top related