evaluation, selection, and implementation of free/libre/open-source software

8/14/2019 Evaluation, Selection, and Implementation of Free/Libre/Open-Source Software

1/27

Overview

Process

Criteria

Resources

Evaluating, Selecting, and Implementing

Free/Libre/Open-Source Software

Marko Schtz

Department of Mathematical SciencesUniversity of Puerto Rico at Mayagez

Mayagez, PR

07 May 2009

Marko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
http://goforward/http://find/http://goback/


2/27

Overview

Process

Criteria

Resources

Outline

1 Overview

2 Process

3 Criteria

4 Resources



3/27

Overview

Process

Criteria

Resources

Workshop Structure

presentation of background

independent work

review



4/27

Overview

Process

Criteria

Resources

Introduction

the problem

> FLOSS sought to fill some need

> wealth of FLOSS available: 100,000s of projects

stages of the process

identify candidate

review

compare

analyze

implement

requirements

criteria resources supporting the process

> existing evaluation frameworks

> project documentation templatesMarko Schtz Evaluating, Selecting, and Implementing Free/Libre/Open-Source


5/27

Overview

Process

Criteria

Resources

Stages

involve user- and developer community

clarify and document requirements

identify candidates

review candidates

shortlist and prepare comparison

analyze results implement


O i


6/27

Overview

Process

Criteria

Resources

Clarify and Document Requirements

elicit

> user interviews> user surveys> brainstorming sessions

FLOSS development differs from proprietary development

> FLOSS: do one thing right and inter-operate> proprietary: cover many features, use lock-in (data formats)

to make users stay within application

> formulate neutral requirements

By e.g. requiring a single application for all identified needs

many possible FLOSS combinations will be ruled out

document use cases


O i


7/27

Overview

Process

Criteria

Resources

Identify Candidates

perform search on the Internet, including specialized sites:

http://sourceforge.net, http://directory.fsf.org,

http://freshmeat.net, http://debian.org,http://icewalkers.com, http://cpan.org.

research individual features/requirements to find

candidates

use name of well-known proprietary software with the termopen source on google


Overview
http://sourceforge.net/http://directory.fsf.org/http://freshmeat.net/http://debian.org/http://icewalkers.com/http://cpan.org/http://cpan.org/http://icewalkers.com/http://debian.org/http://freshmeat.net/http://directory.fsf.org/http://sourceforge.net/http://goforward/http://find/http://goback/


8/27

Overview

Process

Criteria

Resources

Identify Candidates (cont.)

use existing lists

> A MITRE 2003 report lists 115 FLOSS generallyrecognized as safe

> The Generally Recognized As Mature (GRAM) contains39 FLOSS

> Les Trophes du Libre> Qualification and Selection of Open Source Software> sourceforge.net project of the month, community choice

awards> problem with all these lists: outdated

- can only be used as indicators


Overview
http://www.terrybollinger.com/dodfoss/dodfoss_html/index.htmlhttp://www.dwheeler.com/gram.htmlhttp://www.trophees-du-libre.org/http://www.qsos.org/http://sourceforge.net/community/index.php/potm/http://sourceforge.net/community/index.php/potm/http://www.qsos.org/http://www.trophees-du-libre.org/http://www.dwheeler.com/gram.htmlhttp://www.terrybollinger.com/dodfoss/dodfoss_html/index.htmlhttp://goforward/http://find/http://goback/


9/27

Overview

Process

Criteria

Resources

Review Candidates

use existing reviews and comparisons

mailing lists user and developer documentation

project statistics

code review

collect findings


Overview


10/27

Overview

Process

Criteria

Resources

Shortlist and Prepare Comparison

weed out using must-have, must-not-have requirements,

e.g. must have unencrypted, standards-based backup

format

feed back findings to user community

feed back findings to developer community

goal: less than a handful solutions (might consist of several

programs)


Overview


11/27

Overview

Process

Criteria

Resources

Analyze Results

document lessons learned in previous stages

weigh and evaluate criteria> absolute> relative ranking

obtain approval from intended users, administration,

management, customer


Overview


12/27

Overview

Process

Criteria

Resources

Implement

invite some early adopters

prototype alongside current solution collect and maintain automated black-box tests for most/all

use cases

allow for ample transition time

encourage constant feedback from users/stake-holders


Overview


13/27

Process

Criteria

Resources

Criteria

often criteria are competing

> example: code size vs number of features

kinds of criteria

> requirements

> project community

> project source code

> project evolution> deliverables> trials


Overview


14/27

Process

Criteria

Resources

Community

developers (number of, background of)

conferences

project activity (recent/sustained)

some projects have activity bursts and times where they

fall asleep

size, organization

leadership mentoring

GSoC, etc


Overview


15/27

Process

Criteria

Resources

Source Code

technologies

> portability hidden issues: Java may be less portable thane.g. C

> teams existing competence in those technologies

> teams desire, capability to learn these technologies> extends to e.g.

- programming language- database system- desktop environment- communications protocols

How well is the implementation documented?

code size

> is an expenditure!!! bigger is worse, not better> all that code has to be maintained


Overview


16/27

Process

Criteria

Resources

Project Evolution

How long does the project exist?

history

> How has it proceeded since then?> Have there been prolonged phases of inactivity?> How has developer community evolved?> Major obstacles/challenges faced?

maturity

> Is it based on a tried concept, architecture?

> What are the revisions it went through?> Will be different for different kinds of applications> Installations

- What are the showcase deployments/installations?- How many installations are there?


Overview

P


17/27

Process

Criteria

Resources

Deliverables

usability

quality

security

feature list: breadth/focus

documentation


Overview

P


18/27

Process

Criteria

Resources

Deliverables: Usability

often misused:

> ease of installation, initial startup: valuable for casual, newusers

> frequent/power users need growth path, e.g.:

- scripting- external control- plug-ins

setup prerequisites

> which are shared with other software that is likely to beused? e.g. requires KDE, requires PostgreSQL.Amortization.

> done once, updates should be rare. Amortize over usernumber, usage time.

> estimated time to update to future versions


Overview

Process


19/27

Process

Criteria

Resources

Deliverables: Quality

minor releases & point/patch releases

> criteria for version number increase> developer oriented might rely of code repository

open bugs

> bugs need to be reported> how many reporters?> what kind of bugs are reported?> assignment of appropriate priority> bug-fixing sprints held> avoidance of regressions> reactivity to security relevant bugs> security relevant bugs relative to software class> what is considered critical?> as with most criteria here: absolute numbers mean little


Overview

Process


20/27

Process

Criteria

Resources

Deliverables: Security

security vulnerabilities

> how many> how critical> over what period

out-of-box configuration secure?


OverviewProcess


21/27

Process

Criteria

Resources

Deliverables: Feature List

the project might be too broad then

> risk the team doea not develop sufficient expertise in onethe individual features

> one of the most successful and longlived approaches hasbeen the UNIX approach of small tools that do one thing,one thing only and that do that one thing right.


OverviewProcess


22/27

Process

Criteria

Resources

Trials

form an early adoption team: a group of users who start

using the software in real productive work: must be users

who want to use the new software

early adoption team needs to feed back as much as

possible on their experiences, issues, problems, questions,

surprises, etc

very carfully listen to early adoption team


OverviewProcess


23/27

Process

Criteria

Resources

Business Readiness Rating (BRR)

http://www.openbrr.org

evaluation methodology

provides

> template> some sample evaluations


OverviewProcess
http://www.openbrr.org/http://www.openbrr.org/http://goforward/http://find/http://goback/


24/27

Criteria

Resources

Qualification and Selection of Open Source Software

(QSOS)

http://www.qsos.org

evaluation methodology

provides

> template> browser-based software to work with templates> sample evaluations


OverviewProcess
http://www.qsos.org/http://www.qsos.org/http://goforward/http://find/http://goback/


25/27

Criteria

Resources

ReadySET

http://readyset.tigris.org

project documentation library

provides

> XHTML-based forms for software-engineering projects


OverviewProcess
http://readyset.tigris.org/http://readyset.tigris.org/http://goforward/http://find/http://goback/


26/27

Criteria

Resources

Discussion

BRR and QSOS are often too rigid, e.g.:

> QSOS only allows three values per criterion: no, partial, full> BRR and QSOS strive for evaluation of criteria independent

deployment context- e.g. QSOSs books available- cannot be compensated with assignment of weights

> both can be applied with regard to deployment contextworst case: loss of comparability

ReadySET sometimes too software-development oriented

strip out parts too software-development oriented


OverviewProcess


27/27

Criteria

Resources

Independent Work

pick domain you know something about e.g.

> database systems>

photo management software> course management software> statistics software> data visualization> enterprise resource planning

develop a realistic scenario use the process described


evaluation, selection, and implementation of free/libre/open-source software

Documents