software and large software development

7/27/2019 Software and Large Software Development

1/55

LARGE SCALE SOFTWARE

DEVELOPMENT

IF3250 PROYEK PERANGKAT LUNAK

4 SKS

Sem I 2013/2014


2/55

Large Systems

Here in Syracuse at least two very large softwaresystems have been successfully implemented:

Over The Horizon Radar (OTHR) contained about3,000,000 lines of code, written mostly in FORTRAN withsome C and some assembly language as well.

BSY-2 submarine battle management system software is

several times larger than OTHR, written mostly in Ada.

A 5 million line system would require something like:

effortofyearspersonyeardays

xdaylines

lines947

/240

1

/22

000,000,5=


3/55

Large Systems (2)

So, to complete the project in 2.5 years would requirethe services of at least 380 well trained softwaredevelopers.

Two points are almost self evident:

a system this large must be partitioned into many rela-tivelysmall, nearly independent components in order to get it towork

it would be much better not to create such a large sys-temat all, but rather, to build most of it from reused swcomponents, reserving new code for new requirements.


4/55

SW Product Model

Architecture defines product components and allocates processing to

them

defines external product behavior

Requirements Specification describes what constitutes correct operation

it is the basis for testing and evaluation

Product Specification

defines an architecture for the system describes software design and implementation

specifies a software build process


5/55

SW Product Model (2)

Test Plan defines procedures for unit, integration, validation, qualification, and regression

testing

qualification test procedures are emphasized

Prototype Code

verifies design for critical processing, analyzes implementation problems as theyarise

Product Code

Code for each component of the product, implemented as software modules.

test stub attached to each module, used to establish basic software cycling andnearly correct operation

Test Code test drivers for unit, integration, and qualification tests

Test Report


6/55

Large Scale (LS) Software Development

Complexity in Software High LOC, High amount of Data Acces, High computation

elements, etc

Complexity in Requirements High Requirements Size, Requirements Changes, conflicting

requirements, etc

Complexity in location differences

Complexity in platform differences

Complexity in politics Different rules, policies

Complexity in Resource Allocations


7/55

Coordination in LS SW Development

The root cause is work coordination between team

developers

intra- and inter- team coordination

Major factors are Communication

Capacity

Cooperation


8/55

What should we be aware of?

Software Change

Software Configuration Management (SCM)

Change Control


9/55

Software Change Management

Environment Changes

Computer Hardware

Competition

Business Processes

External Requirements (Y2k, Euro,)

New Standards (XML, MP3, e-Speak,)

All Software Demands Change Incremental Development

Maintenance (Corrective, Perfective, Adaptive)


10/55

Source Control Management

Manage Constant Software Change

Improve Debugging Productivity

Trace Defect Reports to Correct Versions of Source Code

Understand Change History Reproducibility

Build (Compile) Software Accurately, Repeatably

Grow Project Size

Avoid Over-writing and Losing Changes Concurrent Development (Improve TTM)

Coordinate Work with Development Policies


11/55

Versioning

Revision / Version - Single copy of a source file as it

appeared at given point in its change history

(checkpoint)

File / Element - Complete change history; allversions

Branch - Set of successive versions; parallel to other

branches Delta - Change from one file version to the next


12/55

Version Tree


13/55

Multiple File Configuration Management

Variant - File based on another, co-exists with

original

Derived Object - File automatically generated from

source files Change Set - Set of versions across multiple files

making up a single logical change (e.g., feature or

fix) Baseline - Consistent set of versions across all files


14/55

Baseline

fubar.c foo.c bar.c


15/55

Coordinating Work

Problem: Multiple people working on same file atthe same time can result in lost work.

Solution 1: Serialize work with file locking.

Solution 2: Work in isolation (branch) andmerge.

Branching and merging scales much better.

Merging can be semi-automated using 3-waymerge algorithms.


16/55

Merging - Version Tree

0

1

2

1

0

Common

Ancestor

ConflictingChanges

3

Merge

Result


17/55

Formal definition (IEEE)

SCM is the process of identifying and defining the

items in the system, controlling the change of these

items throughout their life cycle, recording and

reporting the status of items and change requests

and verifying the completeness and correctness of

the items.

Related Terms: Source Control, Version

Control


18/55

Managing Change

Change Control -Formal processes for tracking andapproval of software modifications.

Managed with SCM and defect tracking tools.

Usual Policy Areas: Documentation - Change request / defect report

filed.

Approval - Change request reviewed by releasemanagement or technical board.

Quality Requirements - Code reviewed, regressiontests passed.

Policies often change by release stage and in responseto problems.


19/55

Development Process

Key Components of Development Process:

Software Configuration Management

Change Control Quality Control


20/55

Principles of Large-Scale Software

Development

Reproducibility

Policy Enforcement versus Policy Auditing

Process, Process, Process

A process must be defined to be improved

Automation

A process must be automated to be repeatable

Continuous Functional Integration and Test


21/55

Software Integration

Integration: Combining multiple features orcomponents developed in isolation.

Large projects are subject to constant change, soisolation on a stable base is necessary.

Continuous Functional Integration and Test (CFIT)

Changes integrated throughout development.

Changes not allowed to regress functionality orquality.


22/55

HP-UX Development

220,000+ Source Files

~2.5 Million Versions

52,000+ Derived Files (Single Nightly Build)

900+ User Accounts

More Than 1 Terabyte of Disk Space


23/55

HP-UX Concurrent Development

Normally 2-3 Releases Under Development

Several Releases Under Maintenance

Heavy Use of Branching and Merging


24/55

HP-UX Multisite Development

Four Primary Development Sites

Three in USA (Three Time Zones)

One in India

One USA Site Hosts Three Logical Sites Partner Companies

Several Secondary Sites


25/55

Tiered Software Integration Model

1. Developer Task - Edit, Compile, Test

2. Submit - Batch Changes, Acceptance Test3. Baseline - Nightly Build, Regression Test

4. Integration - Monthly Packaging, Partner Test

5. Release - Semi-annual Delivery, Production Use


26/55

Integration Levels

Build

Submit

Baseline

Integration Cycle

Release

Modifications


27/55

Process Evolution

Continuous Process Improvement

Root Cause Analysis (RCA)

Capability Maturity Model (CMM)1. Initial

2. Repeatable

3. Defined

4. Managed5. Optimizing


28/55

Tools


29/55

Commercial (license cost per concurrent user, approx 2-6Keach) Rational/IBM XDE Java Edition 2002/XDE .NET Edition 2002

Borlands Together (acquired TogetherSoft)

Telelogics System Architect 10 (acquired Popkin Software) Microsofts Enterprise Architect 2002

Ilogixs Rhapsody

Gentlewares Poseidon for UML

Free Tigris ArgoUML (Open Source) EclipseUML (Open Source)

StarUML (Open Source)

UML-supported CASE tools


30/55

System Architect 10

Telelogic (acquired Popkin Software in 2005)

Types of modelling supported: Business Process Modelling Object/Component Modelling

Data Modelling Structured Analysis and DesignInterface: Drawing Workspace Browser - hierarchical navigation (of diagrams and definitions) Definition dialogues Matrix EditorImport/Export Formats Comma Separated Vale (csv), XML


31/55

System Architect 10

Reporting: Proprietary, MS Word templates, HTML Generation

Object Modelling UML and old methods (Booch, OMT, etc.)

Code Generation C++ (+ reverse engineering), Java (+ reverse engineering), Visual

Basic, CORBA IDL, Javascript

Schema Generation

Data Defintion Language of SQL, e.g. Access, ORacle, etc.

GUI screen painter


32/55

Together

comprehensive, multi-language modeling platform forsoftware practitioners who design, develop, and/orcommunicate software architectures

Two versions Together 2006 Release 2 for Eclipse (3.2)and Together 2006 for Visual Studio

support for UML 2.0, OCL, XMI, Design Patterns, Java5, and CORBA/IDL

Language Support: Eclipse Version Java and C++ Visual Studio Version C# and VB.NET

Database Support SQL Server, Oracle, DB2 andSybase


33/55


34/55

Complex Softwarehttp://www.intelligent-files.com/complex-software.html

Almost everything, around and about applicationsoftware undergoing change, conspires to move iton from being justsoftware, to being complexsoftware

Complex software arises from a culmination ofmany factors.

The simplest measure of software complexityis via an

algorithm called McCabe's Cyclomatic Complexity Complex software can be defined through a

summation of distinct measures.


35/55

Bringing complex software to ground level, this is what you feel, as adeveloper working with complex software: complex software will make it hard to determine where to make a

change to add a feature or fix a bug.

complex software will cause what is called cascading changes. A simple

change suddenly requires you to change ten, twenty, even thirtysoftware files.

complex software when changed will produced many unintended sideeffects, most of these being unwanted, even destructive.

complex software will contain much duplication. This means, that formany changes, you will need to repeat yourself, sometimes many times

over. complex software is hard to maitain. Data patches, manual releases, the

lack of automated tests all add up to disempower IT maintenance teams.



36/55

Contibuting factors

the developers, their psychology, fear, communication skills, desirefor feedback.

the business. its risk appetite, rewards structure, its ability to recruit& retain.

the market. socio-economics, technology change, regulatory climate,customer mobility

the project its leadership, sponsors, its agility & budget, its pace ofdevelopment

the competition systems, their technology & responsiveness, their

differentiators the customertheir loyalty, their need for customization, their financial

disposition



37/55

Ultra Large Scale (ULS) System

Ultra-large-scale system (ULSS) is a term used in fields includingComputer Science, Software Engineering and Systems Engineering torefer to software intensive systems with unprecedented amounts ofhardware, lines of source code, numbers of users, and volumes ofdata.

The scale of these systems gives rise to many problems:they will be developed and used by many stakeholders acrossmultiple organizations, often with conflicting purposes and needs;they will be constructed from heterogeneous parts with complexdependencies and emergent properties; they will be continuouslyevolving; and software, hardware and human failures will be the

norm, not the exception.

(http://en.wikipedia.org/wiki/Ultra-large-scale_systems)
http://en.wikipedia.org/wiki/Computer_Sciencehttp://en.wikipedia.org/wiki/Software_Engineeringhttp://en.wikipedia.org/wiki/Systems_Engineeringhttp://en.wikipedia.org/wiki/Systems_Engineeringhttp://en.wikipedia.org/wiki/Software_Engineeringhttp://en.wikipedia.org/wiki/Computer_Science


38/55

Ultra Large Scale SoftwareDevelopment


39/55

The scale and complexityhttp://www.sei.cmu.edu/uls/

The scale and complexity of systems is increasing dramatically. Ultra-large-scale (ULS) systems are systems of unprecedented scale

in some of these dimensions: lines of code

amount of data stored, accessed, manipulated, and refined

number of connections and interdependencies number of hardware elements

number of computational elements

number of system purposes and user perception of these purposes

number of routine processes, interactions, and emergent behaviors

number of (overlapping) policy domains and enforceable mechanisms number of people involved in some way

ULS S Ch i i


40/55

ULS System CharacteristicsUltra-Large-Scale Systems; The Software Challenge of the Future

(Software Engineering Institute Carnegie Mellon)

Decentralization The scale of ULS systems means that they will necessarily be

decentralized in a variety of waysdecentralized data, development,evolution, and operational control.

Inherently conflicting, unknowable, and diverse requirements

ULS systems will be developed and used by a wide variety ofstakeholders with unavoidably different, conflicting, complex, andchanging needs.

Continuous evolution and deployment:

There will be an increasing need to integrate new capabilities into a ULSsystem while it is operating. New and different capabilities will be

deployed, and unused capabilities will be dropped; the system will beevolving not in phases, but continuously.


41/55

ULS System Characteristics (2)

Heterogeneous, inconsistent, and changing elements: A ULS system will not be constructed from uniform parts: there will

be some misfits, especially as the system is extended andrepaired.

Erosion of the people/system boundary

People will not just be users of a ULS system; they will beelements of the system, affecting its overall emergent behavior.

Normal failures

Software and hardware failures will be the norm rather than theexception.

New paradigms for acquisition and policy The acquisition of a ULS system will be simultaneous with the

operation of the system and require new methods for control.

ULS Ch ll


42/55

ULS ChallengesUltra-Large-Scale Systems; The Software Challenge of the Future

(Software Engineering Institute Carnegie Mellon)

Design and Evolution

Orchestration and Control

Monitoring and Assessment


43/55


44/55

ULS Challenges (3)

Orchestration and Control By orchestration we mean the set of activities needed to make the

elements of a ULS system work in reasonable harmony to ensurecontinuous satisfaction of the mission objectives. Orchestrationinvolves management and administration but at a scale well

beyond that of traditional, centralized, relatively fine-grainedcontrols. Orchestration requires a combination of up-front design,overall policy promulgation and enforcement, and real-timeadjustment of operating parameters.

Orchestrating a ULS system requires supportinginterdependencies and controlling the consequences of local

actions with respect to their effect on the emergent whole, eventhough each part of a system might be acting to maximize itslocal utility.


45/55

ULS Challenges (4)

To succeed in developing and operating ULS systems,

new knowledge, technologies, and methods in the

following areas are needed:

online modification

maintenance of quality of service

creation and execution of policies and rules

adaptation to users and contexts

enabling of user-controlled orchestration


46/55

ULS Challenges (5)

Monitoring and assessment

The effectiveness of ULS system design, evolution, and

orchestration has to be evaluated. There must be an

ability to monitor and assess ULS system state,behavior, and overall health and well being.

The criteria for success or overall health are different

for ULS systems than for smaller systems designed to

accomplish a task that does not change as the system isused.

D i i hi h ULS S i


47/55

Domains in which ULS Systems are emerging

http://www.sei.cmu.edu/uls/

Defense

The Northrop report argued that "the U.S. Department of Defense (DoD) has a goal of

information dominance this goal depends on increasingly complex systems characterized by

thousands of platforms, sensors, decision nodes, weapons, and warfighters connected through

heterogeneous wired and wireless networks. These systems will push far beyond the size of

today's systems by every measure They will be ultra-large-scale systems."[1]

Financial trading

Following the flash crash, Cliff and Northrop[2] have argued "The very high degree of

interconnectedness in the global markets means that entire trading systems, implemented and

managed separately by independent organizations, can rightfully be considered as significant

constituent entities in the larger global super-system. The sheer number of human agents

and computer systems connected within the global financial-markets system-of-systems is so

large that it is an instance of an ultra-large-scale system, and that largeness-of-scale hassignificant effects on the nature of the system". [2]
http://en.wikipedia.org/wiki/Ultra-large-scale_systemshttp://en.wikipedia.org/wiki/2010_Flash_Crashhttp://en.wikipedia.org/wiki/Dave_Cliff_(professor)http://en.wikipedia.org/wiki/Ultra-large-scale_systemshttp://en.wikipedia.org/wiki/Ultra-large-scale_systemshttp://en.wikipedia.org/wiki/Ultra-large-scale_systemshttp://en.wikipedia.org/wiki/Ultra-large-scale_systemshttp://en.wikipedia.org/wiki/Dave_Cliff_(professor)http://en.wikipedia.org/wiki/2010_Flash_Crashhttp://en.wikipedia.org/wiki/Ultra-large-scale_systems


48/55

Domains.... (2)

Healthcare

Kevin Sullivan has stated that the US healthcare system is "clearly anultra-large-scale system"[7] and that building national scale cyber-infrastructure for healthcare "demands not just a rigorous, modernsoftware and systems engineering effort, but an approach at the

cutting edge of our understanding of information processing systemsand their development and deployment in complex socio-technicalenvironments".[7]

Others

Other domains said to be seeing the rise of ultra-large-scalesystems include government, transport systems (for example airtraffic control systems), energy distribution systems (for examplesmart grids) and large enterprises.
http://en.wikipedia.org/wiki/Ultra-large-scale_systemshttp://en.wikipedia.org/wiki/Ultra-large-scale_systemshttp://en.wikipedia.org/wiki/Ultra-large-scale_systemshttp://en.wikipedia.org/wiki/Ultra-large-scale_systems


49/55

Managing Complexity

Software engineers have devised a number of fundamental tools of the trade forhelping to manage complexity [1]:

Decomposition: The most basic technique for tackling large problems is to divide them

into smaller, more manageable chunks, each of which can then be dealt with in relative

isolation (note the nearly decomposable subsystems in Figure 1). Decomposition helps

tackle complexity because it limits the designers scope.

Abstraction: The process of defining a simplified model of the system that emphasizes

some of the details or properties, while suppressing others. Again, this works because it

limits the designers scope of interest at a given time.

Organization: The process of defining and managing the interrelationships between the

various problem-solving components (note the subsystem and interaction links of Figure 1).

The ability to specify and enact organizational relationships helps designers tackle

complexity by: enabling a number of basic components to be grouped together andtreated as a higher-level unit of analysis, and providing a means of describing the high-

level relationships between various units.

[1] Booch, G. Object-Oriented Analysis and Design with Applications. Addison Wesley, 1994.

Very Large Scale Software


50/55

Very Large-Scale Software

Engineering (VLSSE)

Architectural specification (system, services (data,

protocols, processes), team/organization.

Configuration management and version control

Process automationbuilds, regression tests, etc.

Reliance on informalisms for development

(Open Source Software Development and Very Large-Scale SoftwareEngineering (Walt Scacchi Institute for Software Research University of

California, Irvine Irvine, CA, 92697-3425 USA 19 July 2005))


51/55

VLSSE (2)

Information density vs. knowledge coverage

VLSEE needs for ASB to consider: Test-beds for full-size system mock-ups (probably built from

OSS and Computer Game components)

Visualization aidsarchitecture, configuration, socio-technical networks

Empirical knowledge about existing VLSEE practices for bothOSS and CSS

Computer-supported cooperative organizational learningenvironments (CSCOLE) supporting long-term organizationaltransformation.


52/55

VLSSE - The Process

Configuration management and version control

Architectural specification and maintenance

Collaboration, leadership, control and conflict

management Community development and support

Software source code and artifact data mining

G d B h d i b t ki bi


53/55

Grady Boochs advices about making big

software development projects a success

The fundamentals never go out of style, never forget the fourfundamentals: Create crisp and resilient abstractions.

Maintain a good separation of concerns.

Create a balanced distribution of responsibilities.

Focus on simplicity. You need a regular rhythm of releases.

"Establishing that rhythm provides predictability and sustainability."Regularity lets everyone plan on what functionality or features will beinjected into the next round, regardless of release frequency.

Focus upon growing executable architectures. IT managers need to govern around the architectural decisions rather

than raw, running, naked code. "The code is the truth," Booch says. "Butthe code is not the whole truth.

Grady Boochs advices
http://en.wikipedia.org/wiki/Abstraction_(computer_science)http://en.wikipedia.org/wiki/Abstraction_(computer_science)


54/55

Grady Booch s advices...(http://www.cio.com/article/373215/5_Things_Grady_Booch_Has_Learned_About_C

omplex_Software_Systems?page=2&taxonomyId=3020)

Create social structures that encourage innovation while still preserving

predictability.

Teamwork is an essential ingredient in any large software project, so

companies should give attention to creating relationships that work.

Businesses need predictability (when will this software really ship?) and

they also want innovation (make it do something cool!).

Building a social structure to support both those goals isn't necessarily

easy, but the successful projects find a balance.

Keeping developers out of "blasted meetings" so they can get things

done. Have fun.

Successful projects come from teams that are jazzed about what they're

doing. "Most people want to build beautiful, elegant things,"


55/55

Tugas

Bagi kelompok @5 orang, dikumpul (Hardcopy) Selasa(3/9/2013)

Cari paper terkait large scale atau complex software,pelajari, dan siapkan presentasi singkatnya (utk Selasa)

Contoh: Coordination in Large-Scale Software Team

Survey answered by 775 Microsoft software engineers

Parallel Changes in Large Scale Software Development: AnObservational Study

Coodination in Large-Scale Software Development: Helpful andUnhelpful Behaviors

Defining Execution Viewpoints for a Large and ComplexSoftware-Intensive System

software and large software development

Documents