accounting for non functional and project requirements - cosmic and ifpug development - talmon...
TRANSCRIPT
Accoun&ng for Non-‐Func&onal and Project requirements
in so9ware project performance measurement, benchmarking and es&ma&ng:
COSMIC and IFPUG developments
IWSM/Mensura Conference, Krakow, October 2015
Talmon Ben-‐Cnaan (IFPUG), Charles Symons (COSMIC)
IWSM MENSURA Krakow, October 2015
Agenda
• IntroducJon: Why the need for standards for Non-‐FuncJonal Requirements (NFR) and for Project Requirements and Constraints (PRC)?
• The joint COSMIC/IFPUG development of a Glossary of NFR and PRC terms
• The COSMIC Guideline on how to consider NFR and PRC
• The IFPUG SoTware Non-‐FuncJonal Assessment Process (SNAP) to measure NFR
• Conclusions and next steps • QuesJons and debate
IWSM MENSURA Krakow, October 2015
The ac&vi&es of so9ware project performance measurement, benchmarking and es&ma&ng
need consistent data and terminology
Measuring project
performance
Project estimating
Benchmarking
Projectdata &
benchmark repository
Recording & measuring software system
project requirements
IWSM MENSURA Krakow, October 2015
Func&onal size measurements are used consistently across all ac&vi&es
• There are three types of requirements for a soTware system project
• FuncJonal User Requirements (FUR) • Non-‐funcJonal Requirements (NFR) • Project Requirements and Constraints (PRC)
• FuncJonal size measurement methods, e.g. the COSMIC or IFPUG methods, are used consistently to measure the size of FUR across all acJviJes
But what about NFR and PRC?
IWSM MENSURA Krakow, October 2015
There is no good exis&ng defini&on of Non-‐Func&onal Requirements (NFR)
Example: ISO 24756 definiJon 1): “A so%ware requirement that describes not what the so%ware will do but how the so%ware will do it. Example: so%ware performance requirements, so%ware external interface requirements, so%ware design constraints, and so%ware quality a>ributes.” Comment: only ‘design constraints’ define ‘how the soTware will do it’
IWSM MENSURA Krakow, October 2015
Next, each ac&vity has defined its own data and terminology for NFR and PRC 2)
Requirements Recording (50)
IEEE 804, ISO 9126 Wikipedia
Requirements Sizing (36)
VAF, TCA, SNAP
Benchmarking (48)
ISBSG, SEI
Project Estimating (39)
COCOMO II
One common NFR (= “Interfaces”)
IWSM MENSURA Krakow, October 2015
Past surveys have found varying numbers of NFR terms
• ‘at least’ 161 terms (Chung et al 3))
• 122 terms in a structured hierarchy (Saito et al 4))
• 108 terms (Symons 2))
• ISO/IEC SQuaRE 5) standard lists 32 Quality terms
A complete, standard list of NFR and PRC terms may be impossible
IWSM MENSURA Krakow, October 2015
Past aSempts to measure a collec&ve size of NFR are now discredited
Albrecht’s 10 components of the ‘Value Adjustment Factor’ 6)
14 components (IFPUG) 7)
19+ components (MkII FPA) 8)
….. but they did have value when they were first invented.
IWSM MENSURA Krakow, October 2015
Conclusions
• There is no common understanding in the soTware industry of what is meant by NFR
• ExisJng standards for NFR and PRC are not coherent • These issues handicap acceptance of methods for:
• quanJfying requirements, • developing project performance benchmarks, • project esJmaJng.
These were the drivers for COSMIC and IFPUG to develop the joint Glossary of NFR and PRC terms
IWSM MENSURA Krakow, October 2015
Agenda
• IntroducJon: Why the need for standards for Non-‐funcJonal Requirements (NFR) and for Project Requirements and Constraints (PRC)?
• The joint COSMIC/IFPUG development of a Glossary of NFR and PRC terms
• The COSMIC Guideline on how to consider NFR and PRC
• The IFPUG SoTware Non-‐FuncJonal Assessment Process (SNAP) to measure NFR
• Conclusions and next steps • QuesJons and debate
IWSM MENSURA Krakow, October 2015
Words
What do we mean when saying • A date is a date.
• Right!
• It is not my type
IWSM MENSURA Krakow, October 2015
We have agreed many aspects of NFR and PRC over the past year
• Scope of the Glossary 9) • Clarity on:
• ‘Requirements’ versus ‘constraints’ • The ‘things’ that NFR and PRC apply • DisJnguishing and defining NFR and PRC
• Structuring NFR (aligned with SQuaRE) and PRC to classes à groups à terms
• The terms: • 67 NFR terms, mainly from ISO and ISBSG (27 COSMIC/IFPUG)
• 19 PRC terms, mainly from PMI and ISBSG
IWSM MENSURA Krakow, October 2015
We have clarified the terms Requirements and Constraints
Thesaurus: Requirement: “a thing demanded or obligatory” Constraint: “limitaJon or restricJon”
The difference is not always clear: • A requirement that the so%ware shall use C# is
also a constraint • ‘Latency’ can be a requirement for some real-‐Fme
systems but a constraint for the Mars Rover system
• Staffing skills can be a project requirement or a constraint
Constraints
Requirements
We use requirements for convenience. We use constraints only when it is helpful to disAnguish constraints from requirements.
IWSM MENSURA Krakow, October 2015
The scope of requirements and constraints
Requirements and constraints can apply to • The soCware; • The data maintained or used by the soTware; • The technology to be used, e.g. the planorms; • Other deliverables, e.g. documentaJon or training; • The combined hardware/soCware system, e.g. a response Jme;
and some constraints come from the environment
IWSM MENSURA Krakow, October 2015
Data
Technology
Hardware/Software System
Project
Environment
Other Deliverables
Software
Project, System, Product
A project: a temporary endeavor to achieve defined objecJves of delivering a product by defined dates.
Following a Project, there is a Product in place (A new Product or a Product that was changed by the project). A product is a hardware/ soTware system or an item of soTware such as a soTware package
Product A
Product C
Product B
IWSM MENSURA Krakow, October 2015
Requirements for a SoTware System Project
Project Requirements and Constraints (PRC)
System and SoTware Non-‐FuncJonal
Requirements (NFR)
SoTware FuncJonal User
Requirements (FUR)
System Environment Requirements
Technical Requirements
Quality Requirements
(SoTware System)
Quality Requirements
(Data)
Classifica&on of Requirements
IWSM MENSURA Krakow, October 2015
System and SoTware Non-‐FuncJonal Requirements (NFR)
System Environment Requirements
Technical Requirements
Quality Requirements
(SoTware System)
Quality Requirements
(Data)
Classifica&on of NFR
1 Quality of the data maintained by the soTware
2 System performance 3 CompaJbility 4 Ease of use 5 System reliability 6 Control of access 7 Maintainability 8 Ease of deployment 9 System or soTware architecture or design
1 Context 2 ApplicaJon Domain 3 ImplementaJons 4 User Base
1 OperaJonal Planorm 2 Database 3 OperaJonal Planorm
constraints 4 Development
requirements
Class
Group
Type
IWSM MENSURA Krakow, October 2015
COSMIC and IFPUG defini&ons
FuncAonal User Requirements (FUR) -‐ ISO/IEC 14143-‐1 DefiniAon
Adopted by COSMIC and IFPUG A sub-‐set of the user requirements. Requirements that describe what the soCware shall do, in terms of tasks and services.
NOTE: FuncJonal User Requirements relate to but are not limited to:
• data transfer (for example Input customer data; Send control signal) • data transformaJon (for example Calculate bank interest; Derive
average temperature) • data storage (for example Store customer order; Record ambient
temperature over Jme) • data retrieval (for example List current employees; Retrieve latest
aircraT posiJon)
IWSM MENSURA Krakow, October 2015
COSMIC and IFPUG defini&ons
Non-functional requirement – COSMIC and IFPUG Definition Any requirement for a soCware system or for a soCware product, including how it should be developed and maintained, and how it should perform in operaAon, except any funcAonal user requirement for the soCware. Non-‐funcJonal requirements concern: • the soTware system or soTware product quality; • the environment in which the soTware system or soTware product
must be implemented and which it must serve; • the processes and technology to be used to develop and maintain the
soTware system or soTware product, and the technology to be used for their execuJon.
IWSM MENSURA Krakow, October 2015
COSMIC and IFPUG defini&ons
Project Requirements and Constraints - Definition Requirements that define how a soCware system project should be managed and resourced, or constraints that affect its performance. Requirements may include:
• The targets the project should achieve (e.g. budget, delivery date, product quality);
• The project management processes that should be used; • How the project should be governed and resourced.
Constraints may include: • LimitaJons on the project resources; • Dependencies on other projects outside the control of the project concerned.
IWSM MENSURA Krakow, October 2015
67 Terms
Examples
COSMIC / IFPUG DefiniJon
NFR sub-‐type (Quality, System Environment Requirements, Technical Requirements) Quality: 9 groups
Environment: 4 groups Technical: 4 groups
ISO DefiniJon
IWSM MENSURA Krakow, October 2015
COSMIC and IFPUG strongly recommend their joint Glossary
• Developed through > 20 iteraJons over a year • Reviewed by many experts (academic and pracJJoners) from around the world
• Available for free download from: • www.cosmic-‐sizing.org • www.ifpug.org
• We welcome feedback and comments
IWSM MENSURA Krakow, October 2015
Agenda
• IntroducJon: Why the need for standards for Non-‐funcJonal Requirements (NFR) and for Project Requirements and Constraints (PRC)?
• The joint COSMIC/IFPUG development of a Glossary of NFR and PRC terms
• The COSMIC Guideline on how to consider NFR and PRC
• The IFPUG SoTware Non-‐FuncJonal Assessment Process (SNAP) to measure NFR
• Conclusions and next steps • QuesJons and debate
IWSM MENSURA Krakow, October 2015
The COSMIC Guideline builds on the joint COSMIC/IFPUG Glossary
Contents • Terminology, NFR/PRC definiJons
• ClassificaJon of NFR/PRC terms • Glossary of NFR/PRC terms • EvoluJon of NFR in a project • How to deal with NFR/PRC in performance measurement, benchmarking & esJmaJng
• Measurement of NFR (interim)
Glossary9) ü ü ü
Guideline10) ü ü ü ü ü
ü
IWSM MENSURA Krakow, October 2015
Requirements that first appear as NFR may evolve, wholly or partly, into FUR
Outline Funct- -ional
Requts.
Outline NFR
Project Requts. & Constraints
Require- ments
Analysis
Definition & Design
Build, Test &
Implement
Architecture
Approx. Funct- -ional
Requts.
Detailed NFR
Imple- mented software system
or software product
Detailed FUR
…… as demonstrated by Al Sarayreh, Abran and others, e.g. [11]
IWSM MENSURA Krakow, October 2015
The ‘addi&onal’ FUR can be measured by approximate or standard COSMIC sizing
Outline Funct- -ional
Requts.
Outline NFR
Project Requts. & Constraints
Require- ments
Analysis
Definition & Design
Build, Test &
Implement
Architecture
Imple- mented software system
or software product
Approx. Funct- -ional
Requts.
Detailed NFR
Detailed FUR
Size by analogy or expert judgement
Approx. COSMIC size measurement
Precise COSMIC size measurement
IWSM MENSURA Krakow, October 2015
The Guideline gives examples for all Quality NFR how they may evolve as a
project progresses
FuncAonality that may evolve from the NFR, that can be measured
‘Middleware’ funcJonality to enable portability across mulJple DBMS or OS or hardware
Graphical User Interface funcJons; and FuncJonality to assist users, e.g. ‘Help’ funcJons
FuncJonality to import external data in real-‐Jme so that it is available for immediate use
IniAal NFR term
Portability
Usability
Response Jme
‘Residual’ or ‘Real’ NFR statement
The specific environments across which the soTware must be portable
The specific usability requirements (e.g. ‘must be usable by public with no training’)
-‐ The response Jme target; -‐ Specific hardware; -‐ Low-‐level programming language.
Examples:
IWSM MENSURA Krakow, October 2015
COSMIC has no plans to develop a collec&ve size measure for NFR
Inherent difficulJes:
1. How to form a collecJve size measure that will add pracJcal value, long-‐term?
2. The large number and variety of NFR
3. How to collect effort data related to NFR when NFR evolve into FUR?
IWSM MENSURA Krakow, October 2015
1. Collec&ve size measures are common and o9en valuable
Examples of valuable collecJve size measures. (All are mathemaJcally invalid, but they work.) • The IFPUG Value Adjustment Factor (when first introduced) • Indices that affect stock market senJment, e.g. the PMI • Measures of ‘biological age’ • Total Factor ProducJvity
CondiJons for a usable collecJve size measure: • A finite, stable, well-‐defined set of components • If possible, a common unit of measure for all components • A simple formula for the index (avoid complex maths)
IWSM MENSURA Krakow, October 2015
2. There is a large number and variety of NFR. Many overlap in meaning
How many separate NFR should we include in a collecJve size measure? (10, 14, 19+, 67, 108, 122, 161+ …?)
Maintain-‐ ability
Test-‐ ability
Flex-‐ ibility
Adapt-‐ ability
Extend-‐ ability
Evolv-‐ ability
Modula-‐ rity
Modifia-‐ ability
Reus-‐ ability
Port-‐ ability
IWSM MENSURA Krakow, October 2015
3. How to capture the effort associated with implemen&ng NFR?
…. given that NFR evolve as a project progresses into FUR ….. ?
.... & noJng that effort to implement NFR varies with:
• soTware domain, • type of NFR, • the parJcular ‘soluJon’ for the NFR, • technology, re-‐use, etc.
IWSM MENSURA Krakow, October 2015
Instead, we propose basic sets of NFR/PRC to record and measure for performance
measurement, benchmarking, es&ma&ng
NFR and PRC Class
Terms
Quality NFR Response time, Transaction rate Security, Privacy Maintainability, Reusability Interfaces, Operational processing mode
System Environment NFR
Application type, sub-type Implementations (numbers of) Maximum number of concurrent users
Technical NFR Operational platform type Programming language DBMS software
Project Requirements and Constraints (PRC)
Many, but not all of the 19 project requirements and constraints are worth recording. Example: if staff experience levels, processes and methods and tools are the same for all projects, then they need not be recorded for internal purposes.
IWSM MENSURA Krakow, October 2015
…. and advise how to deal with NFR in performance measurement, benchmarking and es&ma&ng
• EsJmate and measure total funcJonal size, including funcJonality that evolves from NFR
• Within a defined ‘environment’, i.e. • soTware domain (business, real-‐Jme) • soTware architecture, technology • project type (new, enhancement, etc)
…. collect and record data on ‘real’ NFR and PRC so that you can make like-‐for-‐like comparisons (This is normal pracJce for benchmarking acJviJes, plus a few addiJonal parameters.)
IWSM MENSURA Krakow, October 2015
Guideline Conclusions
The COSMIC ‘Guideline for dealing with Non-‐FuncFonal Requirements in project performance measurement & esFmaFng’ • builds on the joint COSMIC/IFPUG Glossary • provides pracJcal advice on how to deal with manageable sub-‐sets of NFR and PRC
and will be available in October for free download from www.cosmic-‐sizing.org
IWSM MENSURA Krakow, October 2015
Agenda
• IntroducJon: Why the need for standards for Non-‐funcJonal Requirements (NFR) and for Project Requirements and Constraints (PRC)?
• The joint COSMIC/IFPUG development of a Glossary of NFR and PRC terms
• The COSMIC Guideline on how to consider NFR and PRC
• The IFPUG SoTware Non-‐FuncJonal Assessment Process (SNAP) to measure NFR
• Conclusions and next steps • QuesJons and debate
IWSM MENSURA Krakow, October 2015
FPA and SNAP covers functional and non-functional requirements
PRC – do not impact soTware size
NFR – measured with SNAP Points
FUR – measured with FP
Guidelines to prevent doubled counJng
IWSM MENSURA Krakow, October 2015
PRC affects produc&vity and not size
• Effort depends on PRC classes. No standard / industry mathemaJcal formula
• Examples: Project type; (same size different effort) • Effort may be formulated for some PRC (“producJvity drivers”)
• Examples: Skill level; LocaJon; Schedule compression (“crashing”) (same size different effort)
• Effort (or deviaJon from esJmaJon) can be explained by some PRC
• Examples: Risk that was materialized; scope creep
IWSM MENSURA Krakow, October 2015
SNAP
• SNAP idenJfies fourteen non-‐funcJonal characterisJcs (“sub categories”) that classify the way NFR are met.
• In each sub-‐category, the size is assessed within a counJng unit (SCU). The SCU is part of the definiJon of the sub-‐category.
• Non funcJonal size is determined by a set of parameters / complexity levels. The parameters that define each sub-‐category answer the quesJon “what factors affect the effort to deliver a NFR”
• Example: The effort to add input methods (e.g. barcode reader, fax server, reading CSV file) with same funcJonality depends on number of added input methods
IWSM MENSURA Krakow, October 2015
SNAP
• Parameters can be used as a table of values (e.g. “High, medium, low”, “<10, 10 to 15, 16+”) or as a mulJplier (“SP = 3*# addiJonal input methods), or a combinaJon of the above
• Once the parameters are assessed, the size of the SCU is calculated. SP size is the sum of the SPs of the SCUs
IWSM MENSURA Krakow, October 2015
SNAP sub categories are independent of the way NFR are classified / defined
Accuracy
Performance
Ease of use / customer satisfaction
Response Time
Requ
iremen
ts
CharacterisJcs
IWSM MENSURA Krakow, October 2015
SNAP sub-‐categories are independent of the way NFR are classified
Accuracy
Ease of use / customer satisfaction
Response Time
ISO
912
6 ISO
250
10
CO
SM
IC /
IFP
UG
Glo
ssar
y
Requ
iremen
ts
CharacterisJcs
Time Behavior
Performance
IWSM MENSURA Krakow, October 2015
Data Func&ons and Transac&onal Func&ons are not sufficient to size NFR
The effort to deliver NFR is also affected by • Database ac&vi&es • Dealing with UI proper&es • Extensive Logical / Mathema&cal opera&ons • Batch processes that do not cross the boundaries and are not exposed to users
• Mul&ple inputs/output • …..
IWSM MENSURA Krakow, October 2015
Example 1 – The requirements
NFR: Improve CRM system performance for “Retrieve and View Customer Details“-‐ by loading the ‘customer details’ screen in 3 seconds or less. (Currently it takes 0.5 sec to 6 sec.) Problem analysis: The system is slow when the user (call center representaJve) opens a big customer with many products and many interacJons. It also happens when users’ virtual memory is low
IWSM MENSURA Krakow, October 2015
Example 1 -‐ the design
1. Increase the virtual memory of ‘Windows’ to maximum; 2. Create a database view, which includes data from three tables
(Assigned Products, TransacJons, and Payments) (Assuming 30 different DETs are fetched).
3. Add Logic to idenJfy which customers will be loaded to the new view
4. Add a field to ‘Customers’ table to indicate whether recent informaJon moved to the new view and add logic to idenJfy which transacJons will be copied (not seen by the users, hence non-‐funcJonal change).
5. A batch process runs in the background every 2 hours, refreshing this view with large and most acJve users
IWSM MENSURA Krakow, October 2015
Example 1 – SNAP count
• Increasing virtual memory: InstallaJon of a larger size virtual memory is a hardware configuraJon and not a soTware change. It is not counted using soTware sizing methods.
• CreaJng a database view (CUST_DETAIL_SUMMARY_ TEMP) and adding an indicator (field) to ‘Customer’ table:
• Count SP using 3.2 Database changes; SNAP count is based on two database changes, 20-‐50 DETs, 2-‐5 RETs
• Adding a batch process: (this batch job does not cross the boundary, hence it is not counted using FP)
• Count SP using 3.3 Batch Processes. SNAP count is based on 30 DETs and 3 FTRs
• Adding logic to meet the NFR: Count SP using 1.2 Logical and MathemaJcal OperaJons. SNAP count is based on 3 FTRs, type = ‘Logical’ and 10 DETs used for the added logic
IWSM MENSURA Krakow, October 2015
IFPUG Coun&ng Process
Gather available documentaJon
Determine counJng purpose, scope, boundaries and
parJJons
IdenJfy requirements as FUR, NFR. Separate mixed requirements to FUR and NFR
Measure Data FuncJons
Measure TransacJonal FuncJons
Associate NFR to sub-‐categories
Calculate funcJonal size
Determine SNAP size of each sub
category
Calculate SNAP size
Document and report
FuncJonal
Non-‐FuncJonal
IWSM MENSURA Krakow, October 2015
Es&ma&on and benchmark
Es&ma&on 𝑬𝒇𝒇𝒐𝒓𝒕=
[𝑷𝑫𝑹↓𝟏 (𝑷𝒓𝒐𝒋𝒆𝒄𝒕↑′ 𝒔 𝑪𝒉𝒂𝒓𝒂𝒄𝒕𝒆𝒓𝒊𝒔𝒕𝒊𝒄𝒔)×𝑭𝒖𝒏𝒄𝒕𝒊𝒐𝒏𝒂𝒍 𝑺𝒊𝒛𝒆]+[ 𝑷𝑫𝑹↓𝟐 (𝑷𝒓𝒐𝒋𝒆𝒄𝒕↑′ 𝒔 𝑪𝒉𝒂𝒓𝒂𝒄𝒕𝒆𝒓𝒊𝒔𝒕𝒊𝒄𝒔)×𝑵𝒐𝒏−𝒇𝒖𝒏𝒄𝒕𝒊𝒐𝒏𝒂𝒍 𝑺𝒊𝒛𝒆]
PDR – ProducJvity Delivery Rate Note: Feedback from users show that PDR2 diverges in 3 of the 14 sub-‐categories. NSFFC will reformulate them based on collected data Benchmarking • Based on linear regression to extract PDR1 and PDR2: • Two different baselines, two business values to customers
IWSM MENSURA Krakow, October 2015
Agenda
• IntroducJon: Why the need for standards for Non-‐funcJonal Requirements (NFR) and for Project Requirements and Constraints (PRC)?
• The joint COSMIC/IFPUG development of a Glossary of NFR and PRC terms
• The COSMIC Guideline on how to consider NFR and PRC
• The IFPUG SoTware Non-‐FuncJonal Assessment Process (SNAP) to measure NFR
• Conclusions and next steps • QuesJons and debate
IWSM MENSURA Krakow, October 2015
COSMIC and IFPUG have come closer over the past year
Close exchange of ideas has led to: • Agreeing to disJnguish NFR and PRC • AdopJng PMI definiJons for PRC terms • Many refinements to the classificaJon structure and Glossary contents – through > 20 iteraAons
Further, we agree that: • NFR increase the size of the soTware, and should be measured
• Performance measurement, benchmarking and project esJmaJng must consider both FUR and NFR
IWSM MENSURA Krakow, October 2015
Conclusions
Whilst COSMIC & IFPUG sJll have different views on how to measure NFR, we have collaborated very well to agree:
• DefiniJons and classificaJon of NFR and PRC • The Glossary of NFR and PRC terms, v1.0
We strongly recommend these to the soTware
engineering community
IWSM MENSURA Krakow, October 2015
Next steps
• Enhance the Glossary to take into account ‘Data Quality’ requirements
• Enhance the Glossary (and define measurements?) to take into account ‘Other Deliverables’ such as training and documentaJon This is a quesJon to YOU: Should we?????
Thank you for your a^enAon
Talmon Ben-‐Cnaan (www.ifpug.org );
Charles Symons (www.cosmic-‐sizing.org ); [email protected]
IWSM MENSURA Krakow, October 2015
References
1. ISO/IEC/IEEE 24765:2010 ‘Systems and soTware engineering – vocabulary’. 2. Symons, C.R., ‘AccounJng for Non-‐FuncJonal Requirements in ProducJvity Measurement, Benchmarking and
EsJmaJng’, UKSMA/COSMIC InternaJonal Conference on SoTware Metrics & EsJmaJng, London, 27/28 October 2011.
3. L. Chung, B. Nixon, E. Yu, and J. Mylopoulos, "Non-‐funcJonal Requirements in SoTware Engineering,“ Kluwer Academic Publishing, 2000.
4. Saito, Y., Monden A., Matsumoto K., ‘EvaluaJon of non-‐funcJonal requirements in a request for proposal (RFP)’, Nara InsJtute of Science & Technology, Japan, at InternaJonal Workshop on SoTware Measurement (IWSM), Nara, 2012..
5. ISO/IEC FCD 25010: Systems and soTware engineering, –System and soTware product Quality Requirements and EvaluaJon (SQuaRE) –System and soTware quality models
6. Albrecht, A. J., ‘Measuring applicaJon development producJvity’, In Proc. of the IBM ApplicaJons Development Symposium, pp. 83-‐-‐92. Monterey, California (1979)
7. IFPUG, CounJng PracJces Manual, Release 4.3.1 – Appendix C
8. Symons, C.R., SoTware sizing & esJmaJng: Mk II FPA’, John Wiley & Sons, 1991, ISBN 0 471 92985 9
9. ‘Glossary of terms for Non-‐FuncJonal Requirements and Project Requirements used in soTware project performance measurement, benchmarking and esJmaJng’, Version 1.0, September 2015
10. ‘Guideline on Non-‐FuncJonal & Project Requirements: How to consider non-‐funcJonal and project requirements in soTware project performance measurement, benchmarking and esJmaJng’, version 1.0, October 2015
11. Khalid T. Al-‐Sarayreh, Alain Abran and Juan J. Cuadrado-‐Gallego, ‘A Standards-‐based model of system maintainability requirements’, Journal of SoTware: EvoluJon and Process, 2013, Vol. 25, no. 5, pp. 459-‐505