university of groningen understanding and supporting ... · architectural decisions in practice and...

University of Groningen

Understanding and Supporting Software Architectural DecisionsTofan, Dan

IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.

Document VersionPublisher's PDF, also known as Version of record

Publication date:2015

Link to publication in University of Groningen/UMCG research database

Citation for published version (APA):Tofan, D. (2015). Understanding and Supporting Software Architectural Decisions: for ReducingArchitectural Knowledge Vaporization. [Groningen]: University of Groningen.

CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).

Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.

Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.

Download date: 23-04-2020

https://www.rug.nl/research/portal/en/publications/understanding-and-supporting-software-architectural-decisions(29d4f043-5066-49dd-a500-ce4842c12e2b).html



500438-L-sub01-bw-Tofan500438-L-sub01-bw-Tofan500438-L-sub01-bw-Tofan500438-L-sub01-bw-Tofan

Chapter 3

Architectural Decisions in Practice

Based on: Tofan, D., Galster, M., and Avgeriou, P., Difficulty of Architectural

Decisions – a Survey with Professional Architects. In Proceedings of the 7th

European Conference on Software Architecture . 2013.

In this chapter, we investigate characteristics of architectural decisions, as

well as factors that contribute to the difficulty of decisions. We compared

characteristics of decisions and difficulty factors for junior and senior

architects. Furthermore, we studied if decisions with good outcomes (as

perceived by the architects) have different characteristics and different

difficulty factors than decisions with bad outcomes (i.e. bad outcomes, as

perceived by the architects). We performed a survey with 43 architects who

described 43 good and 43 bad decisions from their industrial practice .

Characteristics of decisions include the time taken to make a decision and the

number of decision makers. Also, we learnt that dependencies between

decisions and the effort required to analyze decisions are major factors that

contribute to the difficulty of architectural decisions. We found that good

decisions tend to consider more alternatives than bad decisions.


34 3. Architectural Decisions in Practice

3.1 Introduction

Increased understanding of architectural decisions in practice enables

researchers to propose approaches that help practitioners improve architectural

decision making and therefore reduce overall architectural knowledge

vaporization. Although there is much interest in the community on

architectural decisions, little work has been done towards a deeper

understanding of architectural decisions in practice. This chapter offers a step

in this direction, by focusing on understanding two aspects: characteristics of

architectural decisions in practice and factors that contribute to the difficulty of

making decisions. As presented in Chapter 1, architectural decisions are

particularly difficult decisions.

Using GQM (Basili and Caldiera, 1994), the goal of this study is to analyze

architectural decisions, for the purpose of understanding architectural

decisions in practice, from the perspective of software architects, in the context

of real-world projects.

We refine this goal into four research questions.

RQ1. What are the characteristics of architectural decisions?

By defining measurable characteristics of architectural decisions, we can

perform comparisons between decisions. Based on literature (Kruchten, 2008),

our experience and discussions with practitioners, we define the following

basic characteristics of decisions as metrics for RQ1.

1. Actual (i.e. while making the decision) and elapsed (i.e. actual plus

various interruptions) time spent making a decision. For example, an

architect spent five hours (actual time) over three days (elapsed time)

to make a decision.

2. Number of people directly (i.e. decision makers) and indirectly (i.e.

influenced the decision, but did not make the decision) involved in

decision making.

3. Number of alternatives considered in the beginning and later over an

extended period of time during the decision making process.

4. Number of quality attributes considered for a decision.


3.2. Related Work 35

Since the difficulty of a decision is a major aspect of decision quality (Yates et

al., 2003), we propose RQ2. Answering RQ2 will help researchers and

architects understand the factors that increase difficulty of decisions.

RQ2. What factors make architectural decisions difficult?

To answer RQ2, we defined a list of factors using literature and discussions

with experts (Section 3.3.1). The resulting metrics were 22 factors (Table 3.4).

These were used as input for our survey and survey participants rated them on

a Likert scale.

The level of experience of architects influences their decision making (van

Heesch and Avgeriou, 2010; van Heesch and Avgeriou, 2011). We propose

RQ3 to investigate how difficulty and characteristics of decisions vary with the

level of experience. This helps researchers propose targeted solutions to

address the difficulties perceived by either junior architects or experienced

architects.

RQ3. What are the differences (on difficulty and characteristics of decisions)

between junior and senior software architects?

Besides difficulty, decision outcome is the other major aspect of decision

quality (Yates et al., 2003). We investigate the differences between decisions

with a more preferable outcome (i.e. good decisions) and decisions with a less

preferable outcome (i.e. bad decisions). Answering RQ4 highlights

characteristics and difficulty factors linked to good and bad outcome of

architectural decisions.

RQ4. What are the differences between good and bad architectural decisions?

To answer the research questions, we conducted a survey with software

architects in industry. Section 3.2 presents related work. Section 3.3 presents

survey design. Section 3.4 contains the analysis of results which are further

discussed in Section 3.5. Section 3.6 presents conclusions.

3.2 Related Work

Work related to this chapter is discussed for each research question. We start

with the research question on characteristics of architectural decisions (RQ1).

We could not find any study that focuses on identifying or describing

characteristics of architectural decisions. Based on observations of various

architecture teams, Kruchten recommends roles, responsibilities and time



allocation for architects (Kruchten, 2008). Based on (Kruchten, 2008), we

consider that involved roles, involved responsibilities (e.g. involved directly or

indirectly in the decision making) and the time allocated (e.g. actual, elapsed)

for making decisions are characteristics of architectural decisions.

Some surveys with practitioners provides insights on architectural decision

making (Clerc et al., 2007), and knowledge sharing for architectural decisions

(Farenhorst et al., 2009). However, we could not find any study that

investigates the characteristics of architectural decisions in RQ1.

Our research question on difficulty of architectural decisions (RQ2) draws

inspiration from work on the quality of decision making (Yates et al., 2003).

Yates et al. (Yates et al., 2003) consider that the difficulty of decisions is

similar to a cost of making decisions, so reducing cost increases the quality of

decision making. We consider this also applies to architectural decisions

because they are a subcategory of decisions. Unfortunately, we could not find

related work that focuses on the difficulty of architectural decisions.

Related work on junior architects versus senior architects (RQ3) includes two

surveys and a case study on the reasoning processes of naïve and professional

architects (van Heesch and Avgeriou, 2010; van Heesch and Avgeriou, 2011;

van Heesch et al., 2013).

- Results of the first survey indicate that naïve architects (i.e.

undergraduate students) do not make trade-off between requirements,

fail to validate dependencies between decisions, and do not evaluate

critically their decisions (e.g. insufficient risk assessment) (van

Heesch and Avgeriou, 2010).

- Results of the case study indicate that naïve architects who use decision

viewpoints are more systematic in the exploration and evaluation of

decision alternatives (van Heesch et al., 2013).

- Results of the second survey (van Heesch and Avgeriou, 2011) indicate

that professional architects very often search for many design

alternatives in their decision making, unless they already have a

solution in mind. Also, professional architects prefer familiar

alternatives, because unfamiliar alternatives need more analysis effort

(van Heesch and Avgeriou, 2011). Additionally, professional


3.3. Survey Design 37

architects do not consider risk assessment as very important (van

Heesch and Avgeriou, 2011).

In our survey, we ask participants about risks, trade-offs as difficulty factors

for their architectural decisions. No related work compares characteristics and

difficulty of architectural decisions for professionals with different levels of

experience.

For our research question on good versus bad decisions (RQ4), we found much

research to avoid bad decisions. For example, ATAM and CBAM help

discover potentially bad architectural decisions (Kazman and Klein, 2001).

They do this by documenting costs, benefits, and uncertainty. Other authors

describe architectural smells as bad architectural decisions, which harm quality

attributes (Garcia et al., 2009). However, we could not find any work which

compares characteristics and difficulty of good and bad architectural decisions.

3.3 Survey Design

To design this survey, we used the recommendations and guidelines from

(Ciolkowski et al., 2003; Kitchenham and Pfleeger, 2008). Data was collected

using an online questionnaire. In this section, we present questionnaire

development, evaluation, and data collection.

3.3.1 Questionnaire Development and Evaluation

To develop the questionnaire, we took the following steps, based on

(Kitchenham and Pfleeger, 2008):

1. Review literature 2. Discuss with experts 3. Pilot questionnaire 4. Publish questionnaire

In the first step, we reviewed existing literature on architectural decisions (e.g.

(van Heesch and Avgeriou, 2010; van Heesch and Avgeriou, 2011;

Zimmermann, 2011)) and decision research (e.g. (Yates et al., 2003)). From

the literature, we identified a list of factors that contribute to the difficulty of

architectural decisions. Based on these factors, we prepared a first version of

the questionnaire that we used as a starting point for interviewing practitioners.



For the second step, we interviewed four senior architects, each with at least

ten years of experience as an architect. We asked each architect to identify two

architectural decisions they had been involved in, and discussed the

questionnaire items for both decisions. Afterwards, we asked the architects to

propose other items that contribute to the difficulty of a decision to be included

in the questionnaire. The architects also provided thoughtful feedback on the

structure of the questionnaire.

For the third step, we piloted the questionnaire with six other persons that

included researchers and practitioners. We used their feedback to further

improve the questionnaire. For example, we eliminated some items so that the

survey would take less than 15 minutes to complete (practitioners are unlikely

to finish long surveys (Kitchenham and Pfleeger, 2008)). Additionally, we

rephrased some questions to increase their clarity.

In the final step, we published the questionnaire online. The final version of

the questionnaire showed the following to participants. The full questionnaire

is available in the Appendix.

1. Welcome message 2. Control question 3. Background questions 4. Describe good decision 5. Rate 22 factors for the good decision 6. Describe bad decision 7. Rate 22 factors for the bad decision 8. Add other factors 9. Thank you message

The first page had a welcome message with an overview of the survey,

including targeted audience, required effort and incentive. The next page

checked that participants had been involved directly in making architectural

decisions during the last two years. The survey continued only after a positive

confirmation from the participant.

The survey continued with a few questions about the background of the

participant. Next, participants were asked to indicate a good architectural

decision (‘good’ according to their judgment), and describe its characteristics,

in terms of the metrics for RQ1 (e.g. number of alternatives). Next,

participants were asked to rate the 22 statements in Table 3.4 about the

difficulty of their good architectural decision on a Likert scale. Afterwards,


3.3. Survey Design 39

participants were asked to indicate a bad architectural decision, and describe

its characteristics. We asked participants to judge themselves what bad

architectural decisions were. Participants were asked to rate the same 22

statements about the difficulty of their bad decision.

Finally, participants could optionally add other items that contribute to the

difficulty of architectural decisions. Also, participants could send me a short e-

mail, to receive a copy of survey results or to offer other feedback.

3.3.2 Data Collection

Our target population was software architects who were directly involved in

making software architectural decisions during the last two years. This ensured

that survey participants could answer reliably to the questions about their

architectural decisions. We dropped the requirement of having the job title

‘architect’ because it might be over-used in the industry, without indicating

architecting activities. Also, people often make architectural decisions without

having a formal role as architects (e.g. senior software developers).

Members of our target population have busy schedules, so they are unlikely to

participate in surveys. Thus, we tried to limit the time to complete the survey

to around 15 minutes. To reach our target population, we used several

approaches. First, we sent invitations to architects in our personal networks,

asking them to participate in the survey and to further distribute it. Second, we

posted invitations to the survey on the website of a social networking site for

professionals. Third, we ran paid ad campaigns on LinkedIn (as detailed in

(Galster and Tofan, 2014)). In the advertisement we targeted professionals

working for companies in the ‘Computer Software’, Internet, or ‘Information

Technology and Services’ industries, from any part of the world, with job

functions in ‘Information Technology’ or Engineering. Also, we selected the

following job seniority: CXO, Director, Manager, Owner, Partner, Senior, or

VP, with skills in software architecture. These filters resulted in around 91,000

professionals and that could potentially see our ad. Finally, we run an ad

campaign with Google AdWords. Here, we did not find options for defining

our target audience. Instead, we defined a set of keywords (e.g. ‘software

architecture’). People searching for such keywords could see our ad for the

survey.



We ran the survey from the 1st of October 2012, until 7

th of January 2013,

using a third-party, web-based tool. In total, 219 persons started the survey.

163 persons provided partial answers (i.e. respondents answered a few

questions, and then abandoned the survey). Twelve participants did not pass

the control question about involvement in architectural decisions. Finally, 43

persons provided answers to all questions in the survey. We analyze the 43

answers in the next section.

3.4 Results Analysis

We use descriptive statistics to present the background of the survey

participants, answers to RQ1 and RQ2. Also, we use statistical tests to answer

RQ3 and RQ4.

3.4.1 Participants Background

Answers were provided from 23 countries on five continents. Regarding their

job title, most respondents described themselves as software architects

(twenty), senior software engineers (seven), or enterprise architects (five). The

other respondents described themselves as managers, business analysts, system

architects, and other roles. Table 3.1 summarizes the years of experience that

participants had in their current role and as architects. We notice that most

respondents had three to five years of experience in their current roles. Most

participants had six to ten years of experience as software developers. The

ranges for years of experience for architects and developers were normally

distributed.


3.4. Results Analysis 41

Table 3.1. Number of participants with corresponding years of experience and

roles.

Years of experience In current role As architect As developer

0-2 years 5 12 3

3-5 years 19 10 11

6-10 years 10 13 12

11-15 years 3 7 8

>15 years 6 1 9

The 43 survey participants answered questions about 86 architectural decisions

(43 good and 43 bad decisions). Architectural decisions were from domains

shown in Figure 3.1. Most decisions (31) belonged to other domains, such as

financial trading, insurance, or advertising.

Other, 31

E-commerce, 12Telecommunication, 10

E-government, 10

Embedded systems, 8

Banking, 6

Healthcare, 6

Transportation, 3

Figure 3.1. Domains of decisions.

3.4.2 RQ1 - Characteristics of Architectural Decisions

Participants indicated the actual and the elapsed time they spent for making the

86 architectural decisions. For example, an architect can spend three working

days over ten working days for a decision, resulting in a ratio of actual versus

elapsed time of 30%. In the survey, participants could indicate time in minutes,

hours, days, weeks, or months, based on their preference. We converted their

responses into working days, by considering that one working day has eight

hours, one working week has five days, and that one working months has 22

working days.

Outlier values bring a negative influence on the interpretation of averages and

standard deviations. To avoid that, we eliminate outliers. We obtain the results



in Table 3.2. On average, architectural decisions took about eight working

days, elapsed over around 35 working days. The average ratio indicates that,

overall, a third of the elapsed time is spent on the actual decision making.

Table 3.2. Metrics for actual time, elapsed time (in working days), ratio of actual

versus elapsed time, and number of directly and indirectly involved persons in

the architectural decisions.

Metric Actual Elapsed Ratio Direct Indirect

Average 7.85 34.74 0.34 3.12 7.05

Standard deviation 9.22 70.59 0.22 1.54 8.91

Minimum 0.50 0.63 0.02 1 0

Maximum 44 600 1 8 50

Mode 1 5 0.5 3 3

Participants indicated how many people were involved directly and indirectly

in making the architectural decisions. The number of indirectly involved

persons does not include the directly involved persons. We collected this

metric because we could not find any related work that describes how many

decision makers are usually involved directly in architectural decisions in

practice. For example, if most architectural decisions are made in groups, then

more work is needed for group decision making, rather than individual

decision making. We eliminated outliers for the number of people both for

direct and indirect involvement, and we obtain the results in Table 3.3.

Table 3.3. Metrics for the number of alternatives considered in the beginning,

and for an extended period of time. Last column has the number of quality

attributes.

Metric Beginning Extended time # Quality attributes

Average 2.91 1.96 4.74

Standard deviation 1.43 0.84 4.19

Minimum 1 0 0

Maximum 8 4 30

Mode 3 2 3

When making an architectural decision, more alternatives can be considered.

Since little is known about how many alternatives architects usually consider



in their decision making, we asked participants to indicate the number of

alternatives they considered at the beginning of their decision making process,

and the number of alternatives they studied for an extended period of time.

Architects consider quality attributes in their decisions. However, it is not clear

how many quality attributes they consider in practice, so we asked them to

indicate this number for their decisions. After eliminating outliers, we obtain

the metrics for the number of alternatives and quality attributes in Table 3.3.

3.4.3 RQ2 - Difficulty of Decisions

Participants rated 22 statements (Table 3.4) with factors on the difficulty of

their decisions, indicating their level of agreement with the statements, using a

five-point Likert scale with the following values: strongly disagree, disagree,

neutral, agree, strongly agree, and not applicable.

Table 3.4. Factors to describe the difficulty of a decision.

ID The decision was difficult because…

F1 you received conflicting recommendations from various sources about

which decision alternative to choose

F2 there were no previous similar decisions to compare this decision

against

F3 it was hard to identify a superior decision alternative from the

alternatives under consideration

F4 the decision required a lot of thinking from you

F5 it was hard to convince stakeholders to accept a certain decision

alternative

F6 stakeholders had strongly diverging perspectives about the decision

F7 you needed to influence some stakeholders without having formal

authority over them

F8 the decision had too many alternatives

F9 the decision had too few alternatives

F10 analyzing alternatives for this decision took a lot of effort

F11 some quality attributes were considered too late in the decision making



process

F12 too many people were involved in making the decision

F13 dependencies with other decisions had to be taken into account

F14 the decision had a major business impact

F15 you had to respect existing architectural principles

F16 serious negative consequences could result from the decision

F17 too little time was available to make the decision

F18 you had a lot of peer pressure

F19 of the trade-offs between quality attributes

F20 you lacked experience as an architect

F21 you lacked domain-specific knowledge (e.g. new customer)

F22 more information was needed to reduce uncertainty when making the

decision

Results for each factor are summarized in Figure 3.2 (left). From the bar

charts, we notice the following. First, participants indicated most agreements

(including strong agreements) with statements on dependencies with other

decisions (F13 for 69 decisions), major business impact (F14 for 60 decisions)

and serious negative consequences (F16 for 59 decisions).

Second, participants indicated most disagreements (including strong

disagreements) with statements on having too many alternatives (F8 for 57

decisions), too many people involved in decision making (F12 for 49

decisions), lack of domain-specific knowledge (F21 for 46 decisions), and

having too few alternatives (F9 for 45 decisions).

Third, participants indicated most neutral standpoint with statements on

respecting existing architectural principles (F15 for 24 decisions), needing lot

of effort for analyzing decision alternatives (F10 for 21 decisions), and having

much peer pressure (F18 for 21 decisions).

Fourth, participants indicated very few statements were ‘not applicable’ to

their decisions. Most ‘not applicable’ answers were obtained for influencing

some stakeholders without formal authority (F7 for six decisions).



Figure 3.2 (right) shows average values for all factors, calculated as follows.

We assign numerical values to the Likert scale: strongly disagree (1), disagree

(2), neutral (3), agree (4), and strongly agree (5). Not applicable values are

ignored. We acknowledge the ongoing discussions on treating a Likert scale as

either an interval or categorical type of data (e.g. (Jamieson, 2004)). Still, we

use averages because they are intuitive and easy to understand for a large

audience.

From Figure 3.2 (right), we notice that dependencies with other decisions

(F13) and major business impact (F14) have highest average agreements

across participants. Negative consequences (F16) received second highest

average. Effort for analyzing alternatives (F10), lack of similar decisions (F2)

and requiring a lot of thinking (F4) received high agreements from

participants.

We notice that some factors have averages that suggest disagreement that they

contribute to the difficulty of architectural decisions, i.e. have averages smaller

than three (neutral). For example, we notice that either too many (F8) or too

few (F9) alternatives contribute little to difficulty. The same goes for lack of

experience (F20) and domain-specific knowledge (F21). However, these two

factors need to be considered in the context that many participants were senior

architects, who might already have enough experience and knowledge.



4 5 2 4 7 7 4 7 50

9 10

2 1 3 1 3 2 0

125 3

20 1719 14

2116 20

50

40

14

26

39

9 10

20

16

3429

21

28 41

21

17

1016

17

17

17

5

9

17

21

11

12

6

14

24

10

19

21

20

20 8

9

28

27

2828

17 25

30

15 21

34

29

15

49 33

25

36

2225

37

23

26

35

1226 17 23

20 17

21

4 2

17 8 9

20

27

1223

6 6 63

5

17

51 4

04 4 6

1 1 0 3 1 0 1 2 0 2 3 20

1 1

F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 F13 F14 F15 F16 F17 F18 F19 F20 F21 F22

Strongly disagree Disagree Neutral Agree Strongly agree Not applicable ID Avg.

F13 3.88

F14 3.88

F16 3.74

F10 3.63

F2 3.61

F4 3.6

F7 3.55

F22 3.49

F3 3.48

F6 3.35

F19 3.33

F1 3.3

F5 3.27

F15 3.27

F18 3.05

F11 3.01

F17 2.93

F21 2.82

F20 2.73

F9 2.71

F12 2.69

F8 2.52

Figure 3.2. Survey results for each factor (left), and sorted average values for

factors (right) - a higher average indicates stronger agreement with the difficulty

of a factor.

Participants could indicate optionally other factors that contribute to the

difficulty of architectural decisions. We identified three other difficulty

factors: technology evolution (mentioned by six participants), insufficient

knowledge (four participants), and organizational politics (two participants).

First, technology places unpredictable constraints on decisions, such as

uncertain backwards incompatibility of future versions of a platform. Second,

insufficient knowledge refers to lack of professional knowledge, lack of peers

to discuss non-standard issues, and lack of documentation. Third, participants

indicated that organizational politics contribute to the difficulty of decisions,

and require awareness from architects.

3.4.4 RQ3 - Differences between Junior and Senior Architects

We divide survey participants in junior and senior architects, in an arbitrary

manner. We consider that junior architects have up to five years of experience

as architects. Senior architects have six or more years of experience as



architects. Based on this separation, 22 junior and 21 senior architects

answered the survey.

To compare the answers from junior and senior architects, we use a statistical

test. We compare two independent groups and cannot assume that the data is

normally distributed. Thus, we use the Mann-Whitney U test (Field, 2009), a

non-parametric test. We investigate the differences between junior and senior

architects on the answers offered to the 22 factors in Table 3.4, and the eight

metrics in Table 3.2 and Table 3.3.

We use SPSS 20 to apply the Mann-Whitney U test on the survey data. ‘Not

applicable’ answers are treated as missing values. We obtain significant

statistical differences (p-value less than 0.05) between junior and senior

architects for five difficulty factors and two characteristics. Table 3.5 presents

the Mann-Whitney U test results (U, Z, and p), median and average values for

junior and senior architects. Results for difficulty factors without statistical

significance are omitted.

We notice that junior architects found conflicting recommendations on what to

consider for a decision (F1) as more significant to making a decision difficult.

Also, in contrast to senior architects, junior architects found that if lots of

thinking is required (F4), decisions become more difficult. In turn, senior

architects found that decisions become more difficult if they have a major

business impact (F14). Finally there are differences between junior and senior

architects on experience (F20) and domain-specific knowledge (F21); these

differences can be expected to some extent, because senior architects have

more experience and domain-specific knowledge.



Table 3.5. Results of the Mann-Whitney U test are displayed on the first four

columns. Median and average values for junior (J) and senior (S) architects are

displayed on the last four columns. Lines with p-value > 0.05 have gray

background.

ID U Z p Mdn.

J

Mdn.

S

Avg.

J

Avg.

S

F1 565 -2.49 .006 4 3 3.61 2.97

F4 722 -1.80 .036 4 3 3.82 3.38

F14 716 -1.71 .044 4 4 3.73 4.05

F20 429 -4.42 .000 3 2 3.25 2.19

F21 699 -1.93 .027 3 2 3.74 2.59

Actual time 553 -3.21 .001 2.75 9 4.87 19.67

Elapsed time 742 -1.57 .058 15 21.5 22.87 62.32

Ratio time 621 -2.62 .004 0.25 0.35 0.29 0.42

Direct people 883 -.36 .361 3 3 3.28 3.78

Indirect

people

901 -.20 .422 4 4 8.79 24.12

Alternatives

beginning

780 -1.28 .100 2.5 3 2.77 3.62

Alternatives

extended

916 -.07 .474 2 2 2 2.08

No. QAs 764 -1.40 .081 3 4.5 4.23 5.07

We notice that senior architects spend significantly more actual time on their

decisions than junior architects (four times on average). The ratio of actual on

elapsed time is higher for senior architects, with a clear statistical significance.

We also notice a tendency of senior architects to consider more alternatives in

the beginning of the decision making. Also, senior architects tend to consider

more quality attributes than junior architects; however the statistical

significance is not sufficient.



3.4.5 RQ4 - Differences between Good and Bad Decisions

Each survey participant answered questions about a good and a bad

architectural decision. Comparing participants’ answers on the two types of

decisions increases our understanding on the quality of architectural decisions,

by analyzing the link between the two aspects of quality: difficulty (the 22

factors in Table 3.4) and outcome (good and bad decisions). Furthermore, we

analyze the link between the characteristics of architectural decisions and their

outcome (e.g. are there differences between the time spent on good or bad

decisions).

We compare differences between the 43 good and 43 bad decisions using the

Wilcoxon signed ranks test (Field, 2009), a non-parametric statistical test for

comparing groups of two related samples. We treat ‘not applicable’ answers as

missing values. Similar to the analysis in Section 3.4.4, we investigate the

differences between good and bad decisions related to the data in Table 3.2

and Table 3.3. The results of the Wilcoxon signed rank test are presented in

Table 3.6. Results on difficulty factors with statistically insignificant values

are not included.



Table 3.6. Results of the Wilcoxon signed rank test, medians and averages for

variables and metrics that correspnd to good (G) and bad (B) decisions. Lines

with p-values > 0.05 have gray background.

ID Z p-

value

Mdn.

G

Mdn.

B

Avg.

G

Avg.

B

F9 -2.48 0.006 2 3 2.47 2.95

F11 -3.07 0.001 2 4 2.61 3.4

F13 -2.12 0.018 4 4 4.07 3.7

F17 -2.63 0.004 2 3 2.67 3.19

F18 -2.03 0.022 2.5 3 2.81 3.29

F20 -1.85 0.04 3 3 2.6 2.86

Actual time -0.51 .308 4 5 14.44 9.76

Elapsed time -0.73 .235 20 20 38.02 46.25

Ratio time -0.09 .463 0.33 0.33 0.37 0.34

Direct

people

-0.55 .294 3 3 3.37 3.63

Indirect

people

-1.40 .082 4 4 17.14 15.42

Alternatives

beginning

-2.41 .007 3 2 3.65 2.72

Alternatives

extended

-1.92 .029 2 2 2.21 1.84

No. QAs -2.68 .003 4 3 5.02 4.26

We notice differences on having too few alternatives (F9), with a tendency for

disagreement with F9 on good decisions, and for neutral with F9 on bad

decisions. For bad decisions, participants indicated that some quality attributes

were considered too late (F11), in contrast with good decisions. Also,

dependencies with other decisions (F13) are more difficult for good than bad

decisions. Participants disagreed on too little available time (F17), much peer


3.5. Discussion 51

pressure (F18), and lack of experience (F20) for the good decisions, in contrast

with the bad decisions.

Regarding the metrics for decisions characteristics, we notice statistically

significant differences between the number of alternatives considered at the

beginning of the decision making process, number of alternatives studied for

an extended period of time, and the number of quality attributes. For all these

three parameters, the good decisions had higher numbers. The results for the

parameters about time and number of involved people have insufficient

statistical significance.

3.5 Discussion

Little is known in the literature about the duration of real-world architectural

decisions. We found that the time needed for architectural decisions varies

much. An architectural decision takes an average actual time of around eight

working days, over an average elapsed time of 35 working days. Also,

architects spend around one third of the elapsed time on the actual decision

making (Table 3.5). The survey results indicate no significant differences

between good and bad decisions regarding actual and elapsed time (Table 3.6).

However, participants considered they had enough time for the good decisions,

and not enough time for the bad decisions.

Another insight from this study is that the actual time junior architects spend

on making a decision is one quarter of that spent by senior architects (Table

3.5). We expected senior architects to spend less or similar amounts of time to

junior architects, because of their extra experience. A possible explanation is

that senior architects might deal with higher impact decisions than juniors, so

they need the extra time. A future, more precise comparison should use a ratio

of time per decision impact, which can be quantified as the estimated cost of

reversing the decision.

Another insight from this survey concerns the number of people involved in

architectural decisions. The importance of stakeholders in architectural

decisions is widely recognized in the literature. Stakeholders are always

involved indirectly in decision making. However, we could not find any

literature on the direct involvement of stakeholders in architectural decisions,

as actual decision makers, rather than decision influencers. To improve



decision support, researchers need to know if architectural decisions are

typically made by one person (i.e. the architect) or by groups of persons (i.e.

one or more architects, and other stakeholders). For example, researchers can

propose group decision making approaches, if a relevant proportion of

architectural decisions are made in groups. A surprising result is that only 14%

of the decisions in the survey were made by individuals. The typical

architectural decision has three decision makers (Table 3.2). Consequently,

group architectural decision making is a much needed research direction.

Regarding difficulty of decisions, we notice that dependencies with other

decisions are a very important factor for the perceived difficulty of decisions.

Results from a related survey (van Heesch and Avgeriou, 2011) indicate that

architects often come across such dependencies. Moreover, researchers

proposed various approaches for handling decisions dependencies (e.g. (Jansen

et al., 2009)). Our survey confirms the relevance of the topic, and the need for

disseminating research results to practitioners. We also found that analysis

effort and lack of similar (or previously made) decisions increase difficulty of

decision making. This suggests that practitioners welcome approaches that

help them analyze decisions, and appreciate examples of similar decisions as

opportunities to reuse architectural knowledge.

Regarding differences between junior and senior architects we found that

junior architects need help to address the difficulties of analyzing decisions,

such as handling conflicting recommendations. This is not relevant for senior

architects. We consider that existing documentation approaches help junior

architects. However, documentation could be improved by adding capabilities

for analyzing decisions (e.g. what-if analysis).

Regarding differences between good and bad decisions, the survey results

indicate that good decisions have more alternatives than bad decisions.

Therefore, as a rule of thumb, we recommend practitioners to identify three or

more alternatives. Also, this study confirms that practitioners should pay

attention to quality attributes and decisions dependencies while making

architectural decisions.


3.6. Conclusions 53

3.5.1 Validity Threats

We present potential study limitations, in terms of internal, construct, and

external validity (Ciolkowski et al., 2003; Kitchenham and Pfleeger, 2008). To

increase internal validity, we piloted the questionnaire and refined it to ensure

that participants could understand it. Also, we added explanatory text with

small examples to the questions, so that participants could easily interpret the

questions. Another threat to internal validity is that our findings were derived

from polarized instances of architectural decisions (i.e. good and bad), rather

than typical decisions. Not adding typical decisions to the questionnaire was a

necessary trade-off for keeping the survey duration within 15 minutes.

To make sure we measure the difficulty of decisions, we used existing work

(Yates et al., 2003) as a starting point for characterizing decisions and to

conceptualize difficulty, as suggested by (Ciolkowski et al., 2003). Moreover,

we discussed our conceptualization with experienced architects, who helped us

refine it. Furthermore, participants could answer with ‘not applicable’ to the

survey items on difficulty of their decisions. The very low numbers of ‘not

applicable’ answers indicates that the survey items indeed measure difficulty

of decisions.

We increased the external validity of this survey by recruiting participants

through posting survey invitations in venues for professionals, and by using

paid ad campaigns. These efforts resulted in a high diversity of respondents,

compared to only using respondents from personal networks or only answers

from a specific region or company.

3.6 Conclusions

In this chapter, 43 participants provided information about 86 architectural

decisions. We found that dependencies among decisions, analysis effort and

lack of similar and previously made decisions make architectural decisions

difficult. Moreover, junior architects need decision analysis support more than

senior architects. Also, we found that considering more alternatives may lead

to better decisions. Additionally, this survey confirmed the impact of quality

attributes and decisions dependencies on the quality of architectural decisions.

In addition, this survey confirms the importance of reducing architectural



knowledge vaporization (a challenge also identified in the previous chapter),

so that architects can access similar and previously made decisions.

This survey provided insights on the duration and number of people,

alternatives and quality attributes in architectural decision making, which

suggests the need of further research on group architectural decision making.

To offer decision analysis support, Chapters 6 and 7 present approaches for

making architectural decisions (including group decisions). Chapter 8 presents

an open-source tool that includes support for group decision making, analyzing

alternatives and dependencies with other decisions.

After studying practical architectural knowledge management and architectural

decisions in Chapter 2 and Chapter 3, the next chapter presents a systematic

mapping study of literature on architectural decisions, in which we identify

and synthesize existing research on architectural decisions.

Acknowledgment

We thank the study participants for their help.

university of groningen understanding and supporting ... · architectural decisions in practice and...

Documents