agile alm tool comparison

Vendor Landscape: Agile ALM

Info-Tech Research Group 1

Vendor Landscape: Agile ALM

ALMs for the poor, and for the rich.

As development platforms, coding methodologies, and devices increase in number, Agile Application Life Management (ALM) tools support integrations with an ever-increasing range of systems.

Introduction

� Enterprises seeking to select a solution for Agile Application Lifecycle Management (ALM) technology

This Research Is Designed For: This Research Will Help You:

�Understand what’s new in the Agile ALM market.

� Evaluate Agile ALM vendors and products for your


� Their Agile ALM use case may include:

• Providing testing support for a quality-centric development environment.

• Supporting Agile development.

• Integrating lifecycle management with Integrated Development Environments.

� Evaluate Agile ALM vendors and products for your enterprise needs.

�Determine which products are most appropriate for particular use cases and scenarios.

Executive Summary

Info-Tech evaluated eight competitors in the Agile ALM market, including the following notable performers:

Champions:

• CollabNet TeamForge, a comprehensive, full-featured ALM tool at a reasonable price. Despite its impressive range of offerings, CollabNet supports the SMB.

• Micro Focus Borland ALM, a long-standing leader in the market with a broad feature offering and strong product support.

• Serena Suite, a robust offering backed by strong customer support and a broad reseller network.

1. ALM and BPM start to intermingle:

Premium offerings give the ability to customize process. Evaluate your need for, and the cost of, process customization before you make your purchase.

2. The range of platforms is growing:

Premium products support development

Info-Tech Insight


support and a broad reseller network.

Value Award:

• TechExcel DevSuite, a comprehensive offering with a price tag far below its comparable competitors.

Premium products support development across the growing range of platforms and development methodologies, including mobile, web, and infrastructure.

3. New entrants offer attractive options:

Niche offerings can serve your needs at a fraction of the cost of a premium tool. For example, consider specialized tools for Agile and testing integration.

Market Overview

• Long-standing vendors in the ALM space can trace their roots to the 1980s, or even earlier.

• Since the 1990s, ALM tools have played a key part in the project manager’s arsenal, allowing him or her to track project status and progress towards objectives.

• Managers have paid top dollar for ALM suites that could track project data and provide meaningful reporting.

• Agile ALM has moved into the Cloud. Major vendors of cloud collaboration software have moved into the ALM space and new SaaS-only ALM tools have appeared.

• Process flexibility has become key. Instead of a strict adherence to Agile or waterfall development, most firms have pursued a middle path and customized their methodology to meet their own needs. Some Agile ALM tools cater to the need for flexibility.

How it got here Where it’s going


track project data and provide meaningful reporting.

• As development environments have evolved, the tools and components being used to manage this process have grown to become integrated and convenient, covering all phases of the development lifecycle including architecture, testing, and deployment with a single common interface.

Agile ALM tools cater to the need for flexibility.

• Quality control has become a core part of ALM. Several major ALM tools are built around testing tools and process maturity.

• Going forward, expect stronger integration between ALM tools and the ecosystem of products supporting development, such as testing, PLM tools, and IDEs.

As the market evolves, capabilities that were once cutting edge become default, and new functionality becomes differentiating. Traceability has become a table stakes capability and should no longer be used to differentiate solutions. Instead, focus on workflow and reporting to get the best fit for your requirements.

Agile ALM Vendor Landscape selection / knock-out criteria:Market share, mind share, and market consolidation

• CollabNet TeamForge. A relatively recent entrant with a strong offering and a reasonable price tag.

• Micro Focus Borland ALM. A large, ALM-focused vendor that provides strong support for its product.

Included in the Vendor Landscape:

• The Agile ALM space continues to see new entrants, new products, and new features. Ongoing demand for development on mobile and web platforms creates new opportunities for vendors focused on a particular market niche.

• For this Vendor Landscape, Info-Tech focused on those vendors that have a strong market presence and/or reputational presence among small to mid-sized enterprises.


• PTC Integrity. A strong ALM offering tightly integrated into a PLM platform.

• Parasoft Concerto. A product targeted towards development shops with a high level of process maturity.

• Serena Suite. The complete ALM package controlled by a complete business process engine.

• SmarteSoft SmartSuite. ALM tools built around a quality-control platform.

• TechExcel DevSuite. A flexible tool that supports development on a variety of platforms.

• ThoughtWorks Suite (Mingle, Twist, Go). A strong, Agile-focused product offering that provides some flexibility

within the Agile framework.

Agile ALM Criteria & Weighting Factors

30%

20%20%

30%Features Usability

AffordabilityArchitecture

50%

Product

Product Evaluation

The five year TCO of the solution is economical.

Affordability

The delivery method of the solution aligns with what is expected within the space.

Architecture

The solution’s dashboard and reporting tools are intuitive and easy to use.

Usability

The solution provides basic and advanced feature/functionality.

Features


50%

Vendor

Vendor Evaluation

Vendor is committed to the space and has a future product and portfolio roadmap.

Strategy

Vendor offers global coverage and is able to sell and provide post-sales support.

Reach

Vendor is profitable, knowledgeable, and will be around for the long term.

Viability

Vendor channel strategy is appropriate and the channels themselves are strong.

Channel

what is expected within the space.Architecture

30%

30%

15%

25%Viability Strategy

ReachChannel

The Info-Tech Agile ALM Vendor Landscape

Champions receive high scores for most evaluation criteria and offer excellent value. They have a strong market presence and are usually the trend setters for the industry.

Innovators have demonstrated innovative product strengths that act as their competitive advantage in appealing to niche segments of the market.

Market Pillars are established players with very strong vendor credentials, but with


very strong vendor credentials, but with more average product scores.

Emerging players are newer vendors who are starting to gain a foothold in the marketplace. They balance product and vendor attributes, though score lower relative to market Champions.

For an explanation of how the Info-Tech Vendor Landscape is created please see the slide entitled “Vendor Evaluation Methodology” in the appendix.

Every vendor has its strengths & weaknesses;Pick the one that works best for you

Product Vendor

Features UsabilityAfford-ability

Viability Strategy Channel

Micro Focus

PTC

CollabNet

ReachArchi-tecture

Overall Overall


SmarteSoft

Parasoft

Serena

TechExcel

ThoughtWorks

For an explanation of how the Info-Tech Harvey Balls are calculated please see the slide entitled “Vendor Evaluation Methodology” in the appendix.

Legend =Exemplary =Good =Adequate =Inadequate =Poor

What is a Value Score?

TechExcel captures the highest value score with a comprehensive, affordable offering

The Value Score indexes each vendor’s product offering and business strength relative to their price point. It does not indicate vendor ranking.

Vendors that score high offer more bang for the buck (e.g. features, usability, stability, etc.) than the average vendor, while the inverse is true for those that score lower.

On a relative basis, TechExcel maintained the highest Info-Tech Value ScoreTM of the vendor group. Vendors were indexed against TechExcel’s performance to provide a complete, relative view of their product offerings.

Champion


*Thought-Works

0

*Parasoft

0

*PTC

0

Micro Focus

13

SmarteSoft

22

Serena

55

CollabNet

64

TechExcel

100

Average Score: 31.5

Price-conscious enterprises may wish to give the Value Score more consideration than those who are more focused on specific vendor/product attributes.

For an explanation of how the Info-Tech Value Index is calculated please see the slide entitled “Value Index Ranking Methodology” in the appendix.

For an explanation of how normalized pricing is determined please see the slide entitled “Product Pricing Scenario & Methodology” in the appendix.

*Vendor declined to provide pricing.

Table Stakes represent the minimum standard; without these a product doesn’t even get reviewed

The products assessed in this Vendor LandscapeTM meet, at the very least, the requirements outlined as Table Stakes.

Many of the vendors go above and beyond the outlined Table Stakes, some even do so in multiple categories. This section aims to highlight the products capabilities in excess of the criteria

The Table Stakes What Does This Mean?

Feature Description

Basic task management

The product allows the user to createtasks, assign resources to those tasks, and view project status reporting.

Graphical displays The product displays tasks and other objects as editable, graphical objects.


If Table Stakes are all you need from your Agile ALM solution, the only true differentiator for the organization is price. Otherwise, dig deeper to find the best price to value for your needs.

the products capabilities in excess of the criteria listed here. Traceability The product ties together objects created

at various stages of the lifecycle, showing how tasks relate to requirements, etc.

Links to external documents

The product allows the user to link to external documents for requirements and other content.

Central data store The product stores data in a centralized repository, not on local disk.

Advanced Features are the market differentiators that make or break a product

Feature What We Looked For

Requirements Management

Users can enter requirements in a hierarchical structure.

Change Management The product packages and traces changes.

Workflow The product provides automated workflow.

Source Code Management

The product manages source code or integrates with SCM solutions.

Advanced Features

Info-Tech scored each vendor’s features offering as a summation of their individual scores across the listed advanced features. Vendors were given one point for each feature the product inherently provided. Some categories were scored on a more granular scale with vendors receiving half points.

Scoring Methodology


Management integrates with SCM solutions.

Task Management The product manages timelines and tasks.

Testing The product provides test management.

Defect/Bug Tracking The product manages and traces defects.

Resource Management

The product permissions users to edit objects and evaluates resource capacity.

Reporting & Analytics The product analyzes project progress and provides meaningful reporting.

Release Management The product helps with release planning.

Content Management The product organizes documents.

Each vendor offers a different feature set; concentrate on what you need

Collab-Net

PTC

MicroFocus

Req. Mgmt

Work-flow

Task Mgmt

TestingRes. Mgmt

Defect Track

SCMReport-

ingRel.

MgmtContent Mgmt

Evaluated FeaturesChangeMgmt


Parasoft

Serena

Smarte-Soft

Tech-Excel

Thought-Works

=Feature absent=Feature partially present/pending=Feature fully presentLegend

Product:Employees:

Headquarters:Website:

Founded:Presence:

TeamForgeUnavailableBrisbane, CACollab.net1999Private company

CollabNet offers a strong product with a broad range of features and a reasonable price tag

Champion• A recent entrant to the ALM space (but in SCM since 1999),

CollabNet offers the range of functionality normally expected by large organizations, but targets the SMB market with a relatively affordable price tag.

Overview

• TeamForge provides all of the functionality associated with full-featured ALM, from requirements through to release.

• Despite its robust offering and some well-known clients, CollabNet has a large number of SMB clients as well.

• CollabNet has strong support offerings, both for its clients and

Strengths


$1

Info-Tech Recommends:

CollabNet offers a very strong product at a reasonable price, supported by a recognized vendor in the source code management space.

• CollabNet has strong support offerings, both for its clients and its channel partners.

• CollabNet’s limited revenue and emphasis on North America sales restricts its global reach. However, the company has experienced strong revenue growth.

• TeamForge’s native resource management solution lacks functionality, although it offers basic capacity management.

Challenges

3 Year TCO: Tier 8; between $250K and $500K

$1M+

Product:Employees:


Founded:Presence:

Borland ALM1,200Newbury, UKmicrofocus.com1976LON: MCROFY11 Revenue: $436M

Micro Focus provides full-featured ALM with strong customer and reseller support

Champion• With 30 years of experience in application management, Micro

Focus continues to lead the market with a strong offering.

Overview

• The Borland ALM suite benefits from Micro Focus’s strong global reach and extensive experience in ALM.

• Micro Focus combines large size with a focus on ALM.• Micro Focus offers strong support for its products, including

multi-language, 24x7 support.

Strengths



Organizations seeking a premium product and willing to pay a substantial amount should consider Micro Focus.

multi-language, 24x7 support.• Micro Focus’s reseller networks stretch across four continents.

• The Borland ALM suite excludes some key functionality such as resource utilization management, assuming that the user will integrate with a project management tool.

• Borland ALM does not have burn-down reports or agile task management reporting.

Challenges

$1


$1M+

Product:Employees:


Founded:Presence:

Serena Suite700Redwood City, CAserena.com1980Private company

Serena provides all the ALM tools, packaged within a powerful process orchestration engine

Champion• Serena continues to achieve competitive differentiation

through its Serena Business Manager, a full-fledged process management solution that orchestrates development.

Overview

• Serena includes all of the functionality associated with full-featured ALM, from requirements to release management.

• Serena has proven longevity in the ALM space, having entered the market over 30 years ago.

• Serena’s support offices and reseller network stretch across

Strengths



Serena will make sense for organizations needing a high level of process customizability and workflow, and willing to make a substantial investment.

• Serena’s support offices and reseller network stretch across four continents. Support is available in ten languages.

• Serena support centers provide only callback service on weekends.

Challenges

$1


$1M+

Product:Employees:


Founded:Presence:

DevSuite155Lafayette, CAtechexcel.com1999Private company

TechExcel offers flexibility and a range of ALM features; support offering is weak

Innovator• Despite being a long-standing player in the ALM business,

TechExcel has only a small share of the market.

Overview

• DevSuite includes most of the functionality associated with full-featured ALM, from requirements to task management.

• DevSuite offers a high degree of flexibility, making it useful for development on a range of platforms.

• TechExcel integrates with a variety of development tools and

Strengths



Organizations seeking a full-featured ALM solution should consider TechExcel, particularly if they have limited support needs.

• TechExcel integrates with a variety of development tools and supports development using diverse methodologies.

• TechExcel offers relatively weak support for a major player, providing telephone support only during business hours.

• TechExcel lacks key release management functionality and some reporting and analytics capabilities as well.

• Despite its longstanding presence in the market, TechExcel lacks a strong global revenue footprint.

Challenges

$1


$1M+

Product:Employees:


Founded:Presence:

ConcertoUnavailableMonrovia, CAparasoft.com1987Private company

Concerto provides a range of functionality and targets high-maturity development shops

Market Pillar• A long-standing player in the ALM market, Parasoft has a wide

array of patent holdings that gives it a competitive edge in the marketplace.

Overview

• Concerto includes most of the functionality associated with full-featured ALM, including requirements and task management.

• Concerto can integrate with .NET and Eclipse IDEs.• Concerto allows the user to create custom workflows in UML.• Parasoft has a strong market focus on the ALM space.

Strengths


$1


Concerto will provide strong support for organizations of any size that require a high level of development process maturity.

• Parasoft has a strong market focus on the ALM space.

• Concerto does not include release management functionality and Parasoft has no intention of adding this.

• While Parasoft has clients in the SMB space, Concerto targets a level of process maturity (CMMI 3 or above) that most Info-Tech SMBs do not have.

• Parasoft has a limited footprint compared to larger players.

Challenges

Pricing was not made available

$1M+

Product:Employees:


Founded:Presence:

Integrity5,000Needham, MAptc.com1985NASDAQ: PMTCFY10 Revenue: $1B

PTC Integrity provides a range of ALM offerings within a Project Lifecycle Management context

Emerging Player• PTC acquired MKS in 2011 to integrate MKS’s Integrity ALM

offerings into PTC’s suite of Product Lifecycle Management (PLM) tools.

Overview

• Integrity provides almost all of the functionality associated with full-featured ALM, from requirements to task management.

• Integrity supports both Agile and waterfall methodologies, plus the ability to graphically edit processes.

• Integrity benefits from strong integration with other lifecycle

Strengths


$1


Integrity has a strong offering for larger organizations, or any organization seeking to integrate ALM with a broader PLM suite.

• Integrity benefits from strong integration with other lifecycle management tools sold by PTC.

• The target client for the product has at least 25 software developers, making it a difficult proposition for many SMBs.

• PTC limits phone support to weekdays, unusual for a vendor of this size.

• While PTC has a long history in the PLM space, its ability to manage the ALM side of the business remains unproven.

Challenges


$1M+

Product:Employees:


Founded:Presence:

SmarteSuite50Austin, TXsmartesoft.com1999Private company

SmarteSoft provides full-featured ALM at a substantial price; support offering is weak

Emerging Player• SmarteSoft entered the ALM field in 2005 with its SmarteSuite

of tools.• The suite centers around SmarteQM, a quality-oriented ALM

and testing tool.

Overview

• Despite its positioning as a test automation tool, SmarteQM has most of the functionality associated with ALM.

• Half of SmarteSoft’s clients are still mid-sized businesses, a relatively strong focus on the SMB client.

• Customizable workflow in SmarteQM allows users to adapt to

Strengths



SmarteSoft’s credentials in test management software will make this a suitable choice for users with heavy testing and quality management needs.

• Customizable workflow in SmarteQM allows users to adapt to various development platforms and methodologies.

• SmarteSoft offers relatively weak support given the cost of SmarteQM: initial contact is usually via email with a 24-hour turnaround time.

• Given its recent entry, SmarteSoft lacks proven longevity in the space, a concern given the high price tag.

Challenges

$1


$1M+

Product:Employees:


Founded:Presence:

Mingle, Twist, GoUnavailableChicago, ILthoughtworks-studios.com1993Private company

ThoughtWorks offers a strong offering for Agile development but limited support for waterfall or other methods

Emerging Player• A recent entrant into the Agile ALM market, ThoughtWorks

provides a robust offering but limited support for non-Agile development activity.

Overview

• ThoughtWorks includes most of the functionality associated with full-featured ALM, from requirements to task management.

• ThoughtWorks has an organizational focus on Agile ALM and long-standing experience in Agile development.

Strengths



ThoughtWorks can make sense for Agile-oriented organizations, or organizations seeking training and help in moving towards Agile.

long-standing experience in Agile development.• ThoughtWorks testing is designed for a variety of platforms.

• ThoughtWorks offers email-only support and a one-day turnaround, a relatively weak support offering.

• ThoughtWorks has a strong focus on the Agile methodology, limiting the usefulness of the tool for waterfall or other methods.

• As well, integration with .NET is not offered.

Challenges

$1


$1M+

The Info-Tech Agile ALM Vendor Shortlist Tool is designed to generate a customized shortlist of vendors based on your key priorities.

Identify leading candidates with the Agile ALM Vendor Shortlist Tool

• Overall Vendor vs. Product Weightings

• Top-level weighting of product vs. vendor criteria

• Individual product criteria weightings:�Features

This tool offers the ability to modify:


�Features�Usability�Affordability�Architecture

• Individual vendor criteria weightings:�Viability�Strategy�Reach�Channel

Look for a vendor that provides a full-featured testing solution as part of the ALM offering.

Quality-sensitive development shops require tight integration with a top-notch testing tool

Quality-sensitive development1

2

Exemplary Performers

Viable Performers


Agile focused

Development integrated

2

3

Viable Performers

Adequate Performers

4

Seek an ALM product with native support for Agile processes and artifacts.

Agile-focused environments should seek a tool with strong support for Agile


2


Viable Performers


Agile focused


2

3

Viable Performers

Adequate Performers

4

Look for native integrations as well API-based integrations for niche IDEs.

Organizations requiring development integration should seek flexible integration capabilities


2


Viable Performers


Agile focused


2

3

Viable Performers

Adequate Performers

4

Appendix

• Vendor Evaluation Methodology

• Value Index Ranking Methodology

• Product Pricing Scenario & Methodology


Vendor Evaluation Methodology

Info-Tech Research Group’s Vendor Landscape market evaluations are a part of a larger program of vendor evaluations that includes Solution

Sets that provide both Vendor Landscapes and broader Selection Advice.

From the domain experience of our analysts, as well as through consultation with our clients, a vendor/product shortlist is established. Product

briefings are requested from each of these vendors, asking for information on the company, products, technology, customers, partners, sales

models, and pricing.

Our analysts then score each vendor and product across a variety of categories, on a scale of 0-10 points. The raw scores for each vendor are

then normalized to the other vendors’ scores to provide a sufficient degree of separation for a meaningful comparison. These scores are then

weighted according to weighting factors that our analysts believe represent the weight that an average client should apply to each criteria. The

weighted scores are then averaged for each of two high level categories: vendor score and product score. A plot of these two resulting scores

is generated to place vendors in one of four categories: Champion, Innovator, Market Pillar, and Emerging Player.


For a more granular category by category comparison, analysts convert the individual scores (absolute, non-normalized) for each

vendor/product in each evaluated category to a scale of zero to four whereby exceptional performance receives a score of four and poor

performance receives a score of zero. These scores are represented with “Harvey Balls,” ranging from an open circle for a score of zero to a

filled in circle for a score of four. Harvey Ball scores are indicative of absolute performance by category, but are not an exact correlation to

overall performance.

Individual scorecards are then sent to the vendors for factual review, and to ensure no information is under embargo. We will make corrections

where factual errors exist (e.g. pricing, features, technical specifications). We will consider suggestions concerning benefits, functional quality,

value, etc; however, these suggestions must be validated by feedback from our customers. We do not accept changes that are not

corroborated by actual client experience or wording changes that are purely part of a vendor’s market messaging or positioning. Any resulting

changes to final scores are then made as needed, before publishing the results to Info-Tech clients.

Vendor Landscapes are refreshed every 12 to 24 months, depending upon the dynamics of each individual market.

Value Index Ranking Methodology

Info-Tech Research Group’s Value Index is part of a larger program of vendor evaluations that includes Solution Sets that provide both Vendor

Landscapes and broader Selection Advice.

The Value Index is an indexed ranking of value per dollar as determined by the raw scores given to each vendor by analysts. To perform the

calculation, Affordability is removed from the Product score and the entire Product category is reweighted to represent the same proportions.

The Product and Vendor scores are then summed, and multiplied by the Affordability raw score to come up with Value Score. Vendors are

then indexed to the highest performing vendor by dividing their score into that of the highest scorer, resulting in an indexed ranking with a top

score of 100 assigned to the leading vendor.

The Value Index calculation is then repeated on the raw score of each category against Affordability, creating a series of indexes for Features,

Usability, Viability, Strategy and Support, with each being indexed against the highest score in that category. The results for each vendor are

displayed in tandem with the average score in each category to provide an idea of over and under performance.


The Value Index, where applicable, is refreshed every 12 to 24 months, depending upon the dynamics of each individual market.

Product Pricing Scenario & Methodology

Info-Tech Research Group provided each vendor with a common pricing scenario to enable normalized scoring of Affordability, calculation of

Value Index rankings, and identification of the appropriate solution pricing tier as displayed on each vendor scorecard.

Vendors were asked to provide list costs for Agile ALM software licensing to address the needs of a reference organization described in the

pricing scenario.

Additional consulting, deployment, and training services were explicitly out of scope of the pricing request, as was the cost of enhanced

support options, though vendors were encouraged to highlight any such items included with the base product acquisition. Vendors were asked

to prepare a three-year total acquisition cost for their respective Agile ALM solutions. This three-year total acquisition cost is the basis of the

solution pricing tier indicated for each vendor.

Finally, the vendors’ three-year total acquisition costs were normalized to produce the Affordability raw scores and calculate Value Index

ratings for each solution.


Key elements of the common pricing scenario provided to Agile ALM vendors included:

• A six-site organization with 2,200 employees, located at two locations in each of the US, England, and France. The US locations create

software to be used internally, as well as for external clients, while the development resources in England and France are focused almost

entirely on externally facing applications, including mobile solutions.

• The development organization has grown rapidly through acquisitions. While the teams work well together, projects are often seen as

chaotic when crossing office boundaries.

• The corporate development group has determined that implementing ALM consistently across all locations and projects would greatly

improve the overall efficiency of the collective global development group.

• The corporate development group would like to create visibility for the corporate stakeholders into all projects being executed with real-time

(or near real-time) access to data. Dashboards and reports should be able to be filtered by project, by development group, and across the

entire corporation.

• Development projects are typically run with an Agile approach, following two-week sprints for most projects.

agile alm tool comparison

Documents

agile alm tools

agile alm market

strong alm

agile alm vendors

agile alm space

agile development

alm suites

major alm tools