case study type in a sub-title agenda here 3 agenda business problem & background solution...

Case StudyType in a sub-title agenda here

3

Agenda

• Business Problem & Background

• Solution Approach

• Along the Way: Learnings

• Summary

4

Trends in Enterprise Call Centers

• Cost reduction

• Multiple Outsourcers (agents)

• Multiple Geographies (global)

• Multiple Networks (transport)

• Multiple Technologies (on call center premises)

5

Call Distribution Evolution – ’80s

Long DistanceNetwork

Single enterprise & single site ACD

Automatic Call Distributors

Single Enterprise/ Single Site1980’s

Although business strategies have evolved, call center technology has not kept pace. Many concepts developed in the early 80’s still exist in TDM and IP solutions

Enterprise Customer Enterprise Customer

Avaya

NORTEL

ASPECT

6

Call Distribution Evolution – ’90s

Long DistanceNetwork

Single enterprise with multiple call centers

Network Routing & CTI

Single Enterprise/ Multiple Sites

1990’s

In the 1990s, vendors addressed single enterprise and multi-site routing with network routing solutions

Enterprise Customer Enterprise Customer

Genesys

Cisco

7

New headache, needing a new aspirin

Outsourcer Enterprise

Client Enterprise

Outsourcer Enterprise Outsourcer Enterprise

Client EnterpriseClient Enterprise

Life has become a

Mess….

Multiple Enterprises

Multiple Sites

Multiple Networks

Multiple Technologies

Multiple Operations

Call Distribution Evolution – Now

8

Implementation Models

• Black Hole– Call sent to offshore outsourcer directly from the network– No direct Visibility, Control or Quality

• Forced Footprint– Outsourcer implements Enterprise chosen Technology (e.g. ICM or

Genesys)– Provides a level of dynamic control over call routing and call reporting

(e.g. at pre-routing phase, reports of agent handling through routing software)

• Mini-Telco– Bring the call into a domestic ACD– Peel off calls to go to domestic teams or offshore teams– Extensive control over routing and reporting– Can do compliance recording– Difficult to do agent monitoring

9

Vendor Management

• Outsourcer provides Reporting and Service Level Management for Blackhole implementations

– Typically, this means end of day reports with metrics like SL, abandons, max queue depth, etc.

• Enterprise has insight into overall call center performance through historic reports

– Some get more depth in reporting

• These features cost more, and provided by outsourcer based on their technology implementation

– The more features that the outsourcer provides the enterprise, the higher the cost (software licenses for switch vendor’s reporting software)

10

Issues

• Management (Visibility, Control, Quality) of call delivery to captive and outsourced call centers

• Assessment of outsourcer performance

• Contract management– Staffing level– Customer satisfaction on service levels

• Non-capex solution required

11

Solution Requirements

• An outsourcer management gateway• Matches any deployed technology• Zero footprint at Enterprise & Outsourcer• Integrates with existing call center solutions• Delivered as a managed service• Cost effective, pay as you go model

Solution Approach

13

Technology Trends

• Migration of enterprises to converged networks• Emergence of SIP in the VoIP space• Downward cost curves of VoIP equipment• Leverageable building blocks• Standards

– VoiceXML– CCXML

14

The Solution: Version 1

• Match demand (calls) with supply (trunks leading to outsourcers)

• Let call center use its own ACD• Not be responsible for transport (carriers do that)• Get visibility into all calls

– Impossible to do this with TDM– Being in the SIP signaling path throughout the call tells all

• Provide some control over call destinations– Define capacity based trunks to send calls– Perform dynamic load-balancing across trunks (towards agents)– Tackle the set of problems that are unique to Outsourcing

• Provide enterprises with silent monitoring capability

15

Separate Application from Network

Application Servers

CCXML

Application Server executes the logic, and controls the call within the networkthrough CCXML documents

SIP

16

Inside the Network

SIP

Media Gateway

Media Servers

Call Control Gateways

HTTP/CCXML

17

Application Servers and Networks

Application Server

CCXML

Application Server controls calls within multiple service provider networksusing CCXML documents

SIP

SIP

SIP

18

What Happened

• Funded the idea of a Management Gateway– Nobody would think of funding yet another ACD

• Built the team• Built the techology• Went to market• Customers said they needed to be able to deliver calls

from the mid-point based on agent availability• Crucial customer feedback on viability of such a solution

– ACD reports not useful if the full lifetime of the call is not known– Call center metrics get skewed (to excellent!) if the non ACD

queuing time not included

19

The Solution: Version 2

• Added call delivery based on agent presence– State of line not needed

• Found first customers• Incorporated additional market feedback to

include call transfers: blind/consult/conference• Added many additional ACD oriented features

20

Extension to Architecture

Application Servers

CCXML

Added Agent Login and Presence from browser-based desktop applet

SIP

21

Sample Enterprise Implementation

PHX

PHX

OMA

DEN

ATL

MCO

JFK VPOP

Verizon PSTN

3 VZB DS3

VZB MPLS

LAX VPOP

3 VZB DS3

C1

C2

O (CR)

S3

S1 (ES)

S1 (Manila)

T (Manila)

S2 (Dom. Rep)

6Mb

6Mb

9Mb

1.5Mb

6Mb

W

15Mb

MIA

ATLAC

SJCAC

Internet

Sprint MPLS

9Mb

SykesIPLC

6Mb

15Mb

ROC

Manh

Rich

6Mb

6Mb

305555....

4692617[0-3]..

585550....

770555....

4075551[5-9]..

602555....

402555[1-4]...

Along the Way: Learnings

23

Choice of Technologies

• Linux as a server platform– Experience has been par excellence

• Call Control Gateway implementation– SIP B2BUA– Java or C++: C++– Choice of language determined by developers available

• SIP stack– Open source: generally bare metal, required a lot of

maintenance– Commercial stack: we chose Dynamicsoft

• Previous experience with it at Telera (Genesys)

– General experience• Definitely required in-house maintenance• Robustness issues under load, race conditions, etc.

24

Choice of Technologies

• Media Server– Desired to use any available VoiceXML media server– Conferencing features not standards based– Chose Snowshore/Brooktrout/Cantata/Dialogic Linux-

based software media server

• Application server implementation– Java is an excellent choice– Scales and performs very well

• Used Cisco MGs for development

25

Deployment Experiences: Interop

• First implementation in a carrier’s network required interop testing with Lucent Telica

• Making the CCG work well with the Telica (Interop test)

– Straight call worked fine (as expected)• Issues with REINVITES, codec negotiations, header contents

(E.164 number vs label), etc.

– Took longer than most, as interpretation of SIP can cause behavioral issues

• Made changes to CCG adapt faster than Lucent could issue patches

26

Tenant Identification & Trunks

• Every tenant in system had a unique (internal) dial prefix• Number with dial prefix delivered to the CCG which

needed the digit string to identify SIP trunk group and number to be presented

• Developed a re-direct server that performed number translation, and route identification

• Also needed to deploy CCG on separate IP address per tenant

– Carrier billing required this– Linux supports multiple IPs: CCGs instantiated per tenant

27

Post-Deployment Experiences

– Customer encountered one-way audio issues (on some calls)

• Isolated a specific SBC to capture traffic (a difficult task in a carrier network) and proved that RTP stream was intelligible in one direction

• Incidents disappeared after MG software updated

– Session timers• Telica reinvite appearing on our second leg using Cseq 1• Turned out to be a stack bug (and calls got dropped when

enough of these invites got dropped by stack)

28

Learnings: SIP & VoIP Products

• Interop tested with numerous SIP products:– Media Gateways– Phones (hard, soft, G.711, G.729)– Registrars– Hosted IP PBX (Broadsoft, Sylantro)

• Straight call always worked• Reinvites as a B2BUA can be tricky to debug especially in

a carrier network• Voice quality and echo cancellation issues are very thorny

– One fixed through vendor DSP firmware update– A set of such intermittent problems fixed themselves when MG

updated

29

WAN Reliability

• Even the best Internet backbones have packet loss

– Moved to MPLS after achieving a certain volume of customers

• Application tolerance to packet loss– Needed tuning to retry and back off– Balance real time responsiveness with compensation

for network

Summary

31

In Conclusion

• A real Enterprise problem that needed solving

• Could not even think of tackling this problem without SIP

• Interesting problems to solve along the way due to initial maturity of technologies

• A lot has happened in 4 years

32

Summary

• Problem & Background

• Solution Approach

• Along the Way: Learnings

• Summary