sc10palladinovoip

8/2/2019 SC10PalladinoVoIP

http://slidepdf.com/reader/full/sc10palladinovoip 1/56

Michael PalladinoUniversity of Pennsylvania

ACUTA 39th Annual Conference

San Antonio, TexasApril 20, 20101

A business case for VoIP:How to do more for less



Today’s objectiveTo discuss the University of Pennsylvania’s PennNet Phone project as a loose case study to demonstrate that an open source/open standard approach to

campus‐wide

VoIP

can

be

cost

effective

and

successfulTo help you make similar business decisions by

sharing our lessons learned

2



Today’s Main FocusBackground and historyOptions evaluatedProject goals, architecture, business objectives and risks

Deployment strategyResources, accountability and measurement

Current statusLessons learned and conclusion

3



4



5

PennR1 research university, 12 undergraduate and

graduate schools

including

a medical

school

and hospitals in Philadelphia 22,000+ students7500 students live on campusHealth system has separate management

IT division mostly outsourced (15,000 customers)Operating budget is over $3B



Central IT organization

Robin Beck,

CIO9 units and over 300+ FTE (blended workforce)

Administrative systems

Data, voice, video networks & servicesSupport services for Local ProvidersIT security, standards and policies

Provide technical leadership, infrastructure, standards, support and services centrally for

campus

Information Systems Computing

6



100 FTE/$25M budget/98% chargeback24,000 phone lines/10,000 voicemail users55,000 IP addresses/17,000 email accounts46,000 Ethernet ports

9,000 analog

video

connections/70

channels

60 servicesOperate MAGPI, Internet2 RON in PA, NJ and DE

Customers: Princeton, NJ Edge, ESNet, NOAA, UDel, Penn, Lehigh, Temple, CHOP, Philadelphia OrchestraEducational consortium in PA, recently received $99M

of federal

stimulus

monies

to

build

statewide

network

Networking & Telecommunications

7



Leadership ‐Long View25+ years at PennVery determined

ResNet (1988 to 1997)

Next Generation

PennNet

(1999/$15M)Preparation for our VoIP project started over 15

years ago

Articulated technical vision in PennNet 21: Technology Vision for 21 st Century PennNet in the mid 90’s

Vision/strategic plans/annual

goals

8



Leadership: InfrastructureWe knew by 1996 that in order to meet the

research, academic and administrative needs of Penn in the 21st century, we needed to build a very robust IP infrastructure that both enhanced capacity and increased reliability of our data networks to support future IP‐video and IP‐telephony systems and servicesWe began building our Next Generation

PennNet in 19999



NGP Infrastructure5 data centers with redundant single mode fiber10 gig routing coreDual gig to critical buildings

Gig capable desktop connectionsServices 99.95% reliability

Email, web,

directory,

streaming

video,

ACD,

etc.

Systems with 99.99% reliabilityDNS, DHCP, Authentication, etc.

10



Leadership: OrganizationalI have always believed that organizational

convergence was

equal

to

or

more

important

than

technical convergence for successful integrated service delivery

To that end, we organizationally converged in 2000Network, Video and Telecommunications staff combined under one leadership

Cross‐trained staff to create hybrid data/voice consultants and engineers Began development and eventual deployment of converged service delivery to customers

12



Background: VoIP Staff discussed the broad concepts as early as 1999

From 2000 until 2004 basic protocol and server implementation research was done

By 2004, a model for using existing open source/open standard solutions had been demonstratedTo be sure of our approach, we evaluated other VoIP

solutions in

the

last

5

years

All were found to be expensive and/or bad technical fits with our unified communications plans

13



Drivers for Moving to VoIPWe began investigating VoIP to lower costs,

enhance customer

service

and

increase

functionality

Eliminate the annual cost of $200k of upgrading the

aging cables/conduitsEliminate the $200k annual cost of doing MACs

Speed up

the

MACs

Current moves can take 5‐10 days on the analog infrastructure versus less than 2 hours on VoIP

14



Drivers for Moving to VoIPTo eventually eliminate all contracted staff working on Centrex, for an annual savings of $330kTo reduce usage costs for telephone calls by $400k annuallyTo give the Penn community additional functionality (like voice mails going to email) as we move more to unified communications and an ever ‐increasing mobile environmentTo lower the cost of voicemail and have full

redundancy15



VoIP Options Evaluated Vendor solution (Cisco Call Manager)

Advantages Complete end ‐to ‐end solution

Quicker deploymentFeature rich

DisadvantagesExpensiveVendor dependence

16



VoIP Options Evaluated Carrier ‐Class IP Telephony Software

Broadsoft’s Broadworks

product

locally

hostedAdvantages

Feature richQuicker deployment

Disadvantages ExpensiveSecurity model that was incompatible with an

open network

17



VoIP Options Evaluated Outsource (Verizon’s Hosted IP Centrex)

AdvantagesCurrent vendorKnown product (Broadworks)

Feature richDisadvantages

Not a fully outsourced solutionLacking integration with existing UC servicesProvisioning tools did not scale to our support modelTiming not right (too bleeding edge?)Relatively expensive

18



VoIP Options Evaluated Insource (Open source/open standards)

Advantages

Lower costsCompetition on setsNo license fees

Vendor independence

SIP protocolCould leverage in house engineering staff and in house security and ID management facilities

Directories & user authentication19



Leadership: Open Source

Our staff

has

been

successful

with

other

open

source,

open standard initiatives

Email (Zimbra)

Network management (Nagios)Instant Messaging (OpenFire/Jabber)

Web (Apache/MySQL)Authentication (MIT Kerberos)DNS (BIND)

20



VoIP Options Evaluated Insource (Open source/open standards)

Disadvantages Not feature ‐rich

Time to develop slowDevelopment costs

Development efforts pull resources from other projectsCost of contractors to backfill on other projects

Despite this, in 2005 we launched own VoIP service,

initially called iPhone21



PennNet Phone Goals Replace legacy Verizon Centrex service which

supports over 20,000 administrative lines and the legacy voicemail service (Octel) that supports over 10,000 customers with open source/open standard , production ‐caliber services that are feature ‐rich and cost ‐effective

22



PennNet Phone ArchitectureThe underpinnings of our service is SIP Express Router (SER) as the SIP proxy and registerAsterisk plays a key role, but not as a soft PBX

Asterisk serves only as our voicemail serverServers and gateways replicated for high availabilityPolycom Phones and SIP Trunks from Verizon

Network architecture is layer 2 Quality of Service, separate vLANs & subnets in building networksInitially, campus ‐wide wired voice and voicemail

23



Business ObjectivesLeveraging open source solutions, deliver feature ‐rich, highly‐available telephony solutions at a significantly lower price point

Reduce telephony operating budget over $1M

Return savings to schools to fund strategic initiativesResearch, new buildings, increasing financial aid, etc.

Aggressively pursue additional cost savings through standards based technologies like SIP TrunkingIntegration with other centrally provided IT services (IM,

email &

voicemail)

to

strengthen

our

product

offering

24



Business ObjectivesEliminate legacy infrastructure and services as soon as practical to reduce costs/direct energies to new initiatives

Copper building ‐to ‐building infrastructure (FY’15)

Fire alarms/circuits/dry pairsCard readers/environmental controls

Centrex services

(FY’13)Octel voicemail services (FY’13)

25



Risks of not proceedingInfrastructureAging conduits and cable

Pulp cables/terracotta conduits

Long outages due to no analog redundancy

Very high costs to replace ($3M+)Centrex & Octel services

How long will service be supported and at what costs?Customers (Vonage)

Push by customers as retail VoIP offerings are widespread

Constant need to demonstrate leadership and business value to customers and executive leadership26



VoIP Project FundingWhere did funding for project come from?

We funded it ourselves out of our operating budgetTelecommunications revenue exceeds our expenses

Initially directed excess to fund our Next Generation PennNet Project, more recently R&D on VoIP solutions

We piloted various solutions and did associated ROIs

27



PennNet Phone ‐ ROI

28

$‐

$1,000,000

$2,000,000

$3,000,000

$4,000,000

$5,000,000

$6,000,000

$7,000,000

FY'08 FY'09 FY'10 FY'11 FY'12 FY'13 FY'14 FY'15 FY'16

Investment

Savings



29

Leadership ChallengesThere was a lot of activity with limited progress

Focused leadership was requiredWe were about to embark on a multi ‐year, multi ‐

million dollar project with VoIP/unified communications as our top departmental priority

We hired

a project

manager

who

immediately

set

up project teams and developed a deployment strategy



Deployment StrategyFocus teams on specific deliverables

Perform feature

gap

‐analysis,

define

and

prioritize

required feature developmentDeliver features in phases to control costs/manage risks

Deploy “easy” phone conversions firstSingle line

No infrastructure upgrades required (closet work, wiring)

Offer incentives to convert Free installations and wiring

30



Deployment StrategyDaily/weekly performance management to measure quality assurance across service delivery departments (installations, operations, billing)

Involve campus support groups to increase their project ownership

Monthly/weekly training sessions to support rolloutGain full upper management supportUse existing best practice processes and procedures

31



32

PennNet Phone Project TeamsSteering

Committee

Technical Process BusinessFacilities

NetworkArchitecture

ApplicationArchitecture

TelephonyArchitecture

InstallationsOperations

IntakeSupport

Billing

Provisioning

Communications



Required ResourcesSteering Committee has guided the various teams throughout the project’s history

Quality control, product development, overall roadmap, high ‐level decisions, etc.

The initial resource commitment funded our engineering group’s R&D

As the project moved out of the lab and into production, significantly more resources were necessary from all parts of the organization

33



Required Resources The entire department participates in the project

Administrative staff/Asset ManagerInfrastructure/Project Management

OperationsNetwork ManagementEngineeringSystems and ServicesBilling

Central Provider Support 34



Key Stakeholders (Internal)Executive managementComputing directors from the schools in early stages Support

Centralized support deskLocal Support ProvidersTelephone Support Providers

Stakeholders that help fuel our broader deploymentFacilities Management

Senior business leaders in schools35



Strategic Partnerships (External)Formed strategic partnerships with several vendors and

service providers

Verizon provides PRI and SIP Trunking connectivity to the PSTN

Penn has contributed to and benefited from the development efforts of the Asterisk open source communityOneBIZTONE has played a role in the feature development and design

We have utilized feature ‐rich VoIP sets from Polycom36



AccountabilityClear project goals tied to the overall business objectives

5000 total lines by FY ‘1010,000 total lines by FY ‘11

15,000 total lines by FY ‘1218,000 total lines by FY ’13

ReportingWeekly meetings and reports on progress Monthly reports to customers

Report quarterly to executive leadership on goals and cost savings37



MeasurementIt is a crucial aspect of the success of our project

We measure

many

components

regularlyVoIP and voicemail conversions

Phone line and voicemail disconnects

VoIP trouble ticketsInstallations and maintenance tickets vs. SLA

Mean time to install and repairSystem and service reliability

Data is used to make business decisions, manage

inventory, drive

feature

and/or

service

enhancements

38



Nearly 4,500 production VoIP customers todayDeveloped easy to use web interface to access feature selections (VM options, presence, etc.)Certified the Juniper Session Border Controllers Piloting redundant SIP Trunks from Verizon

Should port over all new conversions in SeptemberShould port over all existing PennNet Phone

customers in OctoberOperating 1.6.1.17 of Asterisk voicemail with 4300 customers

95% adoption rate vs. 40% due to lower costs39

PennNet Phone Current Status



Using web front ‐end to Remedy for trouble tickets, techs self dispatchCustomer programs in place for VoIP support providers

Weekly recurring lunch ‐time learning sessionsVoIP Special Interest Group meetings held quarterly

Strong customer input is guiding product development direction and continued service ownershipExperimenting with IM for faster response times for customer requests/problemsExploring soft ‐clients, wireless phone apps and FMC

No current

plans

to

do

VoIP

for

students.

Still

have

landlines for emergencies. Exploring other options.

PennNet Phone Current Status

40



PennNet Phone Roadmap

41

Time Line FY10 FY11 FY12 FY13

PennNet Phone Roadmap 5,000 sets 10,000 sets 15,000 sets 18,000 sets

SIP Trunking Certify Verizon ITSP

ITSP in Production ITSP ITSP

Asterisk Application Roadmap Rollout 1.6

Rollout Advanced

Features

Migrate Octel Users to Asterisk

Migrate Octel Users to Asterisk

& ACD

Octel Roadmap In service In service In service Retire Octel



Lessons LearnedWe encountered major obstacles in the early phases of the project contributing to low customer adoption rates including

Service stability

Limited service features when compared with Centrex and Octel

High costs

42



Lessons LearnedIn response, we had to slow down deployment until we

Hardened the infrastructure including the PSTN gateways and PRIs to increase reliability and

enhance capacityDeveloped comparable or superior featuresEvaluated and installed lower ‐cost phones

Developed tight processes to increase customer satisfaction and increase migration productivityHeld town meetings to get frank customer

feedback43



Lessons LearnedOur initial IT support model was not well‐ received by the existing Telephone Support Providers in the schools

We needed to adapt to incorporate this support into our model

Large‐scale development efforts take more time than initially

projected. At nearly every step of the way technology components took longer to deploy

Features

New setsHighly reliable PRIsHighly reliable gatewaysSIP Trunking

44



Lessons LearnedWe prepared for an accelerated rollout of our PennNet Phone service by surveying our ACUTA peers on the methods they used to

support large ‐scale VoIP installations Some key changes influenced by our ACUTA peers:

Discontinuing the end user requirement to sign a “Terms of

Service” agreementLeveraging our back office data to assist us with planning the installations in terms of “easy” and “difficult” buildings

Establishing a training program for support providers and customers

By streamlining the installations, support and training services

offered, we were able to expand our PennNet Phone program from small group conversions to large ‐scale building ‐wide

45



Lessons LearnedLarge‐scale projects will challenge your organizational

resources

Tighter accountability may challenge your organization’s cultureMatrix management especially with tighter accountability

will likely challenge your staff and take some time for them to acceptTaking a project from the lab to production is challenging,

especially when a service is under constant developmentPhased feature releases and more intensive testing helped

Data staff do not always understand voice requirements 46



Lessons LearnedUpper management involvement was required for high ‐level resource reallocation

High priority engineering projects were competing for the same technical resources as PennNet

Phone development efforts Projects were reprioritized, resources reallocated and different approaches explored to manage expenses

In these tight fiscal times when additional staff resources are limited, this is proving to be a continuous process for us

47



Lessons Learned (Positive Outcomes)No increase in headcount was necessary with this large insourcing project

Some areas demonstrating significant productivity increases

Some costs lower than expected A large scale project with nearly all staff involved –especially when going well‐ can be energizing for the staff (common cause/focus)Interesting projects challenge/engage and help retain staff

48



Lessons Learned (Business Outcomes)Biggest outcome will be the $1M+ reduction in expenses to our customers Additional savings (over the lower PennNet

Phone costs) are being realized in the elimination of unused lines, sets, voicemail and circuits, as we do comprehensive billing auditsWe expect to lose 2000 ‐3000 linesThis clean up may be financially “painful”

49



Lessons Learned (Business Outcomes)Since Telecommunications surpluses have fueled

many of our departments other initiatives (NGP), this could be financially challenging for us in the futureWe are moving to eliminate all cross ‐subsidies on all 60 services to mitigate this problem

50



ConclusionA large ‐scale development effort in VoIP can

be cost

‐effective

in

the

long

‐runAdditional management cycles and senior

leadership may be necessary for successful

resource managementCustomer support and project ownership are necessaryHaving a product that delivers more functionality at a lower cost point clearly helps

“the sell” 51



ConclusionOther project enhancements are useful

New on ‐line ordering and trouble tickets systemsQuicker mean time to repair and install

Faster, lower cost MACsNew support, training and billing systems

52



ConclusionHaving clear business objectives that are aligned with the University mission gets executive leadership attention

$1M+ for school ‐based initiativesYou will run into many leadership challenges

Upper management supportStaff resourcesTechnical challengesProject re ‐prioritizationEven “mutiny”

53



ConclusionAll of which will keep your role very interesting, but never lose sight of the end business goals and objectives

We generally look to outsourcing to save money. By doing a good business case and

ROI, insourcing may save you moneyYou too can do more with less!

54



ReferencesTalks:Deke Kassabian ‐ February 2005 – VoiceCon Spring 2005 Conference, Open Source VoIP SoftwareMelissa Muth and Dawn Augustino–2006 – EDUCAUSE Mid ‐ Atlantic Regional Conference ‐ Migrating from Centrex to IP Telephony at PennMelissa Muth – April 2006 – ACUTA Spring Seminar ‐ VoIP Security Issues in a University Environment Michele Narcavage – 2006 ACUTA – Migrating from Centrex to IP Telephony at PennMichael Palladino – 2006 ACUTA – Leadership Challenges for the Future: Open Source VoIPMichael Palladino and Naila Machado ‐ July 2006, ACUTA ‐ A Funding Model for a Converged Network Deke Kassabian and Michael Palladino – 2006 – Educause National Meeting Production Quality Open Source VoIP: Next Generation Telephony at Penn – October 2006

Deke Kassabian

– Spring

2007

– VoiceCon

‐Open

Source

VoIP

for

the

Enterprise

– March

2007 Deke Kassabian – Spring 2007 – Internet2 Member Meeting – Production Quality Open Source VoIP: Next

Generation Telephony at Penn – April 2007 Deke Kassabian – October 2008 – Speaker – Asterisk World – VONMichele Narcavage – 2008 ACUTA – Living in a Hybrid World of TDM & VoIPMichael Palladino – October 2009, ACUTA – Strategic Ways to Address Today’s Funding Challenges

Press:Deke Kassabian – July 2007 – Quoted in Network World, Open Source VoIP Hits the Ivy LeagueDeke Kassabian – August 2007 – Wrote article; “Is Enterprise ‐Level, Open Standard, Open Source VOIP A Myth?” Business Communications Review Michele Narcavage – April 2008 – Wrote article; “Living in a hybrid World: Migrating from TDM‐to ‐VoIP" Telecom Reseller

55



Conclusion Thank you !

Michael Palladino

Associate Vice President, ISC Networking & Telecommunications3401 Walnut Street, Suite 221A

Philadelphia, PA [email protected] 215 ‐898 ‐9386

sc10palladinovoip

Documents