source file

Cisco’s Point of View on Virtualization

There is a difference between Virtual and Virtualization, we often mix the

two up with rather confusing results.

Chapter 1: What is Virtual to You?

Virtual is the creation of effective illusions. Virtual LANs are an illusory LAN

giving the semblance of network privacy on shared segments. Virtual Real-

ity is simply 'fake real life.’ Telepresence is a form of Virtual Reality- when

we espouse 'being here is being there' and enable global communications

with a level of interaction and experience that mirrors in-person semantics

we are using an illusion, albeit a digital one. There are Virtual Private Net-

works, not a fake network, but again the illusion of privacy and segmenta-

tion while using shared resources. There are Virtual Applications, where I

have an application running on my mobile device, then it pops on my desk

as I come to work, and my stadium seat as I grab a beer and a dog. ll of

these are examples of the application of the modifier 'Virtual' to a noun that

is generally assumed to be well-understood: Networks, LANs, Reality, etc...

For the sake of this discussion and the general syncing of taxonomy lets

separate the adjective ‘Virtual’ from the use of the noun 'Virtualization'. The

noun-form ‘Virtualization’ expresses in the mind of the general IT populace

something quite different than Telepresence, VLANs, and Storage Proxies

and NFS all of which at some point have claimed to be 'Virtualization'. Lets

assign them to being implementations of the adjective/modifier 'Virtual' and

some of them as technologies that were early innovations that have helped

Virtualization be more efficient, or adopted more quickly. This is not an at-

tempt to denigrate their value or importance, but more importantly find clar-


ity and alignment with the broader IT audience who feels that the term Vir-

tualization is one with more singular definition and purpose. Virtualization,

to most IT audiences, is all about the Virtual Server or Virtual Machine

(used somewhat interchangeably, but we will stick with the virtual machine

to avoid overlapping definitions). The reason for the singularity of the term

Virtualization is because it is about a system and an architecture, derived

from the impact of the VM on IT Architectures. Virtualization is an architec-

ture. In short, what we used to have to do physically we can now do in

software- logically.

Virtualization is about the fundamental shift from hardware definitions of IT

systems to the use of software definitions of the self-same architectures.

Virtualization enables the creation of software architectural definitions by

putting well-understood wrappers or abstractions between components with

a high degree of change, variability, and complexity. Virtualization was fun-

damentally created when the hypervisor was inserted between the server

hardware and the operating system. This abstracted the software identity

of workload and applications from the underlying hardware. This enabled

this new ‘Virtual Machine’ a logical container for an operating system and

its applications to not be inexorably bound to a specific piece of server

hardware. This sea change in the definition of workload has had three

main immediate visible impacts:

1) Rather than binding one O/S and application to each server you can now

run multiple VMs on each server. This is usually one of the initial imple-

mentations of virtualization.


2) The Virtual Machine can move. It can move from one physical server to

another. If you have enough network capacity between the two servers

in question you can move the workload statefully, without dropping client

connections, without changing the addresses in use for global communi-

cations.

3) Homogenization of the x86 server. It used to matter a lot whose server

was bought. The vendor chosen at the point of application development

became in most cases a permanent decision. Because the operating

system included the hardware abstraction layer every device driver had

to be chosen carefully, then the entire package qualified as a single,

monolithic entity, with the application package. In some cases the hard-

ware addresses associated with the server were even used to license

the applications. The advent of virtualization meant that these could be

much more effectively decoupled- removing the vendor lock-in often en-

joyed over application life-cycles.

Chapter-2: The Growth of Virtualization, Breaking your Network

Compounding the value of this abstraction and mobility is the increase in

network capacity and processor performance. Gordon Moore observed

that transistor density was doubling every two years - later branded as

Moore’s Law. For a period of time as transistor density went up engineers

were able to lower the voltages and increase the clock-speed of the pro-

cessor. More transistors meant more memory and more complex process-

ing instruction sets and such and shorter ‘distances’ between them so

things got done more quickly and the chips could run faster and faster. But

at some point in the Gigahertz (billion clock cycles per second) range the


processor vendors ran head-on into the speed of light. In short, the clock

was cycling faster than the ability to signal the output of one instruction

from one side of a chip to another. This meant chips took a bit longer to

make and design as companies started ‘hand placing’ the transistors to en-

sure the the most common outputs were physically close to the next most

common part of silicon that needed that data. It also meant that we were

about at the end of the line on getting faster and faster chips with single

processors.

So while we hit the wall on making faster chips we could still put more tran-

sistors on these chips - that is when we started seeing ‘Dual Core’ and now

‘Quad Core’ chips. Essentially packaging two and then four processors

into one physical die and then placed into a single socket. Most commer-

cial-grade servers are designed with at least two sockets and a direct data

path from one socket to the other and each socket has its own memory.

Every action has an equal and opposite reaction, right? Adding processors

to a socket sounds all well and good but the operating systems and most of

the applications were not designed to take advantage of more than one

processor, in fact a very small subset of all of the code develoed for the x86

space was designed to be ‘parallel’ or ‘multi-threaded’ implying it could

leverage more than one processor concurrently.

The net-net: we have these really fast processors, and most apps can’t

use but 1/4 of them at best. Put two sockets together, and you are at 1/8th.

Being able to put multiple Virtual Machines on each physical server means


that you can use a greater percentage of the CPU resources that are avail -

able for every VM you put on the server.

Let me give a few examples from our own recent experiences as one of the

larger implementors of virtualization technologies:

Recently Cisco acquired Webex, and with this acquisition gained a new

data center in Mountain View, CA. This was a wonderful fringe benefit to

our IT department from our entrance into the Software-as-a-Service space

because our data centers in our main San Jose campus were about out of

capacity. We had built a new data center in Richardson, Texas but have

been finding to our dismay that application migrations of production sys-

tems is the 'long pole in the tent.' So having a local facility, that was not at

max-Q that we could use for handling some near-term capacity require-

ments was a very good thing.

Over the past few weeks our IT department executed a pretty amazing feat:

we moved over 150 servers from the production data centers in San Jose,

to the new Webex facility 10 miles up the street. While this would have tra-

ditionally been a project that would take 6m to 12m on production servers

with the advent of virtualization this project took 8 hours on a sunny Satur-

day afternoon.

Traditionally we would have had to hire movers, plan long outages, deal

with transit-related equipment failures, and re-address applications when

they were received on the far side and cross our fingers hoping nothing

was too hard-coded into the core applications.


We moved 150+ Virtual Machines, in 8 hours. This, by the way, is the net-

work equivalent in bandwidth of every Telepresence System on the Cisco

campus communicating non-stop for over 8 hours as well, about 354 high-

definition real-time video transmissions being moved from one data center

to another. We moved these applications, preserving their addressing,

naming, identity, policy, security, and all that intangible hard-to-see stuff

that is the glue between the IT components but is necessary to IT opera-

tions. A year-long project, now eight hours.

The other example I want to bring up is our own opportunity to embrace

Virtualization and also to thrive from its implementation while delivering in-

creased customer value- simply embrace virtualization for our own appli-

ances where it makes sense.

We sell lots of servers, Cisco calls them appliances. They are an integra-

tion of x86 hardware, usually generic with no special hardware installed,

with Cisco software- sometimes our own O/S, sometimes a hardened

open-source or commercial O/S offering. This model of delivery of network

services has fundamentally worked for our business for the last decade,

starting around 1996 with the introduction of the PIX Firewall and Lo-

calDirector Server Load Balancer. What we gained from this model was

quality and engineering predictability. We could control the firmware ver-

sions, the memory speeds and types, the network interface card, even the

MAC address. In short we used appliances to reduce the amount of vari-

ables and avoid having to build our own O/S that had broad support for ev-


ery variation of hardware that has created the challenging combinatorial

problem we have all seen with driver support.

Cisco ships thousands of servers annually. The question I pose, to us, and

to our customers is this: can they be virtual machines? Can we run a

Voice-over-IP Call Manager in a Virtual Machine? What about a Network

Analysis Module? Netflow Collector? Wireless Controller? WAN Opti-

mization?

Why not? If we can ensure the right resources are available and we can

write our own code to work well with the VM who is to say it cannot be

done?

Cisco gets to reduce the number of servers it buys and sells, packages and

ships. We shift from the key value being the package to the key value be-

ing the software itself and free up engineering resources to focus on deliv-

ering that software value.

Customers in turn reduce support costs, operational complexity and the

number of x86 system varieties they support. They reduce their power

draw, carbon emissions, space consumed, and cooling needed to run our

applications. In the end, a win-win for all.

What prevents this? Nick Carr observed about the legendary inventor

Thomas Edison’s religious attitude towards small power plants, massively

distributed, and DC power distribution the following, “The system that Edi-


son had imagined, and then brought to life, came to be his imagination’s

cage.”

Chapter-3: Virtualization’s Role in IT Architectures

Today’s IT architectures remind me of the Lego block toys of my youth. I

would sit for hours and days combing through a footlocker of parts to find

the exact piece I needed so I could build what my mind imagined.

I never worked off a plan or architecture- I just sat, dreamed, built, and

played. My friend Scott lived across the street from me and we would play

a game of ‘Crash Up Derby’. We would each build a car, then sit on oppo-

site ends of his very smooth front porch, wind up, and push our cars for-

ward into each others. My initial incarnations were Lego-esque versions of

the chariots in Ben Hur bristling with spikes and rods and designed to

somehow Lego the offending wheeled battering ram into oblivion. After

some initial experiences and learning it was time for the second design.

On Mark II I designed a slick delta winged wedge that sat millimeters off the

ground. Scott’s car would hit it, fly up in the air, and land, shattering into

many pieces. Subsequent refinements resulted in a few bruises, cuts, and

our collective grounding as we each found out exactly which buttons could

be pushed, but thats a different story that is much more like ‘Stand By Me’

than ‘Cisco’s Virtualization Story’.

Why wax eloquent on the topic of Lego bricks? Today you buy servers

from one company, operating systems from another, applications from a


third, networks from a fourth, SANs from maybe a fifth, storage from a

sixth, and management from lucky number seven- you get the picture.

IT infrastructure architectures, are often predicated on the proverbial ‘appli-

cation thrown over the fence’ and being told to stand up the infrastructure in

weeks. This is a lot like me sitting down, looking into my war chest of Lego

bricks, and dreaming up something to build with what I know I have on

hand. Most often, there is no plan.

Think of the dots on the top of a Lego brick like a protocol. They enable the

interconnection of the different pieces, they interlock, and if done well they

hold together acceptably. However, my friend Scott found that the laws of

Newton applied to his derby car quite well. When it rode up on the top of

my wedge-shaped racer we discovered flight, a parabolic arc two to three

feet in height. While Scott’s car flew very well, it did not land nearly as

adroitly, impacting in a way that caused all of those pieces to break their in-

terconnection with each other and fall apart. Stress an IT implementation

built without a solid foundation - see what happens - looks similar, but at

least Scott and I never read about ours in a blog back then.

The last lesson we learned from this journey back to fifth-grade was the

part I do not like to admit too often- I lost. Scott found the axle rods. These

were solid pieces of plastic shaped like a ‘+’ that went through the pieces

with the holes in them to allow the wheels to spin. The axles were strong,

Built properly Scott found a way to make a Lego crash-up derby racer that

had no parts to break off, and was impervious to my wedge. This stressed

our organization a bit, causing some conflict between the two of us- I don’t


remember much of the rest, but I think we started talking again a day or two

later.

Scott evolved his architecture. He used a new piece or part, extended the

role of it from an axle to make a wheel turn to becoming the foundation of

his Lego car. From being the part that provided mobility to the part that

provided integrity. Scott found, at age 11, that a good architecture didn’t

need hundreds of different parts - his was simple, elegant, and successful.

Scott FTW.

Virtualization, when coupled to the infrastructure and applications in a

sturdy manner, with real integration, can be simple, elegant, and success-

ful. But it also must be implemented with a purpose, with a plan, with an

architecture.

Chapter 4: What is Cisco doing with and for Virtualization and the Customer Journey

Cisco’s virtualization story? The network is now integrated with the virtual-

ization platform. Cisco has moved from the network being an inhibitor to

the elegant deployment of virtualization, to making our networks embrace

and enable virtualization, to making our networks accelerate virtualization.

Are we done? Never. But at least now we have the protocols, the link-

ages, the connecting bits on top of the Lego bricks, and the smoothly oper-

ating axles that provide the foundation that strengthens everything around

it.


In virtualization we are not the whole, but we enable the whole to be

greater than the sum of its parts and we make everything work better to-

gether - thats what networks inherently do.

An architecture that embraces virtualization, with a foundation that acceler-

ates and enables virtualization is different than what is often deployed to-

day. It’s not that what we have been deploying and building is wrong, its

just that variables have shifted and it is time to look at our architecture dif-

ferently.

Competitive businesses need to be strong, and nimble. We need to be op-

erationally efficient, and innovative. We need to increase business agility,

and corporate productivity.

Virtualization lets us extend the life-cycle of our capital assets, reduce oper-

ating costs, altruistically run cleaner and greener, and put more people on

automating the business and less on manually implementing IT. We move

from months to minutes, and from the physical to the logical.

The journey our customers take is not the same as our roadmap. Not in

the same instance, not in the same order. This was an important learning

point we grasped in the last few months of evolving our virtualization vision.

In some ways while we want it to be ‘all about the network’ we have to real-

ize that ‘its all about the system’ and the network is an important part of that

system.


As Thomas Edison observed when writing later in his life about the inven-

tion of the electrical systems he stated, "It was not only necessary that the

lamps should give light and the dynamos generate current, but that the

lamps must be adapted to the current of the dynamos and the dynamos

must be constructed to give the character of current required by the lamps

and likewise all parts of the system must be constructed with reference to

all other parts, since, in one sense, all the parts form one machine."

Chapter 4.1: Standardize your Infrastructure

The first step then in embracing virtualization must be the standardization

of the underlying infrastructure. Consolidate servers, reduce the number of

disparate platforms, ease IT’s support burden. Bring servers in from

branches- use them more efficiently. Centralize storage, manage informa-

tion and data consistently. Make the investments necessary to allow IT to

scale it’s people resources. Build the infrastructure that will allow IT to im-

plement virtualization successfully in labs and development, then scale it to

production.

Chapter 4.2: Production Virtualization

Production implementations of virtualization is where we really start extend-

ing the life-cycle of our capital assets. Our data centers themselves are the

system. The system view is the building, the cooling, the servers, the stor-

age, the electrical, the flooring, the chillers, the cabling, the racks, the net-

work, the tape archival, the UPS, and the power distribution. The data cen-

ter is the whole, the sum of the parts. This system is being broken every


five years or so because of the same law that increases processor perfor-

mance -- Moore’s Law. If you double the transistor density and do not

lower the voltage you increase the power draw and the heat generation of

each processor. We design the DC system to last, but the facilities infra-

structure is expected to last fifteen years while the IT assets are only ex-

pected to last three to five. The constant refresh of server assets to newer

more powerful processors that take more power, although delivering more

capability and capacity, strains the DC system so that the Data Center as-

a-whole is obsoleted about very five years. This means a mission critical

facility that costs well over $100 Million is obsoleted 30% through its depre-

ciation schedule.

In the move to production virtualization VM usage tends to grow dramati-

cally, and often new servers are brought in with more memory to support

increased VM densities. Be certain to optimize each technology area to

support Virtualization, not just the resident platforms. In this phase of de-

ployment we look to consolidating the SANs, enabling every server to boot

any image. We homogenize the I/O to and from the server just as we ho-

mogenize the server to a smaller number of x86 service platforms. We vir-

tualize our core network services like security and load balancing to enable

all aspects of workload provisioning to be done in software, not just VM

cloning.

Taking a systems and architectural approach to the deployment of virtual-

ization and the data center as a whole allows for the data center to last

longer by using the most efficient infrastructure necessary to support the

workload requirements within their VMs. It also means that workload may


need to be rebalanced across the supporting infrastructure to improve per-

formance,.

Chapter 4.3: Dynamic Virtualization: VM’s Start Moving

This is when VMs start moving, Virtualization becomes Dynamic. This has

caused the biggest change in network architectures in a decade, a regres-

sion almost to the world of the large-flat network where address portability

is enabled. The view of the business is that the ability to balance workload

across multiple machines and move entire applications from one set of de-

vices to another is so valuable that the network architectures must change.

The view of the business is correct. Network architectures must change.

They have to embrace this move, this newfound workload portability across

the racks and rows even across data centers.

This Dynamic Virtualization and concept of workload portability does

change much of what we all have known and applied about networking- but

it also changes operational tasks we have all suffered through - change

control for one. There are some customers who require over 140 signa-

tures to upgrade the software on a switch, because every app owner must

sign off before any shared component is taken out of service. It’s a lot eas-

ier when all the workload has been moved off the shared component prior

to the device ever being taken out of service.

To embrace and accelerate this we have been pioneering new technolo-

gies to enable larger flat networks, while maintaining the concepts of scala-

bility, the protocols for reliability and failover, and the common operating


models we are used to. We created storage systems that let the workload

move without requiring all of the data to directly follow that move. We ac-

cept and embrace this move by embedding our own network operating sys-

tems into the VM so the network identity and profile is consistently ob-

served as the workload moves.

Once workload is portable the question then simply becomes, how far is it

portable? Can I move it from one server to another? Certainly. How about

across data centers? Harder. How about across companies? Impossible...

today.

Chapter 4.4: Virtualization, Utility, and Clouds: A Summation

Virtualization, as an abstraction, divorces the two halves of IT- Infrastruc-

ture and Application. Infrastructure can standardize while Applications can

become increasingly feature-rich, unbound from infrastructure dependency.

Application teams can focus on improving productivity of line-of-business

operations without re-qualifying their applications every time the infrastruc-

ture teams need to upgrade performance capabilities or drive a change de-

signed to improve operational efficiency. With abstraction there is an in-

creased symbiosis between these two halves of IT.

While virtualization enables the application teams to focus on improving

IT’s automation of business process, it also enables the infrastructure

teams to drive operational efficiencies much more rapidly than in an organi-

zation with fixed system dependencies. This abstraction also means that

the applications and their resident operating systems can move, even


across organizational boundaries when operating as part of the right sys-

tem.

When the electrical utility was created in the United States it was initially

envisioned as being deployed in small utilities, direct current powered, and

private plants for the purposes of optimizing the industrialization that was

sweeping all forms of production.

This is strongly similar to the IT architectures of today- deployed by a com-

pany for their own purposes, bespoke tailored to perfectly fit that compa-

nies operating requirements. Off-the-shelf components, assembled on-site

to fit exacting specifications. The creation of a large-scale utility takes ad-

vantage of economies of scale and economical transport to provide service

to a large and distributed population. The further you can efficiently trans-

port a utility the more centralized and efficient the aggregate production

and monetization of that utility can be.

The network provides the most efficient transport possible for IT services to

be delivered, in many cases from anywhere in the world. To further opti-

mize ‘anywhere in the world’ is often taken to mean anywhere that has cost

effective labor, power, tax structures, and information privacy legislation

with stable governments. Network globalization has created tremendous

freedom of choice in where services are delivered from, the next evolution

is to enable freedom of choice in who is providing these services.

The system we envision enables workloads to be portable across compa-

nies, preserving all state necessary for clients to stay connected and all as-


pects of the service to continue uninterrupted - security, load balancing, IP

addressing, accounting, QoS policy, storage, encryption, VPNs, all will

statefully move with their respective workload - across organizational

boundaries. There is a big caveat coming though, if, and only if, the receiv-

ing organization has the capacity to receive the workload and the capability

to meeting the customer’s defined levels of service capability.

This ability to peer workload between organizations is similar to today’s

ability to peer addressing information and network reach-ability state be-

tween organizations- the root technology that enabled today’s Internet.

Workload peering will enable enterprises to seamlessly hand-off virtualized

applications to providers choosing between them based on who can meet

their service levels, capacity requirements, and operating requirements.

The providers will capture new customers faster, provision them more

quickly, and generate revenue months faster than today’s hosting architec-

tures allow.

Workload will flow from business to provider over networks, the most effi-

cient utility transport medium ever invented- the broadest reach, the richest

service, the ‘Ubertility.’ The movement of just one workload from one orga-

nization to another is an order-of-magnitude more data movement than

high-definition video. The movement of entire application suites whether

for business continuance or for capacity on demand or linear growth eco-

nomics will lead to the largest increase in network capacity we have seen in

a decade. What can I say, I liked that part of the 90’s.

source file

Technology