cisco it identity services -...
TRANSCRIPT
Cisco IT – Identity Services Engine (ISE) Deployment
and Best PracticesBassem Khalifé, Cisco IT
CCSSEC-2002
• Introduction
• Security Requirements
• Deployment Strategy
• Design Changes
• Key Challenges
• Learning
• Q&A
Agenda
Cisco IT ISE Global Deployment
Cisco IT ISE Global Deployment (WLAN, LAN)
ISE PSNs Data Center (8) Network Devices Cluster (800+) Auth traffic to ISE PSNs
6,379 3,583
2,232
2,107
Products
•ISE• CITEIS
• Cisco Prime Infrastructure
• Webex
• Jabber
• Splunk
Practice
•Motivation
•Attitude
•Knowledge
•Experience
•Skills
Process
• Product Life Cycle
• Operational Excellence
• Fast IT, Continuous Delivery
• Change Management
• Agile
People
The four P’s
• Introduction
• Security Requirements
• Deployment Strategy
• Design Changes
• Key Challenges
• Learning
• Q&A
Agenda
Capability Major Technical
Outcome
Major Business
Outcome
Secure Guest Network
ION (Internet Only Network)
Simplified single secure
platform (reduce device/server
footprint from 28 to 8)
• High availability
• Secure offering for guests,
partners, and employees
802.1x Auth: WLAN, CVO*,
LAN, and VPN
Complete visibility and control of
devices connecting to the
network
• One scalable policy
enforcement environment
Consistent Assured Network
Access
Scalable enterprise secure
network
• Enhanced Risk Management
• Consistent User Experience
• Improved Operations
*CVO is Cisco Virtual Office, for small office/home office
IT RequirementsIT set out to deliver multiple capabilities with ISE
Access Control
Authentication on
wired & wireless
networks
BYOD
Support Trusted
Device Standard
and enable BYOD
Profiling
Ability to
identify users
and devices on
our network
Endpoint
Protection
Protect the
network
from
infected
devices
Guest Access
Restrict
unauthorized
devices & users to
Internet access
only
Device Control
Secure network
while allowing
mobile device
access*
*Cisco uses a 3rd party MDM and connects to ISE
• Identity of a device on the network
• Quantify the risk
1. Profiling
• User and end device attribution
• Identification of end points on Wireless connections
2. Authentication• Device security
posture identification
• Allows for better policy & security decisions
3. Posture
• Ability to enforce policy decisions based on context
• Untrusted devices have restricted access
4. Enforcement
The Four Stages of a Secure Network
ISE 1.2 Profiling
ISE 1.2 802.1X Auth
WLAN, CVO
ISE 1.3/1.4 802.1X Auth
CVO, Wired, VPN, MDM
ISE 1.4/2.0802.1X Wired Auth Mode
MDM√ √
• Introduction
• Security Requirements
• Deployment Strategy
• Design Changes
• Key Challenges
• Learning
• Q&A
Agenda
“However beautiful the strategy, you should occasionally look at the results.”
Sir Winston Churchill
• Avoid the “Big Bang”• Too many new capabilities to enable in a single deployment.
• “ISE Deployment Bundle” model• Capabilities have been grouped into bundles to enable targeted & manageable deployments
• Multiple clusters consolidated• Pros and cons of single vs. distributed: ISE Limits, Scalability, # EP, Auth, Latency, AD…
• “Start with one cluster and add more if necessary”
• Global Infrastructure Foundation• Deploy global VM infrastructure and ISE servers first
• Guest Network (ION) enabled on a separate deployment to reduce risk
• Enable features (based on “ISE Deployment Bundles”) theatre by theatre
• Use different Virtual IPs by service (e.g., WLAN, LAN, CVO, VPN) for better manageability and ease/speed of control
Cisco IT Deployment Strategy
Cross Functional Ownership for Execution
COO
CTO
SVP IT
VP IT
Any Device Team
SVP
Infra Services
Sr. Dir
Network Services
VP
Ops/Impl
Sr. Dir
Strategy & Security
Security Services
Directory Services
Sr. Dir
Data Centers
Sr. Dir
Arch/Design
SVP
Security & Trust
VP
InfoSec
Security
Requirements
and Policy
Owns Mobile
Devices,
Responsible for
Posture
Enforcement
Provides DC
and Virtual
Infrastructure
Owns and
Manages the
Deployment of
NW Services
Owns and
Operates the
NW
Infrastructure
My team, owns
the ISE infra
and enables
security
services
Owns Active
Directory Infra
and Services
High Level
Architecture
and Design
Operational
Excellence:
99.999%
Availability
Resources for Operations + Resources for Deployment
2 Sr. Engineers + 2 Support Engineers (multi-zone)
• Performance monitoring and tuning, scalability for growth
• HW/SW issues
• Troubleshooting, field issues
• Provisioning of Network Devices, and Users support
• Monitoring and Reporting (ISE reports and Splunk)
• Patch/Upgrade
• Infra/VM issues, Change Request support
• Policy Management
2 Sr. Engineers, 1 Analyst, 1 PM
• Learning and Testing new capabilities
• Service Verification Lab testing and certification support
• Automation of new operational activities
• Limited Availability validation of new features
• Product and platform bug identification and validation
• Data analysis and scalability for new capabilities
• Acquiring knowledge of new capabilities & cross functional environment support
• Documentation
Sample ISE Basic Deployment Roadmap (6 quarters)CY14 Q3 CY14 Q4 CY15 Q1 CY15 Q2 CY15 Q3 CY15 Q4
Fine tune Optimize
Foundation ISE 1.2
Install
ISE 1.3
Upgrade
ISE 1.4
Upgrade
Infra
Design, Proof of Concepts, Data Analysis
Apply
patches
Fine tune Optimize
Network
Guest
Wireless
Monitor
Endpoint Analysis: Wired dot1x MM & Profiling
VPN
Wired
802.1x Authentication
Guest Access
Wireless (WLAN) Auth Deployment
CVO (Home Office) Wireless Auth
VPN Auth
CVO Wired Auth
Limited Sites Wired Auth
Global Wired Auth Deployment
Quarantine/RemediationPosture Assessment & Enforcement (MDM)
Security Group Tagging (SGT)
Advanced Capabilities
Deployment Readiness
Design Engineer Personal Lab
Service Verification Lab
Stage & Pilot
Deploy!
• Production Infrastructure• Network Access ISE 1.3, P2 24 VM servers in one global deployment
• Guest Services ISE 1.2, P13 8 VM servers in one dedicated deployment
• Services• Guest services (ION) (400+ sites, potential 130K+ users & 14K guests per week)
• Internet Only Network access requires pre-registration via ISE guest portal for all users; CWA (central web auth)
• 802.1X Wireless Auth Mode (400+ wlan sites, 90K+ users, ~150K endpoints)
• 802.1X Auth CVO (wireless/wired) (27K Network Devices for home access; ~60K endpoints)
• 802.1X Wired Monitor Mode* (3.5K LAN Switches and Gateways, ~200K endpoints)
• Wireless Policy Enforcement (2 Extranet Partner sites in pilot mode)
• To-date: ~650K Profiled Endpoints; Max of 60K+ Concurrent
Cisco IT ISE Production Deployment Metrics
* Wired Auth deployment currently at 600 NADs
• Introduction
• Security Requirements
• Deployment Strategy
• Design Changes
• Key Challenges
• Learning
• Q&A
Agenda
Original Design for Multiple ISE Deployments
Guest Global
Services
TYO
BGL
AER
RTPALNMTV
HKG
SNG
Secondary ISE PAN/M&T
ISE PSN
Primary ISE PAN/M&T
Primary Secondary Admin Nodes
EIC (6)
MTV AERAll-in-one
Single Global ISE 1.3 Deployment (WLAN, CVO, LAN, VPN)
AER
RTP
ALN
MTV
SNG
Secondary ISE PAN/M&T
ISE PSN
Primary ISE PAN/M&T
Global Deployment: 24 ISE Nodes
20 PSNs; 8 DC (Node Groups)
TYO
HKG
BGL
18,362
9,961
23,969 26,070
32,651
28,124
12,870
5,317
32,856
14,765
40,995
37,481
58,846
51,878
21,384
9,445
-
10,000
20,000
30,000
40,000
50,000
60,000
70,000
AER ALLN BGL HKG MTV RTP SNG TYO
Users
Endpoints/MAC
Users/Endpoints by Node Group
Guestnet Original Deployment
MTV
Sponsor
Portal
(GSS)
internet.cisco.com
Guest Account
Creation
Wireless access
Wired access
NADs AMER
Guest Portal Auth
VMS
Tool
Lobby
Ambassadors
Guest Account
Creation
OEAP
OEAP Device
Provisioning
Script + Store
Wireless access
Wired access
NADs EMEA/APJC
Guest Portal Auth
AER
Before MTV AERAll-in-one
Primary
SecondarySingle Point of Failure
All services will be affected, and
likely to also impact the
secondary node.
Guestnet Redesigned Deployment
MTV
Sponsor
Portal
(GSS)
internet.cisco.com
Guest Account
Creation
Wireless access
Wired access
NADs AMER
Guest Portal Auth
VMS
Tool
Lobby
Ambassadors
Guest Account
Creation
OEAP
OEAP Device
Provisioning
Script + Store
NAD
Configuration
And GSS By
Geo Proximity
Pri
mary
ion-mtv-guest ion-aer-guest
ion-aer-sponsorion-mtv-sponsor
Wireless access
Wired access
NADs EMEA/APJC
Guest Portal Auth
Pri
mary
AER
PPAN Alias
PAN
PAN MnT
MnT
PSN PSN PSN PSN
Primary
Secondary
• Introduction
• Security Requirements
• Deployment Strategy
• Design Changes
• Key Challenges
• Learning
• Q&A
Agenda
ISE Deployment Ecosystem: Building Blocks
ISE(Logical Layer)
ISE (Physical Layer) : ISE Appliance OR VM (Fabric, Compute, Storage)
Network: DNS, NTP, SFTP, UDP, TCP, (& Load Balancers)
Network Access
Devices
Endpoints: Devices,
Users & Supplicants
Enterprise Monitoring: HTTP(S), RADIUS, PEAP, EAP-FAST
User
Provisioning
Mobile Device
Management
Network
Device
Provisioning
ISE Policy
Management
Active
Directory
Call Manager100’s K
10’s K
1 PAN
X PSN
Data
Analysis
(Syslog)
• Trained support team with broad knowledge of environment, across multi time zones
• Troubleshooting using both ISE and Splunk
• Enterprise monitors, load balancer probes, plus other monitors
• Reporting and/or Alerting, covering the ecosystem:• Number of Devices, Endpoints, Auth, failed/passed, by region, device type…
• Profiled devices by group, analysis and validation of profiling results (if used in policy)
• Measure progress of deployment based on pre-migration data
• Splunk ISE app, dashboards providing detailed usage
• Infra utilization, and alerting on CPU, Disk, Memory
Operations Maintenance & Monitoring
Scaling ISE for large scale distributed deployments
• Don’t let replication or misconfiguration become an issue for authentication:
• Tuning the “deployment” (ISE, NADs, and Endpoints)
• RADIUS Accounting
• Profiling
• Authentication(s)
• Latency & Distributed Replication
• Failover & Redundancy
• Tuning the “environment”
• Load Balancers
• Active Directory
Cisco IT Deployment Challenges
Replication Challenges
Iceberg (īs’bûrg’)
(Former state)
ISE Burj
(Current state)
Replication
Issues
Profiling
Error notification
Load Balancer misbehavior
NAD misbehavior
Endpoint misbehavior
Latency
Radius accounting
SNMP errorsMisconfiguration
One change can have huge implications (+ve/-ve):
1. Enabling SNMP and RADIUS Profiling while in an “Accounting Storm”
2. Increasing or decreasing the value of the RADIUS Accounting Suppression
3. Upgrading NADs to recommended OS version
4. Synchronizing timeout settings between NADs and Load Balancers
5. Introducing new “settings” in the product
Snapshot of Key Challenges & Enhancements
Item Owner Impact
Configure ACE for
accounting “stickiness”
Cisco IT High – reduced
accounting traffic
from 6M to 3M
txns per day
Implement eng fix to
enable accounting
suppression
SAMPG High – further
reduction in
accounting traffic
Remove “IP” as a
significant attribute
SAMPG(design
change)
High – removed
traffic from “noisy”
endpoints
Implement WLC OS
updates to fix duplicate
accounting issue
Cisco IT High – reduce
traffic from
wireless network
accounting txns
Implement eng fix for
SNMP polling
SAMPG High – reduce # of
SNMP traffic to
enable CVO
Octo
be
r
September November
Tune the Deployment and the Environment
Configuration
Changes:
NAD and ACE
(LB)
Accounting
Suppression
Fix:
CSCur42723
Removing IP
as Significant
Attribute Fix:
CSCur44879
ISE Global Deployment Profiling Setting PSN Configuration
27K CVO Network Devices configured under 29 subnets
in ISE. SNMP polling is disabled using the new option
“zero” as the devices were not always connected,
resulting in timeout errors (600K), affecting replication.
CSCur95329
Simple fix; Great
value!
Explicitly choose
the polling PSN
Misbehaving Supplicants
EndPoint Profile Auths Per DayCount of
EndPoints% of Total EndPoints
Avg Auths Per Endpoint
Windows7-Workstation 98,394 25,918 20.99 3.8
Apple-iPhone 745,807 17,820 14.43 41.85
Microsoft-Workstation 69,216 16,469 13.34 4.2
Apple-Device 67,167 8,720 7.06 7.7
Workstation 49,834 8,408 6.81 5.93
Android 115,839 5,160 4.18 22.45
OS_X_Mavericks-Workstation 17,529 4,644 3.76 3.77
OS_X_Yosemite-Workstation 17,718 4,276 3.46 4.14
Apple-iDevice 97,862 3,813 3.09 25.67
Android-Samsung-Galaxy-Phone 78,539 3,146 2.55 24.96
Android-Samsung 39,250 3,132 2.54 12.53
Apple-MacBook 14,014 2,883 2.34 4.86
Android-Motorola 70,695 2,226 1.8 31.76
Android-Google 44,835 1,761 1.43 25.46
Wireless auth over 24 hours.
1
3
2
4
ISE Authentication Storm/Meltdown From a “Simple” Change
Network Access
Devices
Endpoints: Devices,
Users & Supplicants
Data
Analysis
(Syslog)
Call Manager
IP Phones
accidentally
enabled for
802.1x auth
1,600 IP Phones
started MAB/dot1x
auth frenzy: 2,600
attempts per day,
per phone= ~4M
Luckily only ~4M auth requests per day due to
limited deployment of a dozen switches. Target scope is 100K IP Phones = 250M Auth
Detection &
troubleshooting
ISE
(Logical Layer)
ISE
(Logical Layer)
Large Deployments – Bandwidth and Latency
PSN PSN
PAN MnT MnTPAN
PSN PSN
PSN
PSN PSN
PSN
PSN PSN
PSN PSN
PSN PSN
PSN PSN
PSN PSN
PSN PSN
200ms
Max round-trip
(RT) latency
between any two
nodes in ISE
1.2/1.3
• Bandwidth most critical between:
• PSNs and Primary PAN (DB Replication)
• PSNs and MnT (Audit Logging)
• Latency most critical between PSNs and Primary PAN.
• RADIUS generally requires much less bandwidth and is more
tolerant of higher latencies – Actual requirements based on
many factors including # endpoints, auth rate and protocols
WLC Switch
RADIUS
Latency Resolution Options
AER
169msALN RTP
TYO
134ms
SNG
186ms
HKG
154ms
MTV
BGL
219ms
45ms
AER
169msALN RTP
TYO
134ms
SNG
186ms
HKG
154ms
MTV
BGL
219m
sX
Option 1
Option 2
+ 45ms
- 45ms
• Introduction
• Security Requirements
• Deployment Strategy
• Design Changes
• Key Challenges
• Learning
• Q&A
Agenda
• Acquire ISE expertise upfront; invest in design
• Fine tune deployment and environment, they must work in tandem
• People, Process, Practice, and Products will derive success – or not
• Follow BU guidelines; will cover 80% of the known challenges• ISE, NADs, Load Balancer configuration
• Requirements from AD, and DNS,
• Profiling configuration
• Bandwidth Calculator
• Listen to the data, and the alerts/alarms
• Do not take the network for granted
• Plan for the unexpected!
Lessons Learned
Participate in the “My Favorite Speaker” Contest
• Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress)
• Send a tweet and include
• Your favorite speaker’s Twitter handle @BJKhalife
• Two hashtags: #CLUS #MyFavoriteSpeaker
• You can submit an entry for more than one of your “favorite” speakers
• Don’t forget to follow @CiscoLive and @CiscoPress
• View the official rules at http://bit.ly/CLUSwin
Promote Your Favorite Speaker and You Could Be a Winner
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.
• Complete your session surveys through the Cisco Live mobile app or your computer on Cisco Live Connect.
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings
• Related sessions
• Introduction
• Security Requirements
• Deployment Strategy
• Design Changes
• Key Challenges
• Learning
• Q&A
Agenda
Hardware CPUs Memory Storage TypeDisk Space
(PAN)
Disk Space
(MnT)
Disk Space
(PSN)
NIC
Speed/Count
Recommended
Min 4 x 2.4 GHz 16 GBDedicated
SAN200 GB 200 GB 100 GB
4 x Integrated
Gigabit NICs
Cisco IT PROD
8 x 2.032 GHz
(Gold
Reservation)
32 GB
Dedicated
SAN (except
HKG/TYO)
600 GB600 GB (1 TB
for tftp)200 GB 1 x 1 GB NIC
NAD Type Profiling Probes
WAN-GW, BB-GW’s DHCP Probe
- ip helpers on VLANs
3750 – LAN Switch
3850 (3.6.2a IOS XE) – LAN Switch
4510
RADIUS Probe
- MAC OUI
- Device Sensor
- ip dhcp snooping
- ip device tracking
SNMP Query Probe
- CDP information
3850 (3.3.4 IOS XE) – LAN Switch
6500
CVO
RADIUS Probe
- MAC OUI
- ip dhcp snooping
- ip device tracking
SNMP Query Probe
- CDP information
WLC/WiSM - Device sensor (DHCP only in 7.2.110 and HTTP in 7.3)
- Disable DHCP proxy (until device sensor available)
WLC
Switch
PSN
PSN
PSN
ALLN
isealln-prd-wlan
isealln-prd-lan
PSN
PSN
PSN
MTVisemtv-prd-wlan
isemtv-prd-lan
ACE Load
Balancer
ACE Load
Balancer
WLC
PSN
PSN
PSN
isemtv-prd-wlan
MTV
isertp-prd-wlanPSN
PSN
PSNRTP
isealln-prd-wlan PSN
PSN
PSNALLN
acs-mtv-p1-1-l
acs-mtv-p1-3-l
acs-mtv-prd
MTV
acs-aln-p1-2-l
acs-aln-p1-5-l
acs-alln-prd
ALLN
Testing High Availability When 1 DC Fails (AER => RTP)
Sample Endpoint Tracking For WLC Migration From ACS to ISE
Thank you