Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU
Jose Manuel Lopez Lujan Senior LMS Coordinator���University of Toronto
Contact information
José Manuel López Luján • Email: [email protected]
• Twitter: @jmanuel_ll
• Blog: http://jose-manuel.me
• G+: http://gplus.to/josemanuel
2
John Calvin • Manager, Data Centres
• Email: [email protected]
University of Toronto
3
79,085 Students 3,229 Academic 5,224 Non-Academic 3 Campuses 7 Colleges
205 Undergraduate programs 79 Graduate programs
4,241,247 Sessions per month
1.5 M Hits/hr peak
2,153,536 Unique Visitors
13,313,110 Visits (Dec 11 Jun 12)
5.0 TB/hr peak
Where are we?
4
Canada 98.0100%
United States 0.7100%
China 0.23%
(not set) 0.1200%
Hong Kong 0.0900%
United Kingdom 0.0700%
South Korea 0.0600%
United Arab Emirates 0.0600%
University of Toronto Statistics
Chrome
Firefox
Internet Explorer
Safari
Android Browser
Opera
IE with Chrome Frame
Mozilla CompaNble Agent
Opera Mini
RockMelt
dem
ogra
phic
s technology
5
App1!
App2!
App3!
Data1!
Collab1!
F5!Load Balancer!
Hitachi 9985!
Infrastructure: Hardware Bb 9.1SP5
Infrastructure: Hardware
6
App1!
App2!
App3!
Hitachi 9985!2 Pools 2 RA!
300G FC !15k RPM!
!
Collab1!
Data1!
SPARC T3-4!4 CPU @ 1.65GHz!16 cores/CPU!8 threads/core!512 threads!512GB of RAM!
LDOMs!32 VCPUs!
64G of RAM!
LDOMs!80 VCPUs!
64G of RAM!
Bb 9.1SP5
7
Infrastructure: Software Bb 9.1SP5
Blackboard Learn 9.1 SP5
Apache 1.3
Pubcookie (DSO)
SSL
Apache 1.3 • No Compression with SSL • No Blackboard Mobile
PubCookie • SSO Solution • Hard to maintain • Custom Authentication Module
8
Looking for service and performance improvements
9
Apache 2.2.x • SSL and Compression working together
Shibboleth • Custom AuthenNcaNon Module for Bb
Bb Mobile
• Possible with Apache 1.3 and PubCookie? • Possible with Apache 2.2.x and Shibboleth?
MinificaNon • Worthwhile without compression?
The Plan
1 out of 4:
10
Using PubCookie Simple to administer
Force Web AuthenNcaNon SSO Page not mobile capable
PROS
CONS
Blackboard Mobile
<Location /webapps/Bb-mobile-bb_bb60>!!satisfy any!!AuthType none!!order deny,allow!!allow from all!
</Location>!
11
Enterprise!LDAP Server!
App4!
mobile.lms.utoronto.ca portal.utoronto.ca
App1!
App2!
App3!Web Login !(pubookie)!
!bbconfig.auth.type=ldap!!
!bbconfig.auth.type=toronto!!
F5
1 out of 4: Blackboard Mobile
Looking for service and performance improvements
Apache 2.2 • Feasible on SP5 yet hard to administer
Shibboleth • Possible with Apache 2.2 yet hard to administer
MinificaNon • Not worthwhile without compression
12
13
OCHO Looking forward to 9.1SP8
The Plan
14
Apache 2.2.x • SSL and Compression working together
Shibboleth • New Authentication Framework
Bb Mobile • Possible with Apache 2.2.x and Shibboleth?
Minification • Worthwhile without compression?
T4-4 • Consolidation and Cryptographic Acceleration
Target version: 2.2.2
15
• Modules
Apache2
16
Compilation
64bit Binary for SPARC
!
CC="cc -m64“ !
CXX="CC -m64“ !
CFLAGS="-m64 -xO2 -DSSL_ENGINE“ !
CXXFLAGS="-m64 -xO2“ !
LDFLAGS="-L/usr/sfw/lib/sparcv9 !
! ! -R/usr/sfw/lib/sparcv9“ !
CCFLAGS="-m64“ !
Shared Modules (DSO)
!
--enable-mem-cache=shared!
--enable-file-cache=shared!
--enable-headers=shared!
--enable-usertrack=shared!
--enable-expires=shared!
Read More
Apache2
Performance.conf
<IfModule mpm_worker_module>!
ServerLimit 1024!
StartServers 341!
MinSpareThreads 64!
MaxSpareThreads 128!
ThreadLimit 128!
MaxClients 1280!
ThreadsPerChild 128!
MaxRequestsPerChild 0!
</IfModule>!
!
Proxy_ajp.conf
<IfModule proxy_module>!
ProxyRequests Off!
ProxyTimeout 3600!
# Shibboleth !
ProxyPassMatch ^(/shib.*)$ !!
ProxyPass /Shibboleth.sso !!
ProxyPass /shibboleth-sp !!
ProxyPass /Shibboleth.sso/Status !!
</IfModule>!
!
!
17
Configuration
# Blackboard secure area !# This will ensure that mod_shib ignore all!# requests except those sent to !# .../execute/shibbolethLogin.!<Location /webapps/bb-auth-provider-shibboleth-bb_bb60/execute/shibbolethLogin>! AuthType shibboleth! Require shibboleth! ShibRequestSetting requireSession 1! Require affiliation ~ ^member@.+$! Require user ~ ^.+$! Require affiliation isstaff! Require affiliation isstudent!</Location>!
Apache2
18
Shibboleth Configuration
# Blackboard Mobile Learn B2 Configuration!# In older installations BBLEARN should be !# changed by bb_bb60!<Location /webapps/Bb-mobile-bb_bb60>! AuthType shibboleth! ShibRequestSetting requireSession 0! Require shibboleth! Require user ~ ^.+$! Require affiliation ~ ^member@.+$! Require affiliation isstaff!</Location>!!
Apache2
Compressed Weight
19
Total Weight
1036.9K 265.7K
Web Compression + SSL
74.3%
VS
vS
20
force to native
© Blackboard Mobile: h`p://help.blackboardmobile.com
web
Blackboard Mobile Learn Authentication Type
Shibboleth and LDAP
21
Implementing a New Authentication Framework
LDAP Server!
App4!
mobile.lms.utoronto.ca portal.utoronto.ca
App1!
App2!
App3!Shibboleth !2.4.3!
F5
Hostname RestricNon Provided by the New AuthenNcaNon Framework
Provider: Toronto Shibb Auth Provider: Toronto LDAP Auth
Shibboleth and LDAP
Shibboleth LDAP
22
Implementing a New Authentication Framework
mobile.lms.utoronto.ca portal.utoronto.ca
Minification
23
Real path MinificaNon MinificaNon CR MinificaNon t $BBHOME/docs 835,860.00 50.93% 3.1886 $BBHOME/webapps/blackboard 183,477.00 58.46% 0.6999 $BBHOME/webapps/assessment 51,225.00 58.01% 0.1954 $BBHOME/webapps/discussionboard 30,919.00 35.52% 0.1179 $BBHOME/webapps/gradebook 277,527.00 54.81% 1.0587 $BBHOME/webapps/caliper 119,764.00 48.56% 0.4569 $BBHOME/webapps/portal 27,595.00 54.81% 1.0600 $BBHOME/webapps/cms+xy 49,532.00 52.59% 0.1889 $BBHOME/webapps/wysiwyg 99,681.00 52.43% 0.3803 $BBHOME/webapps/webeq-‐plugin 15,354.00 52.43% 0.3800 $BBHOME/webapps/taglibs 44,054.00 52.43% 0.3800 $BBHOME/webapps/* 4,936.00 52.43% 0.3800 Grand Total 1734988.00 8.11
1694.32K MR ~ 52.0%
Savings on payload
Minification
• Prematurely released on SP5
• Released on SP8 as certified.
• Implementing YUI Compressor Library
• Grouping and minifying on-the-fly (inside JVM)
• Enabled by default on SP8
24
Blackboard JS Grouping Tool
## Whether related JavaScript files should be grouped together ## ## for be`er HTTP performance ## bbconfig.javascript.group.files=true
Read More.
<script type="text/javascript" src="/branding/__js__/C131DA0400D29916A81632A83B91BAD2.js?v=9.1.50119.0"></script>
25
Minification Blackboard Grouping Tool
Firebug console output sample
Read More.
26
Solaris Cryptographic Framework (SCF)
27
© Sun Microsystems: Using The Cryptographic Accelerators in the ULTRASPARC T1 and T2 Processors.
28
conf/pkcs11.conf
SSLCryptoDevice pkcs11!
64bit Binary for SPARC
!
CC="cc -m64“ !
CXX="CC -m64“ !
CFLAGS="-m64 -xO2 -DSSL_ENGINE“ !
CXXFLAGS="-m64 -xO2“ !
LDFLAGS="-L/usr/sfw/lib/sparcv9 !
! ! -R/usr/sfw/lib/sparcv9“ !
CCFLAGS="-m64“ !
Read more.
Linking Apache2 binary
Solaris Cryptographic Framework (SCF)
29
OCHO Current environment 9.1SP8
Infrastructure: Hardware
30
App1!
App2!
App3!
Hitachi 9985!2 RAID 6 Arrays!
2 TB x 7200 RPM SATA!Carved into 192 GB Ldev!
!
Collab1!
Data1!
SPARC T4-4!4 CPU @ 3.0 GHz!
8 cores/CPU!256 threads!512GB of RAM!
4 x LDOMs!24 vCPUs!
64G of RAM!
1 x LDOM!56 vCPUs!
120G of RAM!
Bb 9.1SP8
App4! 2 x LDOMs!24 vCPUs!
32G of RAM!App5!
31
Infrastructure: Hardware SPARC T4-4 and Oracle VM Server for SPARC v2.2
64 64 64 32 32 120 64 8
Collab1 App1 Ap2 App3 App4 App5 Data1 IO/Controller Free
Memory
32
Infrastructure: Hardware
32 32 32 16 16 56 32 24
Collab1 App1 Ap2 App3 App4 App5 Data1 IO/Controller Free
SPARC T4-4 and Oracle VM Server for SPARC v2.2
vCPUs
33
Infrastructure: Hardware Live Migration
Read More.
Source
Target
34
Infrastructure: Software Bb 9.1SP8
Blackboard Learn 9.1 SP8
Apache 2.2.2 64 bit SPARC
Shibboleth (DSO)
SSL
Apache 2.2.2 • Compression with SSL • SSL Offloading – PKCS11
Blackboard Mobile • NaNve AuthenNcaNon
AuthenNcaNon Providers:
LDAP + Shibbholeth
Shibboleth • LDAP
Performance
35
Benchmark
requests were sent sequentially with different concurrency levels 50k
Proxy SSL
AJP <
https://server/webapps/portal/healthCheck
deflate Concurrency Apache 2.2.2 Apache 2.2.2
SSL-‐H, AJP SSL-‐H, AJP, COM 10 1230.59 1143.12 100 1962.52 1704.3 200 1699.73 1625.22 500 1870.60 1075.2 1000 1214.95 1173.457 2000 1129.87 1234.44
@ 1k request/sec > process ~1.2K req
Performance
36
24%
56%
17%
2% 1% 0% 0% 0%
0 -‐ 1
1 -‐ 3
3 -‐ 7
7 -‐ 13
13 -‐ 21
21 -‐ 35
35 -‐ 60
60+
Load Times
Avg. Page Load Time:
2 .44 SEC
