taking blackboard to the next level: apache 2.2, crypto acceleration, shibboleth, and bbmobile for...
DESCRIPTION
The University of Toronto wanted a faster Blackboard Learn environment and needed to squeeze the most out of its hardware. Apache 1.3 that Blackboard bundles with its BB Learn was not up to the job. By upgrading to Apache 2.2, a new world of options become available. Now, Shibboleth 2 as a Single Sign-on, LDAP as a native authentication scheme for Mobile Learn and OpenSSL PKCS#11 Cryptographic support are all realities for Blackboard.The PKCS#11 support was particularly important, as is leverages the Solaris Cryptographic Framework (SCF) and transparently offloads cryptographic operations to available hardware providers included in the new Oracle T4 chip. UofT migrated all of its Blackboard VMs to a single Oracle SPARC T4-4 after upgrading to BB9.1SP8. The results were remarkable. A single box, 5 rack-U in height, delivers Blackboard to 80k users. Response times are down by 60% and we have no need for SSL-offload at the load-balancer.TRANSCRIPT
![Page 1: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/1.jpg)
Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU
Jose Manuel Lopez Lujan Senior LMS Coordinator���University of Toronto
![Page 2: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/2.jpg)
Contact information
José Manuel López Luján • Email: [email protected]
• Twitter: @jmanuel_ll
• Blog: http://jose-manuel.me
• G+: http://gplus.to/josemanuel
2
John Calvin • Manager, Data Centres
• Email: [email protected]
![Page 3: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/3.jpg)
University of Toronto
3
79,085 Students 3,229 Academic 5,224 Non-Academic 3 Campuses 7 Colleges
205 Undergraduate programs 79 Graduate programs
4,241,247 Sessions per month
1.5 M Hits/hr peak
2,153,536 Unique Visitors
13,313,110 Visits (Dec 11 Jun 12)
5.0 TB/hr peak
Where are we?
![Page 4: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/4.jpg)
4
Canada 98.0100%
United States 0.7100%
China 0.23%
(not set) 0.1200%
Hong Kong 0.0900%
United Kingdom 0.0700%
South Korea 0.0600%
United Arab Emirates 0.0600%
University of Toronto Statistics
Chrome
Firefox
Internet Explorer
Safari
Android Browser
Opera
IE with Chrome Frame
Mozilla CompaNble Agent
Opera Mini
RockMelt
dem
ogra
phic
s technology
![Page 5: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/5.jpg)
5
App1!
App2!
App3!
Data1!
Collab1!
F5!Load Balancer!
Hitachi 9985!
Infrastructure: Hardware Bb 9.1SP5
![Page 6: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/6.jpg)
Infrastructure: Hardware
6
App1!
App2!
App3!
Hitachi 9985!2 Pools 2 RA!
300G FC !15k RPM!
!
Collab1!
Data1!
SPARC T3-4!4 CPU @ 1.65GHz!16 cores/CPU!8 threads/core!512 threads!512GB of RAM!
LDOMs!32 VCPUs!
64G of RAM!
LDOMs!80 VCPUs!
64G of RAM!
Bb 9.1SP5
![Page 7: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/7.jpg)
7
Infrastructure: Software Bb 9.1SP5
Blackboard Learn 9.1 SP5
Apache 1.3
Pubcookie (DSO)
SSL
Apache 1.3 • No Compression with SSL • No Blackboard Mobile
PubCookie • SSO Solution • Hard to maintain • Custom Authentication Module
![Page 8: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/8.jpg)
8
Looking for service and performance improvements
![Page 9: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/9.jpg)
9
Apache 2.2.x • SSL and Compression working together
Shibboleth • Custom AuthenNcaNon Module for Bb
Bb Mobile
• Possible with Apache 1.3 and PubCookie? • Possible with Apache 2.2.x and Shibboleth?
MinificaNon • Worthwhile without compression?
The Plan
![Page 10: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/10.jpg)
1 out of 4:
10
Using PubCookie Simple to administer
Force Web AuthenNcaNon SSO Page not mobile capable
PROS
CONS
Blackboard Mobile
<Location /webapps/Bb-mobile-bb_bb60>!!satisfy any!!AuthType none!!order deny,allow!!allow from all!
</Location>!
![Page 11: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/11.jpg)
11
Enterprise!LDAP Server!
App4!
mobile.lms.utoronto.ca portal.utoronto.ca
App1!
App2!
App3!Web Login !(pubookie)!
!bbconfig.auth.type=ldap!!
!bbconfig.auth.type=toronto!!
F5
1 out of 4: Blackboard Mobile
![Page 12: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/12.jpg)
Looking for service and performance improvements
Apache 2.2 • Feasible on SP5 yet hard to administer
Shibboleth • Possible with Apache 2.2 yet hard to administer
MinificaNon • Not worthwhile without compression
12
![Page 13: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/13.jpg)
13
OCHO Looking forward to 9.1SP8
![Page 14: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/14.jpg)
The Plan
14
Apache 2.2.x • SSL and Compression working together
Shibboleth • New Authentication Framework
Bb Mobile • Possible with Apache 2.2.x and Shibboleth?
Minification • Worthwhile without compression?
T4-4 • Consolidation and Cryptographic Acceleration
![Page 15: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/15.jpg)
Target version: 2.2.2
15
• Modules
![Page 16: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/16.jpg)
Apache2
16
Compilation
64bit Binary for SPARC
!
CC="cc -m64“ !
CXX="CC -m64“ !
CFLAGS="-m64 -xO2 -DSSL_ENGINE“ !
CXXFLAGS="-m64 -xO2“ !
LDFLAGS="-L/usr/sfw/lib/sparcv9 !
! ! -R/usr/sfw/lib/sparcv9“ !
CCFLAGS="-m64“ !
Shared Modules (DSO)
!
--enable-mem-cache=shared!
--enable-file-cache=shared!
--enable-headers=shared!
--enable-usertrack=shared!
--enable-expires=shared!
Read More
![Page 17: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/17.jpg)
Apache2
Performance.conf
<IfModule mpm_worker_module>!
ServerLimit 1024!
StartServers 341!
MinSpareThreads 64!
MaxSpareThreads 128!
ThreadLimit 128!
MaxClients 1280!
ThreadsPerChild 128!
MaxRequestsPerChild 0!
</IfModule>!
!
Proxy_ajp.conf
<IfModule proxy_module>!
ProxyRequests Off!
ProxyTimeout 3600!
# Shibboleth !
ProxyPassMatch ^(/shib.*)$ !!
ProxyPass /Shibboleth.sso !!
ProxyPass /shibboleth-sp !!
ProxyPass /Shibboleth.sso/Status !!
</IfModule>!
!
!
17
Configuration
![Page 18: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/18.jpg)
# Blackboard secure area !# This will ensure that mod_shib ignore all!# requests except those sent to !# .../execute/shibbolethLogin.!<Location /webapps/bb-auth-provider-shibboleth-bb_bb60/execute/shibbolethLogin>! AuthType shibboleth! Require shibboleth! ShibRequestSetting requireSession 1! Require affiliation ~ ^member@.+$! Require user ~ ^.+$! Require affiliation isstaff! Require affiliation isstudent!</Location>!
Apache2
18
Shibboleth Configuration
# Blackboard Mobile Learn B2 Configuration!# In older installations BBLEARN should be !# changed by bb_bb60!<Location /webapps/Bb-mobile-bb_bb60>! AuthType shibboleth! ShibRequestSetting requireSession 0! Require shibboleth! Require user ~ ^.+$! Require affiliation ~ ^member@.+$! Require affiliation isstaff!</Location>!!
![Page 19: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/19.jpg)
Apache2
Compressed Weight
19
Total Weight
1036.9K 265.7K
Web Compression + SSL
74.3%
VS
![Page 20: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/20.jpg)
vS
20
force to native
© Blackboard Mobile: h`p://help.blackboardmobile.com
web
Blackboard Mobile Learn Authentication Type
![Page 21: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/21.jpg)
Shibboleth and LDAP
21
Implementing a New Authentication Framework
LDAP Server!
App4!
mobile.lms.utoronto.ca portal.utoronto.ca
App1!
App2!
App3!Shibboleth !2.4.3!
F5
Hostname RestricNon Provided by the New AuthenNcaNon Framework
Provider: Toronto Shibb Auth Provider: Toronto LDAP Auth
![Page 22: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/22.jpg)
Shibboleth and LDAP
Shibboleth LDAP
22
Implementing a New Authentication Framework
mobile.lms.utoronto.ca portal.utoronto.ca
![Page 23: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/23.jpg)
Minification
23
Real path MinificaNon MinificaNon CR MinificaNon t $BBHOME/docs 835,860.00 50.93% 3.1886 $BBHOME/webapps/blackboard 183,477.00 58.46% 0.6999 $BBHOME/webapps/assessment 51,225.00 58.01% 0.1954 $BBHOME/webapps/discussionboard 30,919.00 35.52% 0.1179 $BBHOME/webapps/gradebook 277,527.00 54.81% 1.0587 $BBHOME/webapps/caliper 119,764.00 48.56% 0.4569 $BBHOME/webapps/portal 27,595.00 54.81% 1.0600 $BBHOME/webapps/cms+xy 49,532.00 52.59% 0.1889 $BBHOME/webapps/wysiwyg 99,681.00 52.43% 0.3803 $BBHOME/webapps/webeq-‐plugin 15,354.00 52.43% 0.3800 $BBHOME/webapps/taglibs 44,054.00 52.43% 0.3800 $BBHOME/webapps/* 4,936.00 52.43% 0.3800 Grand Total 1734988.00 8.11
1694.32K MR ~ 52.0%
Savings on payload
![Page 24: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/24.jpg)
Minification
• Prematurely released on SP5
• Released on SP8 as certified.
• Implementing YUI Compressor Library
• Grouping and minifying on-the-fly (inside JVM)
• Enabled by default on SP8
24
Blackboard JS Grouping Tool
## Whether related JavaScript files should be grouped together ## ## for be`er HTTP performance ## bbconfig.javascript.group.files=true
Read More.
<script type="text/javascript" src="/branding/__js__/C131DA0400D29916A81632A83B91BAD2.js?v=9.1.50119.0"></script>
![Page 25: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/25.jpg)
25
Minification Blackboard Grouping Tool
Firebug console output sample
Read More.
![Page 26: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/26.jpg)
26
![Page 27: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/27.jpg)
Solaris Cryptographic Framework (SCF)
27
© Sun Microsystems: Using The Cryptographic Accelerators in the ULTRASPARC T1 and T2 Processors.
![Page 28: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/28.jpg)
28
conf/pkcs11.conf
SSLCryptoDevice pkcs11!
64bit Binary for SPARC
!
CC="cc -m64“ !
CXX="CC -m64“ !
CFLAGS="-m64 -xO2 -DSSL_ENGINE“ !
CXXFLAGS="-m64 -xO2“ !
LDFLAGS="-L/usr/sfw/lib/sparcv9 !
! ! -R/usr/sfw/lib/sparcv9“ !
CCFLAGS="-m64“ !
Read more.
Linking Apache2 binary
Solaris Cryptographic Framework (SCF)
![Page 29: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/29.jpg)
29
OCHO Current environment 9.1SP8
![Page 30: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/30.jpg)
Infrastructure: Hardware
30
App1!
App2!
App3!
Hitachi 9985!2 RAID 6 Arrays!
2 TB x 7200 RPM SATA!Carved into 192 GB Ldev!
!
Collab1!
Data1!
SPARC T4-4!4 CPU @ 3.0 GHz!
8 cores/CPU!256 threads!512GB of RAM!
4 x LDOMs!24 vCPUs!
64G of RAM!
1 x LDOM!56 vCPUs!
120G of RAM!
Bb 9.1SP8
App4! 2 x LDOMs!24 vCPUs!
32G of RAM!App5!
![Page 31: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/31.jpg)
31
Infrastructure: Hardware SPARC T4-4 and Oracle VM Server for SPARC v2.2
64 64 64 32 32 120 64 8
Collab1 App1 Ap2 App3 App4 App5 Data1 IO/Controller Free
Memory
![Page 32: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/32.jpg)
32
Infrastructure: Hardware
32 32 32 16 16 56 32 24
Collab1 App1 Ap2 App3 App4 App5 Data1 IO/Controller Free
SPARC T4-4 and Oracle VM Server for SPARC v2.2
vCPUs
![Page 33: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/33.jpg)
33
Infrastructure: Hardware Live Migration
Read More.
Source
Target
![Page 34: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/34.jpg)
34
Infrastructure: Software Bb 9.1SP8
Blackboard Learn 9.1 SP8
Apache 2.2.2 64 bit SPARC
Shibboleth (DSO)
SSL
Apache 2.2.2 • Compression with SSL • SSL Offloading – PKCS11
Blackboard Mobile • NaNve AuthenNcaNon
AuthenNcaNon Providers:
LDAP + Shibbholeth
Shibboleth • LDAP
![Page 35: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/35.jpg)
Performance
35
Benchmark
requests were sent sequentially with different concurrency levels 50k
Proxy SSL
AJP <
https://server/webapps/portal/healthCheck
deflate Concurrency Apache 2.2.2 Apache 2.2.2
SSL-‐H, AJP SSL-‐H, AJP, COM 10 1230.59 1143.12 100 1962.52 1704.3 200 1699.73 1625.22 500 1870.60 1075.2 1000 1214.95 1173.457 2000 1129.87 1234.44
@ 1k request/sec > process ~1.2K req
![Page 36: Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU](https://reader034.vdocument.in/reader034/viewer/2022052522/554f519fb4c905524c8b4ed5/html5/thumbnails/36.jpg)
Performance
36
24%
56%
17%
2% 1% 0% 0% 0%
0 -‐ 1
1 -‐ 3
3 -‐ 7
7 -‐ 13
13 -‐ 21
21 -‐ 35
35 -‐ 60
60+
Load Times
Avg. Page Load Time:
2 .44 SEC