mstr meetup
DESCRIPTION
BI Team @ LinkedIn hosted a user group meeting for MicroStrategy customers in bay area. Presentation includes information about LinkedIn, concepts of metadata driven model for business dashboards, customizations using SDK, JSP and JQUERY.TRANSCRIPT
SF MicroStrategy Bay Area Meetup
December 4, 2013
LinkedIn Confidential ©2013 All Rights Reserved
Agenda
• About LinkedIn
• BI Challenges
• Administration Hacks
• Q & A
Our Mission Connect the world’s professionals to make them
more productive and successful.
Our Vision Create economic opportunity for every
professional in the world.
Members First!
©2013 LinkedIn Corporation. All Rights Reserved.
The world’s largest professional network
Over 65% of members are now international
Active Company Pages
>3M
Languages
>30M
>91% Fortune 100 Companies
use LinkedIn Talent Soln to hire
19
259M+ Members Worldwide
Growing at more than two members per second
Students
Technology Stack
Technology Stack
Internal User Groups
Data Analytics
Infrastructure
Biz-Ops
Data
Sciences
Product
Hiring Solutions
Executives
Marketing Solutions
Talent Solutions
Business Intelligence Team
Bhavani Akunuri Sabeer Thajudeen Kevin Simon Poorna Lankipalli
Sandeep Deviah Qun Li Sunil Shirguppi
Agenda
• About LinkedIn
• BI Challenges
• Administration Hacks
• Q & A
Challenges
• BIG DATA - 4 V’s :
• Data Quality & SLA
• Navigation to reports/dashboards
• Multiple sources for answering critical questions
Velocity , Volume , Variety & Veracity
Microstrategy Meetup
December 4, 2013
LinkedIn Confidential ©2013 All Rights Reserved
Metadata Driven Dashboards
by: Kevin Simon, Sr Data Warehouse Engineer
How much would you be benefited if you
had the ability to effect real time change to
your critical dashboards, with little to no IT
involvement?
FOR THE ENGINEERS:
How much would you be benefited if your
users could make changes to their critical
dashboards, without needing to engage you
to make simple changes?
FOR THE BUSINESS USERS:
THE OLD WAY …
The fixed layout of this
dashboard makes it
difficult to manage.
IT resources can be freed to work on other
projects.
Real-time changes can effectively be made to dashboards.
Metrics can be introduced into your production
dashboards, prior to ETL development.
HOW DID WE FIX THE PROBLEM?
With a Metadata driven design…
PROCESS FLOW AND TABLE DESIGN
METRIC_SK METRIC_NAME CAT_ID CAT_ORDER METRIC_ORDER DISPL_FACTOR
100 PAGE VIEWS 100 10 10
200 SUBSCRIBERS (M) 100 10 20 1000000
9000 PV / UNIQUE MBR 200 20 10 1000
Daily ETL
Aggregate
Process
Daily Custom
Metrics
Forecasting
Model (R)
Dashboard
View
DATE_SK METRIC_SK COUNTRY_ID LOCALE_ID METRIC_NAME METRIC_VALUE
2013-11-01 100 500 1000 PAGE VIEWS 1000
2013-11-01 200 500 1000 SUBSCRIBERS 2000
2013-11-01 9000 550 1000 PV / UNIQUE MBRS 1500
Aggregate
Look up
IN CONCLUSION…
A Metadata approach to dashboard
design has allowed for real time change
to critical dashboards.
IT Resources can focus on other things.
More power and control given to users.
Same message across multiple dashboards.
Time to market greatly reduced.
Next Play
Challenges
• BIG DATA - 4 V’s
• Data Quality & SLA
• Navigation to reports/dashboards
• Multiple sources for answering critical questions
Microstrategy Meetup
December 4, 2013
LinkedIn Confidential ©2013 All Rights Reserved
Data Quality & SLA
by: Sabeer Thajudeen, Senior Software Engineer
Dashboard Delivery
M 1 M 2 M n
Daily ETL
Aggregate
Process
Daily Custom
Metrics
Forecasting
Model (R)
Trigger Cube
Validator
Trigger
Internal
Trigger
External
Command Manager
Data
Validation
(BTEQ)
Metric Alert WoW No
Data
M1 Y 20% Y
M2 Y NA Y
M3 N NA NA
Command Manager
- Trigger Event
(Dashboard)
Command Manager
- Trigger Event
(Alert Report)
Workflow
Scheduler
ABORT
……
MSTR Workflow
Challenges
• BIG DATA - 4 V’s
• Data Quality & SLA
• Navigation to reports/dashboards
• Multiple sources for answering critical questions
Microstrategy Meetup
December 4, 2013
LinkedIn Confidential ©2013 All Rights Reserved
Navigation to reports/dashboards
by: Bhavani Akunuri, Senior Software Engineer
Too Many Clicks
Resolution
• Identified top used reports
• Create short links
• Landing page re-design
Demo
Sample Code
Last Updated Timestamp
<web:ifBeanValue bean="frame" property="getDesignMode" value="0">
<web:then>
<div class="mstrMenuLastUpdatedCustom"> <web:displayGuiComponent name="report_last_update" isContainer="true"/> </div>
</web:then>
<web:else>
<web:ifBeanValue bean="frame.rb" property="getXMLStatus" value="6">
<web:then>
<div class="mstrMenuLastUpdatedCustom"> <web:displayGuiComponent name="report_last_update" isContainer="true"/> </div>
</web:then>
<web:else>
<web:displayGuiComponent name="report_design_area" isContainer="true"/>
</web:else>
</web:ifBeanValue>
</web:else>
</web:ifBeanValue>
Challenges
• BIG DATA - 4 V’s
• Data Quality & SLA
• Navigation to reports/dashboards
• Multiple sources for answering critical questions
Q: How are my key products performing?
Q: What caused the variance on KPI?
Q: Does it have a global effect?
Demo
Sample Code
Overlay Effect
<style>
.overlay_effect-ingraphs,
{
display:none;
position:fixed;
opacity:0.7;
height:50%;
width:50%;
z-index:100;
}
</style>
Pop Up Code
<script type="text/javascript">
$(document).ready( function() {
var main = $(this);
$("#ingraphs").click(function() {
loadPopupBox(".overlay_effect-ingraphs",".popupInGraphs");
});
function loadPopupBox(overlay,popup){
$(overlay).fadeIn("slow");
$(popup).fadeIn("slow");
}
});
</script>
Sample Code
Overlay Effect
<style>
.overlay_effect-ingraphs,
{
display:none;
position:fixed;
opacity:0.7;
height:50%;
width:50%;
z-index:100;
}
</style>
Pop Up Code
<script type="text/javascript">
$(document).ready( function() {
var main = $(this);
$("#ingraphs").click(function() {
loadPopupBox(".overlay_effect-ingraphs",".popupInGraphs");
});
function loadPopupBox(overlay,popup){
$(overlay).fadeIn("slow");
$(popup).fadeIn("slow");
}
});
</script>
Next Play
Agenda
• About LinkedIn
• BI Challenges
• Administration Hacks
• Q & A
Microstrategy Meetup
December 4, 2013
LinkedIn Confidential ©2013 All Rights Reserved
Administration Hacks
by: Sabeer Thajudeen, Senior Software Engineer
Swapping iServer names in User Bookmarks
Context:
▪ Disaster (DR) recovery is critical . MSTR infrastructure has been set up for DR
▪ Each time we test DR, VIP and base URL remains the same. However, iServer names change
Impact:
▪ Bookmarks (to access reports directly) include iServer names and will not work
Resolution:
▪ Use MSTR SDK to seamlessly swap incorrect iServer names in bookmarks (External Security
Module)
Link
Implementation Detail
▪ Create a Custom ESM (External Security Module) that extends ‘Abstract External
Security’
▪ Override the function ‘handlesAuthenticationRequest’
– 1) Read iServer names in incoming URL using regex. Matcher and
regex.Pattern libraries ▪ Pattern pat_type_1 = Pattern.compile("Server=.{4}-rpt..", 2);
▪ Pattern pat_type_2 = Pattern.compile("mstrWeb=-.{5}-rpt..", 2);
for(Matcher mat_type_2 = pat_type_2.matcher(requestURL); mat_type_2.find();)
{ //read iServer name in incoming URL
String iServerName = mat_type_2.group();
….
– 2) Replace if iServer name does not match expected server name Example:
//Does iServerName in URL match with expected Server Name
if(! iServerName .equalsIgnoreCase((new StringBuilder("mstrWeb=-")).append(Location).append("*-
").append(Server1).toString()) && ! iServerName .equalsIgnoreCase((new StringBuilder("mstrWeb=-
")).append(Location).append("*-").append(Server2).toString()))
{ ……
//Correct Server Name
requestURL = mat_type_2.replaceAll((newStringBuilder("mstrWeb=-")).append(Location).append("*-
").append(Server1).toString());
….}
}
– 3) Resubmit corrected URL - response.sendRedirect(requestURL);
▪ Update MicroStrategy Config Settings – Application Settings to use Custom ESM
(Web Customization Editor)
Monitoring MSTR via Web
Context:
▪ 24/7 Operations team to monitor and support infrastructure
▪ Handles not just MSTR but Teradata, Hadoop & ETL pipleline
▪ Prefer a web based system for monitoring rather than a desktop client
Solution
▪ Built a Web GUI to monitor critical aspects of the MicroStrategy infrastructure
– Delivery Status of all our critical dashboards
– Cube Refresh Status
– Job monitor for currently active jobs
– User Connection Monitor
InSights Monitor
MSTR Monitor
(Web Application)
Statistics
Tables
MSTR
Application
(iServer)
SDK JDBC
Implementation Detail ▪ Dashboard Delivery Status and Cube Status
– IS_DOCUMENT_STATS and IS_CUBE_REP_STATS
▪ Instantiate WebObjectsFactory: factory = WebObjectsFactory.getInstance();
▪ Force connection to each iServer //Get cluster information
WebClusterAdmin wca = factory.getClusterAdmin();
WebClusters wcs = wca.getClusters();
for(int i=0;i<wcs.size();i++){
for(int j=0;j<wcs.get(i).size();j++){
//Set load balance factor to 0 to force cionnection to other iServer
wcs.get(i).get(j).setLoadBalanceFactor(0);
jobDetails+= getJobDetails(session,factory,httpReqId);
wcs.get(i).get(j).setLoadBalanceFactor(1);
}
}
▪ Inside getJobDetails: //Retrieve job source. Loop through each job to get job info
//Use the same logic for the source WebMonitorTypeUserConnection to obtain session info
strSessionID = session.getSessionID();
source = (JobSource) factory.getMonitorSource(EnumWebMonitorType.WebMonitorTypeJob);
results = source.getJobs();
for (int i=0; i < results.getCount(); i++) {
Job singleJob = results.get(i);
job_id= singleJob.getJobID();
duration = singleJob.getDuration();….
}
Next Play
Credits
▪ Vincent Kwong – Program Manager
▪ Anirudh Gupta & Crowe Bonnie – Meet up Organizers
▪ LinkedIn AV team, Communications & Security
▪ Data Science & Biz-Ops
▪ ETL & Platform Teams
▪ NOC, DS Operations & DBA
Agenda
• About LinkedIn
• BI Challenges
• Administration Hacks
• Q & A