http file shares sharepoint user profiles lotus notes documentum exchange folders custom - bcs...
TRANSCRIPT
Search Architecture and sizing in SharePoint 2013Barry Waldbaum & Thomas MolbachArchitects – Business Productivity CoEMicrosoft
SPC336
What are we going over?
v
wu
Search Architecture
HTTPFile sharesSharePointUser profilesLotus Notes DocumentumExchange foldersCustom - BCS
SharePointSP AppsDevicesNon-SP UX
SharePoint 2013 Search Architecture
SearchAdmin
Content UXCrawl
ContentProcessing Index
QueryProcessing WFE
API
AnalyticsProcessing
FAST Search Index
Public API
Unit of scale/role boundary
Analytics Service
Feeding Chain Index Core
Query Chain
HTTPFile sharesSharePointUser profilesLotus Notes DocumentumExchange foldersCustom - BCS
SharePointSP AppsDevicesNon-SP UX
SharePoint 2013 Search Architecture
SearchAdmin
Content UXCrawl
ContentProcessing Index
QueryProcessing WFE
API
AnalyticsProcessing
FAST Search Index
Public API
Unit of scale/role boundary
Feeding Chain
Crawl Component OOB connectors Extensible through
BCS Local disk cache Crawled items
tracked in Crawl database
Configurations stored in Admin database
Crawl modes Full Crawl Incremental Crawl Continuous Crawl
Crawl
ContentProcessing Index
Crawl
HTTP
File Shares
SharePoint
User Profiles
Exchange
Lotus Notes
Documentum
Custom (BCS)
...Admin
mssearch.exe
SearchAdmin
FAST Search Index
Content Processing Component Stateless node Analyzes content for
indexing Processing flow Dictionaries Schema mapping Stores links and
anchors in Link database (analytics)
Extensible through web service call-outs
Configurations stored in admin database
Crawl
ContentProcessing Index
Crawl
Admin
AnalyticsProcessing
Link
Content EnrichmentWeb Service
SearchAdmin
FAST Search Index
Demo
Crawled and Managed Properties
HTTPFile sharesSharePointUser profilesLotus Notes DocumentumExchange foldersCustom - BCS
SharePointSP AppsDevicesNon-SP UX
SharePoint 2013 Search Architecture
SearchAdmin
Content UXCrawl
ContentProcessing Index
QueryProcessing WFE
API
AnalyticsProcessing
FAST Search Index
Public API
Unit of scale/role boundary
Index Core
Replica
ReplicaReplica
Partitions and Replicas n Partitions 1 Primary Replica ≥ 0 Secondary Replicas
Index Partition details
Details• Partition are a logical portion of the
entire index • Replicas are there for fault tolerance/
increased query throughput• All nodes performs indexing• Journal shipping from primary to
replicas• Partitions can be added but NOT
removed• Index can scale in both horizontal
(partitions) and vertical (replicas) ways
Replica
Content Processing
Replica
Partition #1
Replica
Replica
Replica
Partition #2
Replica
Replica
Replica
Partition #3
Replica
HTTPFile sharesSharePointUser profilesLotus Notes DocumentumExchange foldersCustom - BCS
SharePointSP AppsDevicesNon-SP UX
SharePoint 2013 Search Architecture
SearchAdmin
Content UXCrawl
ContentProcessing Index
QueryProcessing WFE
API
AnalyticsProcessing
FAST Search Index
Public API
Unit of scale/role boundary
Query Chain
SharePointSP AppsDevicesNon-SP UX
Web Front-End
UXQueryProcessing WFE
API
Index
• Query APIs• REST/OData API• Client-Side object model
(CSOM)• Server-Side object model
(SSOM)
• Search center• Result templates• ContentWebPart• RefinerWebPart• Portals and publishing
Admin SearchAdmin
FAST Search Index
SharePointSP AppsDevicesNon-SP UX
Query Processing Component
UXQueryProcessing WFEIndex
• Stateless node• Processing flows• Query analyzer• Linguistics• Dictionaries• Result sources• Schema mapping• Query rules• Query federation • Configurations
stored in admin database
FAST Search Index
API
Admin SearchAdmin
HTTPFile sharesSharePointUser profilesLotus Notes DocumentumExchange foldersCustom - BCS
SharePointSP AppsDevicesNon-SP UX
SharePoint 2013 Search Architecture
SearchAdmin
Content UXCrawl
ContentProcessing Index
QueryProcessing WFE
API
AnalyticsProcessing
FAST Search Index
Public API
Unit of scale/role boundary
Analytics Service
Analytics Service empowers (examples)…• View counts• Sort by popularity• Recommendations• Relevancy based
on usage• Search reports• «Suggested sites
to follow»• «Are you looking
for these again?»
Analytics Processing Component
Crawl
ContentProcessing
QueryProcessing WFE
API
AnalyticsProcessing
Link
Analytics
Reporting
• Map-reduce• Learns by usage• Search Analytics• Usage Analytics• Enriches index by
updating index items
• Usage reports in Analytics database
Event Stream Admin
Index
FAST Search Index
SearchAdmin
Demo
Query Toolhttp://sp2013searchtool.codeplex.com/
Deployment and Scaling
SizingTwo options for scalingScale up with more/faster hardware resourcesScale out with more components across multiple machines
Avoid sharing critical resourcesIndex is disk intensive and crucial in all load scenariosConsider shared load on network, disk and CPU• Within a VM• Between VMs on same physical host
Windows 2012 vs. 2008 R2 Guidance
Physical
Virtualized
CPC
APC
Adm
Crawl
Index
QPC
2012 or 2008 R2 Host
CPC
APC
Adm
Crawl
Index
QPC
2012 Host
VM
CPCAdm
Crawl QPC
VM
Index APC
VM
2008 R2 Host
Why should you benchmark your VMs!
4 Physical servers
Cluster supporting many applications including 11 databases
134 Virtual servers
Enterprise Search Farm
This happens all too often
Small Topology: 10M EnterpriseDistributed
Index
4CPU/16GB/500GB
QPC
4CPU/8GB/300GB
Adm
Crawl
CPC
APC
Index
4CPU/16GB/500GB
QPC
4CPU/8GB/300GB
Adm
Crawl
CPC
APC
8CPU/24GB/700GB
Adm
Crawl
CPC
APC
Index QPC
8CPU/24GB/700GB
Adm
Crawl
CPC
APC
Index QPC
Consolidated
Scaling from 10M to 40M items8CPU/24GB/700GB
Adm
Crawl
CPC
APC
Index
QPC
8CPU/24GB/700GB
Adm
Crawl
CPC
APC
Index
QPC
Scaling from 10M to 40M items4CPU/8GB/100GB
Adm CPC
4CPU/8GB/300GB
Crawl
APC
4CPU/16GB/500GB
Index1
QPC
4CPU/16GB/500GB
Index2
4CPU/16GB/500GB
Index3
4CPU/16GB/500GB
Index4
4CPU/8GB/100GB
Adm CPC
4CPU/8GB/300GB
Crawl
APC
4CPU/16GB/500GB
Index1
4CPU/16GB/500GB
Index2
4CPU/16GB/500GB
Index3
4CPU/16GB/500GB
Index4
QPC
Large Topology: 100M Enterprise
4CPU/8GB/100GB
AdmCraw
l
4CPU/8GB/100GB
AdmCraw
l
12CPU/24GB/500GB
12CPU/24GB/500GB
4CPU/16GB/500GB
APC CPC
APC CPC
Index
Index
Index
Index
QPC
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GBInde
x
Index
Index
Index
QPC
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GBInde
x
Index
Index
Index
QPC
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GBInde
x
Index
Index
Index
QPC
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GBInde
x
Index
Index
Index
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GB
High Density Indexing
Large Topology: 100M (Oct CU)
4CPU/8GB/100GB
AdmCraw
l
4CPU/8GB/100GB
AdmCraw
l
12CPU/24GB/500GB
12CPU/24GB/500GB
8CPU/32GB/1TB
APC CPC
APC CPC
Index
Index
Index
Index
QPC
8CPU/32GB/1TB
8CPU/32GB/1TB
Index
Index
Index
Index
QPC
8CPU/32GB/1TB
8CPU/32GB/1TB
Index
Index
Index
Index
QPC
8CPU/32GB/1TB
8CPU/32GB/1TB
Index
Index
Index
Index
QPC
8CPU/32GB/1TB
8CPU/32GB/1TB
Index
Index
Index
Index
8CPU/32GB/1TB
High Density
!
HD Large Topology: 100M (Oct CU)
4CPU/8GB/100GB
AdmCraw
l
4CPU/8GB/100GB
AdmCraw
l
12CPU/24GB/500GB
12CPU/24GB/500GB
APC CPC
APC CPC
16CPU/64GB/2TB
Index
Index
Index
Index
QPC
16CPU/64GB/2TB
Index
Index
Index
Index
QPC
16CPU/64GB/2TB
Index
Index
Index
Index
QPC
16CPU/64GB/2TB
Index
Index
Index
Index
QPC
8CPU/32GB/1TB
Index
Index
8CPU/32GB/1TB
Index
Index
High Density
!
HD XL Topology: 240M (Oct CU)
8CPU/16GB/100GB
AdmCraw
l
8CPU/16GB/100GB
AdmCraw
l
12CPU/24GB/500GB
APC CPC
16CPU/64GB/2TB
Index
Index
Index
Index
QPC
16CPU/64GB/2TB
Index
Index
Index
Index
16CPU/64GB/2TB
Index
Index
Index
Index
16CPU/64GB/2TB
Index
Index
Index
Index
QPC
16CPU/64GB/2TB
Index
Index
Index
Index
16CPU/64GB/2TB
Index
Index
Index
Index
16CPU/64GB/2TB
Index
Index
Index
Index
QPC
16CPU/64GB/2TB
Index
Index
Index
Index
16CPU/64GB/2TB
Index
Index
Index
Index
16CPU/64GB/2TB
Index
Index
Index
Index
QPC
16CPU/64GB/2TB
Index
Index
Index
Index
16CPU/64GB/2TB
Index
Index
Index
Index
High Density
!APC CPC
12CPU/24GB/500GB
APC CPC
12CPU/24GB/500GB
APC CPC
12CPU/24GB/500GB
Components – Scaling cheat sheetComponent CPU Network Disk Memory
Search administration t t tCrawler tt ttt ttt2 ttContent processing (CPC) ttt3 tt tttAnalytics processing (APC) tt3 ttt4 tt ttIndex ttt tt ttt tttQuery processing (QPC) t tt5 tt
1. Admin: Memory footprint increase with topology size
2. Crawler: Crawled documents are temporarily stored to disk
3. CPC and APC are scheduled with “below normal” priority
4. APC: Network bandwidth increase with scale out, but only in-between APC nodes
5. QPC: Network IOPS increases linearly with number of index partitions and QPS
Disk - 10Mitems
Disk - 100M items
15GB data2GB log
110GB data50GB log
10GB data0.1GB log
80GB data5GB log
Usage dependent
Usage dependent
0.4GB data1GB log
1GB data2GB log
500 GB per VM
500 GB per VM
200 GB per VM
200 GB per VM
Component scaling and disk sizesComponen
tScaling
Crawl DB One DB per 20M itemsSQL IOPS: 10 per 1 DPS
Link DB One DB per 60M itemsSQL IOPS: 10 per 1M items
Analytics DB
Split when reaching100-300GB
Search admin DB
One DB
Indexer Component
10M per VM~ 300 IOPS, 200 MB/s R/W
Analytics Component
Content Dependent
SharePoint 2013 Search in the cloud
For Internet Site (FIS) SharePoint applications with anonymous or authenticated access, development, test, and pilot environments.
For Internal ApplicationsSharePoint 2013 in Office 365 is very powerful and versatile.
Backup and Restore
Backup and RestoreWhat you need to knowIndex in 2013 is designed for robust backup and restoreEverything but the index is in the databasePoint in Time backupBackup does not need to be restored to the same topologyNo query down timeBackup/Restore can make disaster recovery easier
Backup and Restore Scenarios
Moving from QA to ProductionPerform full crawl in non-fault tolerant environment, restore to production
Moving from Production to QAUse a copy of your production index in a simpler QA environment
Disaster RecoveryHot, Warm, Cold
Migration
Search Migration Paths
Re-Implement Search First Migration
Migrate fromFAST Search For SharePoint 2010
Migrate from SharePoint 2010 Search
Four Search Migration PathsSP
20103. Migrate From SP Searcha) Backup Content & SSA DBsb) Restore DBs & SSA
SP 2013
Upgrade to SharePoint
2013 Search Center
SP 2013”SP 2010”
mode
FS4SP
4. Migrate From FS4SPa) Backup/Restore DBs
b) Custom PowerShell (convert DB tables)
c) Migration CmdLet
d) Custom PowerShell (to export/import “other” FS4SP
configuration, as able…)
2. Search First Migrationa) Implement SP 2013 Searchb) Publish SP 2013 Search
Servicec) Consume in SP 2010
1. Re-Implement
Recap
v
wu
See you at the Search booths & Search tables at Ask the Experts WED @6:15pm!
Session Session Room Time
Develop Advanced Search-Driven SharePoint 2013 Apps SPC402 Palazzo I, J Tue 1:45pm
Best practices for Hybrid Search deployments SPC306 Veronese 2401 Tue 5:00pm
SharePoint 2013 Search Analytics SPC340 Palazzo M, N Wed 9:00am
How to manage and troubleshoot Search: A practical guide SPC375 Veronese 2401Wed 10:45am
6 Proven Steps to Get the Best Out of Search in SharePoint 2013 SPC265 Delphino 4001 Wed 1:45pm
Best practices for Information Architecture and Enterprise Search SPC207 Veronese 2401 Wed 1:45pm
Search content enrichment and extensibility in SharePoint 2013 SCP414 Palazzo K, L Wed 1:45pm
Customizing Search experiences with Azure Hosted Data and Bing Maps SPC321 Veronese 2401 Wed 3:15pm
Futuristic Search applications using Kinect and Yammer! SPC405 Palazzo M, N Wed 3:15pm
Search architecture and sizing in SharePoint 2013 SPC336 Titian 2201 Wed 5:00pm
Effective Search deployment and operations in SharePoint 2013 SPC360 Veronese 2401 Thu 9:00am
SharePoint 2013 Search display templates and query rules SPC322 Palazzo M, N Thu 9:00am
Managing Search Relevance in SharePoint 2013 and O365 SPC382 Veronese 2401 Thu 12:00pm
Searc
h R
ela
ted S
ess
ion
s
Questions?
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
MySPCSponsored by
connect. reimagine. transform.
Evaluate sessionson MySPC using yourlaptop or mobile device:myspc.sharepointconference.com