jeff fried & keenan newton senior technical product managers microsoft corporation session code:...

Getting to know FAST Search for SharePointJeff Fried & Keenan NewtonSenior Technical Product ManagersMicrosoft Corporation

SESSION CODE: OSP206

Best of SharePointBest of High-end

Best of Microsoft

• Dynamic user experiences• Advanced content processing• Broad repository connectivity• Extreme scale and performance• Open and Extensible Platform

• Simplicity and low TCO• Familiar, integrated

developer tools• Windows 7 Integration• Robust partner and platform

eco-system

Enterprise Search from

Microsoft• Out-of-the-box intranet search• People Search• SharePoint platform integration• No code customization

Our Vision: A New Choice in Search

Microsoft Enterprise Search- ProductivityProducts for Every Customer Need

OOB General Productivity Search with light customization

Heavily customizable OOB General Productivity Search and easy to build Search-Driven Applications

• Common UI Framework• Social search features and integration• SharePoint platform integration • End user and site administrator enablement

Common across the product line

• Common Connector Framework (BDC)• APIs and developer Experience• Admin & deployment capabilities• Operations advantages (SCOM, scripting)

OSP217 What’s New in Enterprise Search in Microsoft SharePoint Server 2010

FAST Search Extends SharePoint Superset of capabilities; common base functionality and platform

End Users Content

Content search

People search

Common query and federation APIs

Common connectors and content APIs

Best High-End Content Search based on FAST ESP

Unique Social Search from SharePoint Server 2010

Common IT Pro and developer experience

FAST Search for SharePoint

SharePoint Server2010

Advanced Content Processing

+ Profiles

Introducing FAST Search for SharePointA new choice for enterprise search that eliminates compromise

Productivity Search Experience

Search Driven Applications

A Single, Cost-effectiveInfrastructure

Customized productivity search and search driven applications

Social, Conversational++, Visual, Contextual

Connectivity+, Security, Relevance+, Performance, Administration+, Scale+

+

SharePoint Enterprise CAL + FAST Search for SharePoint server license

SharePoint Standard CAL + SharePoint server license

General Productivity searchLight customization

Social, Conversational

Connectivity, Security, Relevance, Performance, Administration, Scale

SharePoint 2010 search and FAST search comparison

Topics

OOB Search ExperienceCustomizing SearchSearch-Driven ApplicationsPlatform Architecture Deployment and Manageability

ContextualTailored to your

role, task, and perspective, making you more productive

ConversationalChange how you

interact with information, giving you better answers

VisualHelp you identify patterns and

discover new insights

Turn information into business outcomes through engaging experiences.

SocialConnect and collaborate to apply your

talent more effectively

OOB Search ExperienceFAST Search for SharePoint

Demo

OOB User Interface SummaryVisual and conversational interaction with precise control

Built on SharePoint Search CenterLeverages all of innovations in SharePointOpen Web Parts, Federation, query suggestions, related queries, Did you mean?

Visual results connects users with contentThumbnails for Word and PowerPointVisual Best Bets highlight premium content Preview in browser without leaving the results

Deep Refinement

Thumbnails

Previews

Sort on any field

Similar Results

Discovery: Find and Explore

Penecillin

LSD

Uranus

Viagra

Safety glass

Infrared radiation

Microwave ovens

Inkjet printers

Corn Flakes

Chocolate chip cookies

Deep RefinementEnables precise control of results

Contains exact countsLeads to discovering non-obvious relationships, key data trends, and deep analysis of your content

Enables conversational experience You will never miss any content; enabling better findability and exploration across the entire result set

Exact Counts

Sorted by frequency

Provides a sorted viewEach refiner is sorted by frequency, from highest to lowest, indicating the importance of each term

Topics


Customize search to meet your business needsKey ingredients to a great customized search experience

Search in the language of your business

Deliver results that are contextually relevant

Tune relevancy to meet diverse needs

Process content with advanced linguistics

Customize the user experience to build engaging applications

Custom Query Suggestions

Query Suggestions & the Term Store

# retreive values from termstore$ts = Get-SPTaxonomySession -Site http://esgismoss14$groups = $ts.DefaultKeywordsTermStore.Groups$groups | select Name, Description, TermSets

# Get a reference to the query application$searchapp = Get-SPEnterpriseSearchServiceApplication -Identity "FAST Query SSA" # Add new query suggestionsnew-spenterprisesearchlanguageresourcephrase -SearchApplication $searchapp -Language En-Us -Type QuerySuggestionAlwaysSuggest -Name "faster“

# Start the job that deploys the query suggestions (runs once every 24 hours)Start-SPTimerJob -Identity "prepare query suggestions" # List query suggestionsGet-SPEnterpriseSearchQuerySuggestionCandidates -SearchApplication $searchapp

# Remove query suggestionsRemove-spenterprisesearchlanguageresourcephrase -SearchApplication $searchapp -Language En-Us -Type QuerySuggestionAlwaysSuggest -Identity "faster"

Maintaining a custom query suggestion list using the SharePoint Term Store repository, instead of putting the information in a list on a site.

Visual Best BetsVisual cues to highlight essential content

Built on SharePoint KeywordsMatches keywords and synonyms that are contextually relevant to users. Include banners, videos, external websites.

Easy and quick to setupPoint and click setup for site admins. Set and forget with content expiration dates . Web Parts allow for easy page customization

Visual Notification

Web Part Flexibility

What’s for Lunch?Visual Best Bets at Hafslund

Audience-specific search experiencesUse User Context to meet the needs of diverse groups

User context

Inform-ation conte

xt

Application context

Social conte

xt

Renee LoEngineeringContoso Consulting”What should I know about implementing ERP?”

Alan BrewerSales ManagerContoso Consulting”What should I know about selling ERP consulting?”

Username & Group Memberships

LocationLanguages

Business UnitDepartment

TeamTime of Day

Preferred SitesSharePoint Audiences

Interests & Current ProjectsContext of Current Task

Quickly build a contextual experienceUser based tools for creating results that are relevant to your users

Pick the right ingredients Match the proper terms and contexts to boost relevancy for targeted users to ensure your users are always finding the right content

One-way synonymsKeywords map to other termsTwo-way synonymsKeywords become equivalent to other termsBest BetsHighlights key resources that are always relevant to a keywordVisual Best BetsExtend Best Bets with pictures, video, Silverlight controlsDocument Promotion / DemotionTailor specific document relevancy

Create new user contextsSite administrators create contexts based on user profiles to deliver relevant results to the right audiences

Create new keywordsSite Administrators have powerful and simple tools to configure the search experience for groups of users

User ContextFAST Search for SharePoint

Demo

Tune relevancy to meet diverse needsA flexible solution for your organizations, groups and individuals

Optimize Relevancy for broad intranet use

Query results with the default relevancy

”I want to know about my customer Woodgrove Bank and customers in Financial

Services"- Alan Brewer, Sales Lead

New Default Sorting

Promotes relevantResults

Quickly tailor relevancy models Deliver the right results to the right people by creating new Rank Profiles

”I want to get right to the technical documents"

- Renee Lo, Consultant

Documentation, RFPs and SOWs are now promoted Same Results

different order Users can select rank profiles in the sort by box or create their own default views by modifying the web part

Rank ProfilesTune relevancy without impacting the default algorithm

Quality Also known as static rank, consists of multiple managed properties including site, URL depth (preference for shorter URLs), and relative importance of links to this document.

Authority Applies when the query word falls in the link or anchor text.

Query Authority Maps the popularity of a document, or the click-through rate when documents are clicked as a result of a query

Freshness Increases the relevancy if a document was recently created or modified, based on the last modified property.

Proximity Applies to where query terms fall and how close they are to each other within a document

Context Increases the rank of a document if the query term is a managed property associated with that document

Managed Property Effects relevancy when a managed property contains a specific value, such as Woodgrove Bank or Financial Services

Out of the box relevancyTuned for great general productivity experience, relevancy improves with click-throughs and link text analysis.

Extend the default algorithmsCreate new default relevancy models. Blend static and dynamic ranking parameters to instantly improve search results.

Custom Rank ProfileFAST Search for SharePoint

Demo

How to create a Rank ProfileIT Pros are empowered to create new profiles quickly

Rank Profiles created in PowerShell by extending the default relevancy algorithm…

… and are exposed in the user interface by modifying the sorting web part.

Advanced Content ProcessingA new level of search quality

End UsersContent

Fed

era

tion

OpenSearch Source

Content Processor

CrawlerIndexerQuery

Processor

Search Center

MetadataRelevanceControl

UserContext

IndexingConnectivity

Index Partition

…

Form

at

Con

vers

ion

Lan

gu

ag

eD

ete

cti

on

En

tity

Extr

acti

on

Lem

mati

za

tion

Map

per

…

Content Processing Pipeline

Breaks down content to the smallest addressable chunks to build meaningUnderstands file encoding, data formats, and written languages Supports 400+ file formats, 80+ languages

Process content with advanced linguisticsAutomatic and detailed analysis creates a great search experience

Map Crawled Properties Maps all of the metadata that was discovered by the various pipeline stages Web Link Analysis Analyzes documents for hyperlinks extracting anchor text which reinforces the authority ranking of a document.

Document Vector Creates a unique representation of a document that reflects important terms and frequency of occurrence. Used to find similar documents.

Date and Time Normalization

Converts dates and times to a standard representation, to handle locale specific representations. For example, knows that 14-Mar-10 is equivalent March 14, 2010.

Entity Extraction Finds terms in the content and maps them to predefined categories. Out of the box support for People, Companies and Locations, but can be extended to any category.

Lemmatization Finds the root of a word for a given language. For English it maps run, runs, running and ran back to a single lemma. Understands language specific grammar and context.

Tokenization Apply the language specific rules for identifying words, concepts, idioms and phrases. Also applies custom word breakers found in part numbers or telephone numbers.

Language Encoding and Detection

Identifies the native written language and locale specific encoding so that the proper dictionaries can be used by the tokenization and lemmatization stages

Format Conversion Extracts plain text from multiple file formats, encodings, and applications

FAST Content Processing Pipeline

Configure Optional Processing StepsXML Properties mapperOffensive Content FilterPipeline Extensibility

Calls external applications for custom item processingField CollapsingAdd Custom Processing

Content classificationGeo-taggingMachine translationSentiment Analysis

Extending Pipeline capabilitiesSafely add additional analysis and processing

What is Custom Processing?Pipeline Extensibility is a specially defined stage that takes a set of crawled properties, as flat text as input and maps output to another crawled property

Custom Processing is Safe Executable arguments and temporary files automatically handled in sandbox with timeouts. Runs before Crawled Property mapping stage, making new metadata accessible in SharePoint

Type Doc ID Title Author Date Size Keywords Companies Expertise Industry ... Body Text

123 Sales Forec… John Doe 2010-04-15 386 KB sales; pipe… Contoso; Tail.. Strategic Con.. Financial Ser… … IT Market Gr…

456 … … … … … … … … … …

963 … … … … … … … … … …

Map metadata to Managed Properties Automatic association of metadata to content

Expertise

Strategic Consulting

Market Analysis

IT Implementation

…

Industry

Financial Services

Manufacturing

Technology

...

Crawled Properties

Managed PropertiesIndex Schema:

Companies

Contoso

Tailspin Toys

Woodgrove Bank

…

Mapped in Central Admin Or PowerShell

Crawled Properties Standard document metadata discovered by the crawler or extracted from the full text by the FAST Content Processing Pipeline

Managed Properties Map one or more Crawled Properties to a single field. Enables sorting, refinement, relevance tuning and fielded searching

Title

Sales Forecast

Any data can befound and mapped

Managed Properties + RefinersFAST Search for SharePoint

Demo

Topics


Search Driven ApplicationsMeet all the search application needs you have across your business

“How do I support theunique search needs of

teams and work that impact our business?”

To do so, you need a search platform that has• A deep understanding of your

information• Flexible relevance to meet

diverse needs• A customizable UX to increase

user efficiency

Sales: 360o Customer Insight

Services: Knowledge Browser

Marketing: Competitive Intelligence

Research & Development:Innovation Portal

Support:Call Center Advisor

Operations:Systems/Logistics Portal

Legal, HR, IT, Finance, ……

Search-Driven ApplicationFAST Search for SharePoint

Demo

OSP218 Customizing FAST Search Server 2010 for SharePoint

6/10/2010 9:45-11:00 AM Room 261

News and external opinion to monitor and send to

clients

Top information from Woodgrove…new market view report to send to clients

Drilldown to single view with all clues about a customer: portfolio, holdings, communications, annual and quarterly customer plans, etc…

Set of Customers to explore

Experts to help, with availability and rating

Immediate actions on selected items

View of information across different pivots, with drilldown

Finds customers beyond direct terms (REIT ~ real estate)

Start with a simple query, quickly find and explore

Content Crawling: bring in data from lots of placesOOB connectors to SharePoint (reports, account documents), exchange public folders, shared files; BDC with customization in SPD (no code) for customer portfolio/holdings

Content processing: creating metadataNames of holdings, offerings, key concepts, companies, peopleSynonyms for key concepts (real estate ~ REIT)

OOB web parts configured for style Federation, People Search, Search actions

Custom web parts for visual navigationRoll-up configured via results collapsingCustom relevance profileSharePoint workflows for act-on-selected-items

How would you create this?

Back End Processing Tasks:Load content from many different places

Out of the box connectors for SharePoint, exchange public folders, and shared filesSharePoint Designer to configure connection to customer portfolio/holdings database

Create custom metadata with content processing pipelineNames of holdings, offerings, key concepts, companies, peopleSynonyms for key concepts (real estate ~ REIT)Roll-ups configured with optional results collapsing stage

Create custom relevance profileDesigners can stylize the User Interface

Apply styles to web partsFederation, People Search, Search actions

Build custom web parts for visual navigationUse SharePoint workflows to perform business specific actions

Leveraging the platform to build applicationsPutting together all of the pieces to build search-driven applications

Search 2010 ArchitectureThe platform for Search Customization

SearchWeb Parts

SharePoint Search Index

OpenSearch /Custom Source

SharePoint Indexer

Federation OM Web Service, RSS

FASTSearch Index

FAST Indexer

What’s New in 2010?Primary Search Web Parts now UnsealedFederation now a key Public OM layer

All Web Parts built on federationQuery alteration, custom Runtimes, blending results from multiple sources

Web Service / RSS Enhancements

FAST Search / SharePoint Search:Shared Web Parts, RSS, Web ServiceShared Federation OMIndex and Crawling SeparateFAST unique Content Processing Pipeline

SearchWeb Parts

Federation / Query OM Web Service, RSS

Content Processing

Topics


FAST Search Service Architecture“Sandwich” with two SSAs

End Users Content

Content search

People search

Common query and federation APIs

Common connectors and content APIs

Best High-End Content Search based on FAST ESP

Unique Social Search from SharePoint Server 2010

Common IT Pro and developer experience

FAST Search for SharePoint

SharePoint Server2010

Advanced Content Processing

+ Profiles Query SSA Content

SSA

FAST Search Server 2010Summary of architectural components

Custom Front-End

OpenSearch or Other Sources

SharePoint Front-end

People Search

Qu

ery

Obje

ct M

od

el

Query and Result

Processing

Security AccessModule

SearchCore

Indexing

Federation Object Model

Query Web Service

AdvancedContent

Processing

Linguistics

WebLink

Analysis

Connectors• Web Crawler• JDBC

Connectors• SharePoint• File Traverser• Web • BDC• Exchange• Notes• Documentum

Microsoft System Center Operations Manager

Monitoring Services

Administration and Schema Object Model

Site Collection Level Admin UI

• Keyword Management• User Context Management• Site Promotion/Demotion

PowerShell

• Schema configuration• Admin configuration• Deployment configuration

Central Administration UI

• Property mapping• Property extraction• Spell-checking

FAST Server(s)

SharePoint Server(s)

Other Server(s)

Content

FAST Search for SharePoint Scaleout

Content Volume

Query Volume

Scale-out multiple “dimensions”Query VolumeContent VolumeIndexing freshness

Redundancy optionsSearchIndexing

*Depends on content and hardware specifics

Search and Indexing

Crawling and Content Processing

Query and Result Processing

Back-end with extreme and flexible scale out options

No theoretical upper bounds!

FS4SP – Main Components to Scale

SharePoint CrawlerCapacity: ~30 mill items per crawler node, SQL server needs to be scaled for increased IO

Web AnalyzerCapacity: ~30 mill items per web analyzer node

Indexer/search nodeCapacity: ~15 mill items per indexer/search nodeNote: Will also be used to scale the Item Processing component

SharePoint Crawler

Crawl DBSP CrawlPeople Crawl

Indexer/Search

FAST-FSTIDX-11Index/Search

Web Analyzer

FAST-WA-1Web Analyzer

FS4SP – Minimum Deployment

Single-tier setup Two-tier setup

SP2010 Farm FAST Search for SP 2010 Farm

SQL 2008 Cluster

Crawl DBSearch Admin DB

All roles

SP2010 Farm

All roles

FAST Search for SP 2010 Farm

All roles All roles

Note:None of these environments are redundant. For a production environment, you should at a minimum consider running the SharePoint and FAST servers in a virtual environment and with a mirrored or clustered SQL server.

FS4SP – Small Deployment

Scope:Query rate: < 10 QPSContent: < 10 M itemsStandard configuration

SP2010 Farm FAST Search for SharePoint 2010 Farm

SQL 2008 Cluster

WFEQuery SSA

WFEQuery SSA


FAST-ADM-1Admin

Content Distributor 112 Docprocs+Web Analyzer

FAST-FSTIDX-11Index (Search)

Indexing Dispatcher 212 Docprocs+

QR Server

FAST-FSTIDX-21(Index) Search

QR Server

SP CrawlPeople Crawl


*

Note:Servers marked with a * are only

needed for redundancy.

*

*

FS4SP – Medium Deployment

Scope:Query rate: < 10 QPSContent: < 40 M itemsStandard configuration

FAST Search for SharePoint 2010 Farm

FAST-ADM-1Admin

Content Distributor 1Web Analyzer12 Docprocs+

FAST-FSTIDX-11Index (Search)12 Docprocs+



QR Server


QR Server

FAST-ADM-2Content Distributor 2

Web Analyzer12 Docprocs+

(Enterprise Crawler)



QR Server

SP2010 Farm

SQL 2008 Cluster

WFEQuery SSA

WFEQuery SSA




FS4SP – Large DeploymentScope:

Query rate: < 10 QPSContent: < 100 M itemsStandard configuration

SP2010 Farm

FAST Search for SharePoint 2010 Farm

SQL 2008 Cluster

WFEQuery SSA

WFEQuery SSA SP Crawl

People CrawlSP Crawl

People Crawl


SP Crawl

FAST-ADM-1Admin

ConfigServerSpelltunerSamAdmin

Content Distributor 1Web Analyzer12 Docprocs+




QR Server


QR Server

FAST-ADM-2Content Distributor 2

Web Analyzer12 Docprocs+



QR Server




QR Server


QR Server



QR Server

FAST-ADM-3Web Analyzer12 Docprocs+

Secure, unified access to informationIndex or federate with content, applications, and services

Access content securelywith a full-featured content crawler and both index- and query-time trimming of results

Federate queriesusing OpenSearch to quickly access existing search indexes and online information services

Extend your reachusing Business Connectivity Services to access your content sources and business applications

OpenSearch Federation

Indexing Connectors

EnterpriseContent

Business Applications

Information Services

User Experience

Search Index

Websites and Intranet

File Shares

IBM Lotus Notes

EMC Documentum

SharePoint Server

Exchange Public Folders

AD & LDAP Profiles

Etc …

http://www.linkedin.com/home?trk=hb_logo



Simplified, powerful administrationA high-end enterprise search solution that’s easy to deploy and manage

Deploy easilyusing wizard-driven installation, a topology designer, and native support for 64-bit virtualization

Manage efficiently with full support for Microsoft System Center and PowerShell scripting to automate tasks

Streamline administrationwith a simplified admin console that helps you manage search services across your enterprise

Enterprise Search from Microsoft

UX ITDXGo beyond the

search boxEliminate

compromiseDo more

with search

Introducing FAST Search for SharePointA new choice for enterprise search that eliminates compromise

Productivity Search Experience

Search Driven Applications

A Single, Cost-effectiveInfrastructure

Customized productivity search and search driven applications

Social, Conversational++, Visual, Contextual

Connectivity+, Security, Relevance+, Performance, Administration+, Scale+

+

SharePoint Enterprise CAL + FAST Search for SharePoint server license

SharePoint Standard CAL + SharePoint server license

General Productivity searchLight customization

Social, Conversational

Connectivity, Security, Relevance, Performance, Administration, Scale

SharePoint 2010 search and FAST search comparison

Summary

FAST Search for SharePoint is NEW

It combines the best of high-end search, SharePoint and Microsoft

It extends SharePoint Server 2010Superset of CapabilitiesHighly capable and easy to manage

BIO206 Improving Business Intelligence with Search 6/10/2010 1:30-2:45 PM Room 238

Track Resources

For More Information: http://sharepoint.microsoft.com

SharePoint Developer Center: http://msdn.microsoft.com/sharepoint

SharePoint Tech Center: http://technet.microsoft.com/sharepoint

Official SharePoint Team Blog: http://blogs.msdn.com/sharepoint

Related Content

Breakout Sessions – See Conference Guide for full list of OSP Track Sessions

Interactive Sessions – OSP Track has 10 Interactive Sessions – OSP01-INT – OSP10-INT

Hands-on Labs – OSP01-HOL – OSP20-HOL

Product Demo Stations: Yellow Section, OSPOffice 2010, SharePoint 2010, Project Server 2010, Visio 2010 have kiosks & demos

Complete an evaluation on CommNet and enter to win!

Play the Microsoft Office & SharePoint Track Tag Contest

Download the Microsoft Tag ReaderOpen the internet browser on your mobile phone and visit http://gettag.mobi

Come to the Expo Hall – Yellow Section OSP Info Desk for Official Rules & Collect Additional Tags from all OSP Track Sessions, Speakers and Expo Hall!

Grand Prize (1)Xbox 360 Prize Package

and Microsoft® Office 2010

Daily Prizes40 copies of

Microsoft® Office 2010

http://gettag.mobi/

61

Q&A

[email protected]@microsoft.com

mailto:[email protected]



© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

JUNE 7-10, 2010 | NEW ORLEANS, LA

jeff fried & keenan newton senior technical product managers microsoft corporation session code:...

Documents

microsoft sharepoint

fast espunique social

product names

windows vista

informational purposes

market conditions

new choice

scripting osp217 whats