msg312: programming sharepoint ™ portal server's search engine joe sherman program manager...

MSG312: Programming MSG312: Programming SharePointSharePoint™™ Portal Server's Portal Server's Search EngineSearch Engine

Joe ShermanJoe ShermanProgram ManagerProgram ManagerSharePoint Portal ServerSharePoint Portal ServerMicrosoft CorporationMicrosoft Corporation

SharePoint Portal ServerSharePoint Portal Server

The flexible portal The flexible portal solution that lets you find, solution that lets you find, share and publish share and publish information easily within information easily within business units and across business units and across enterprisesenterprises Simplest access to Simplest access to

business informationbusiness information Integrated document Integrated document

management to unleash management to unleash portal productivityportal productivity

Easiest and most effective Easiest and most effective way to build and customize way to build and customize intranet portalsintranet portals

AgendaAgenda

Introduction to Search in Introduction to Search in SharePoint Portal ServerSharePoint Portal Server

Write custom search applicationsWrite custom search applications Application: Rich client or browserApplication: Rich client or browser Data API: WebDAV/XML or ADO/OLE DBData API: WebDAV/XML or ADO/OLE DB Digital Dashboard Web PartsDigital Dashboard Web Parts SQL extensions for full-text searchSQL extensions for full-text search

Add subscriptionsAdd subscriptions Extend file types and data sources Extend file types and data sources

Search Web PartSearch Web Part

Full-textFull-textIndexIndex

Full-textFull-textIndexIndex

Core Search ArchitectureCore Search Architecture

DocumentsDocumentsDocumentsDocuments

Content SourceContent Source GathererGatherer

IndexIndexEngineEngine

SearchSearchEngineEngine

Microsoft Microsoft Search ServiceSearch Service

NotificationsNotifications(optional)(optional)

Filter DaemonFilter Daemon

IFiltersIFilters

Protocol Protocol HandlerHandler

User User queriesqueries

Word-Word-breakersbreakers

StemmersStemmers

Custom Search ApplicationsCustom Search Applications

Integrate SharePoint Portal Server with Integrate SharePoint Portal Server with existing applicationsexisting applications

Use richer search featuresUse richer search features Write specialized queries:Write specialized queries:

Schema (e.g., your product line)Schema (e.g., your product line) Sorting (e.g., order by customer)Sorting (e.g., order by customer) Subscriptions (e.g., specs in my area)Subscriptions (e.g., specs in my area)

SQL ModelSQL Model

Extends SQL with full-text predicatesExtends SQL with full-text predicates

Relational SQLRelational SQL SQL in SearchSQL in Search

SELECT SELECT columnscolumns SELECT SELECT propertiesproperties

FROM FROM tabletable FROM FROM scopescope

WHERE WHERE conditioncondition WHERE WHERE conditioncondition**

*New features!*New features!

Database tables and Database tables and viewsviews

Document store and Document store and other indexed contentother indexed content

Sample SQL QuerySample SQL Query

SELECT "DAV:displayname"SELECT "DAV:displayname"

FROM Scope('deep traversal of FROM Scope('deep traversal of "/workspace1" ')"/workspace1" ')

WHERE WHERE "urn:schemas-microsoft:com"urn:schemas-microsoft:com:office:office#Author" = 'Sherri Hart':office:office#Author" = 'Sherri Hart'

AND AND FREETEXT('thermafill')FREETEXT('thermafill')

ORDER BY RankORDER BY Rank



WHERE WHERE "urn:schemas-microsoft:com"urn:schemas-microsoft:com:office:office#Author" = 'Sherri Hart':office:office#Author" = 'Sherri Hart'

AND AND FREETEXT('thermafill')FREETEXT('thermafill')

ORDER BY RankORDER BY Rank

Search ProgrammabilitySearch Programmability

Client AppClient AppClient AppClient App

OLE DB ProviderOLE DB Provider(MSDAIPP)(MSDAIPP)

OLE DB ProviderOLE DB Provider(MSDAIPP)(MSDAIPP)

WebDAVWebDAVListenerListenerWebDAVWebDAVListenerListener

DocumentDocumentStoreStore

DocumentDocumentStoreStore

ExternalExternalContentContentSourceSource

ExternalExternalContentContentSourceSource

ADOADOADOADO

ASP or Web PartASP or Web PartASP or Web PartASP or Web Part

WebDAV WebDAV SEARCHSEARCH

ADO CommandADO Command

OLE DB OLE DB CommandCommand

RecordsetRecordset

RowsetRowset



XML XML ResultsResults



HTTP/WebDAV Or ADO?HTTP/WebDAV Or ADO?

Use ADOUse ADO Integrate with existing ADO appsIntegrate with existing ADO apps Bind to data controls that require ADOBind to data controls that require ADO Prefer to program with ADOPrefer to program with ADO No control of result set sizeNo control of result set size

Use HTTP/WebDAVUse HTTP/WebDAV Bind directly to XML responseBind directly to XML response Use XSL to format resultsUse XSL to format results Control the size of the result setControl the size of the result set Prefer to program with XMLPrefer to program with XML

WebDAV SampleWebDAV Sample

ServerXMLHTTP object (MSXML3.0)ServerXMLHTTP object (MSXML3.0) Custom SEARCH verbCustom SEARCH verb Results returned in XMLResults returned in XML Custom header controls number and Custom header controls number and

paging of resultspaging of results Used by Search Web PartUsed by Search Web Part

Custom WebDAV SearchCustom WebDAV Search

Sample 1 - XMLHTTPSample 1 - XMLHTTP

Dim strQuery As StringDim strQuery As String

Dim xh As New ServerXMLHTTPDim xh As New ServerXMLHTTP

xh.open "SEARCH", strWorkspaceURL, xh.open "SEARCH", strWorkspaceURL, FalseFalse

xh.setRequestHeader "content-type", xh.setRequestHeader "content-type", "text/xml“"text/xml“

xh.send strQueryxh.send strQuery

xh.responseXML.save "sample1.xml"xh.responseXML.save "sample1.xml"

Dim strQuery As StringDim strQuery As String

Dim xh As New ServerXMLHTTPDim xh As New ServerXMLHTTP

xh.open "SEARCH", strWorkspaceURL, xh.open "SEARCH", strWorkspaceURL, FalseFalse

xh.setRequestHeader "content-type", xh.setRequestHeader "content-type", "text/xml“"text/xml“

xh.send strQueryxh.send strQuery

xh.responseXML.save "sample1.xml"xh.responseXML.save "sample1.xml"

Sample 1 – Query StringSample 1 – Query String

<?xml version="1.0" encoding="utf-8"?><?xml version="1.0" encoding="utf-8"?>

<a:searchrequest xmlns:a="DAV:"><a:searchrequest xmlns:a="DAV:">

<a:sql><a:sql>



WHERE "urn:schemas-microsoft-WHERE "urn:schemas-microsoft-com:office:office#Author" = 'Sherri com:office:office#Author" = 'Sherri

Hart'Hart'

AND FREETEXT('thermafill')AND FREETEXT('thermafill')

</a:sql></a:sql>

</a:searchrequest></a:searchrequest>

<?xml version="1.0" encoding="utf-8"?><?xml version="1.0" encoding="utf-8"?>

<a:searchrequest xmlns:a="DAV:"><a:searchrequest xmlns:a="DAV:">

<a:sql><a:sql>



WHERE "urn:schemas-microsoft-WHERE "urn:schemas-microsoft-com:office:office#Author" = 'Sherri com:office:office#Author" = 'Sherri

Hart'Hart'

AND FREETEXT('thermafill')AND FREETEXT('thermafill')

</a:sql></a:sql>

</a:searchrequest></a:searchrequest>

ADO SampleADO Sample

Applications can use ADO or OLE DB to Applications can use ADO or OLE DB to access search resultsaccess search results OLE DB Provider for Internet PublishingOLE DB Provider for Internet Publishing

Connect to workspace URLConnect to workspace URL Set command text using search syntaxSet command text using search syntax Rowset or recordset is constructed Rowset or recordset is constructed

from search result setfrom search result set No control of result set sizeNo control of result set size

Custom ADO SearchCustom ADO Search

Sample 2 – ADOSample 2 – ADO

Dim cmd As New ADODB.CommandDim cmd As New ADODB.CommandDim rs As ADODB.RecordsetDim rs As ADODB.Recordset

cmd.ActiveConnection = cmd.ActiveConnection = "Provider=MSDAIPP.DSO;Data "Provider=MSDAIPP.DSO;Data Source=" & strWorkspaceURLSource=" & strWorkspaceURL

cmd.CommandText = strQuerycmd.CommandText = strQuery

Set rs = cmd.ExecuteSet rs = cmd.Execute

Dim cmd As New ADODB.CommandDim cmd As New ADODB.CommandDim rs As ADODB.RecordsetDim rs As ADODB.Recordset

cmd.ActiveConnection = cmd.ActiveConnection = "Provider=MSDAIPP.DSO;Data "Provider=MSDAIPP.DSO;Data Source=" & strWorkspaceURLSource=" & strWorkspaceURL

cmd.CommandText = strQuerycmd.CommandText = strQuery

Set rs = cmd.ExecuteSet rs = cmd.Execute

Web Part SampleWeb Part Sample

Custom Web parts can execute Custom Web parts can execute searches and display resultssearches and display results

Use ServerXMLHTTP object to invoke Use ServerXMLHTTP object to invoke a search requesta search request

Use XSL to format search resultsUse XSL to format search results SDK Search Web Part SampleSDK Search Web Part Sample

Dictionary lookup of property valuesDictionary lookup of property values Reuses Search Results Web PartReuses Search Results Web Part

Custom Search Web PartCustom Search Web Part

SQL Syntax DetailsSQL Syntax Details

Supports MicrosoftSupports Microsoft®® full-text SQL full-text SQL extensionsextensions FROM clause scopesFROM clause scopes WHERE operators and expressionsWHERE operators and expressions Words\phrases – CONTAINSWords\phrases – CONTAINS Natural language – FREETEXTNatural language – FREETEXT Result orderingResult ordering

FROM ClauseFROM Clause

Specifies scope of searchSpecifies scope of search

FROM Meaning ExampleShallow Shallow traversaltraversal

This This folderfolder

FROM SCOPE(‘FROM SCOPE(‘shallowshallow traversal of traversal of “/workspace1/myfolder”’)“/workspace1/myfolder”’)

Deep Deep traversal traversal (default)(default)

This This folder or folder or any any below itbelow it

FROM SCOPE(‘FROM SCOPE(‘deepdeep traversal of traversal of “/workspace1”’)“/workspace1”’)

WHERE ClauseWHERE Clause

Boolean operatorsBoolean operators AND; OR; AND NOTAND; OR; AND NOT

Arithmetic operatorsArithmetic operators >, <, =, !=, <=, >=>, <, =, !=, <=, >=

Regular expressions on propertiesRegular expressions on properties LIKE; MATCHESLIKE; MATCHES

……WHERE “DAV:getcontentlength” > WHERE “DAV:getcontentlength” > 2500025000

……WHERE “DAV:getcontentlength” > WHERE “DAV:getcontentlength” > 2500025000

CONTAINS( )CONTAINS( )

Matches exact words and phrasesMatches exact words and phrases SupportsSupports

Prefix matchingPrefix matching Boolean conditionsBoolean conditions Proximity (NEAR)Proximity (NEAR)

……CONTAINS(“…#Author", ‘”sherr*”')CONTAINS(“…#Author", ‘”sherr*”')……CONTAINS(“…:description”,CONTAINS(“…:description”,

'thermafill AND patent')'thermafill AND patent')……CONTAINS(“…:description”,CONTAINS(“…:description”,

‘ "thermafill" NEAR "patent“ ')‘ "thermafill" NEAR "patent“ ')

……CONTAINS(“…#Author", ‘”sherr*”')CONTAINS(“…#Author", ‘”sherr*”')……CONTAINS(“…:description”,CONTAINS(“…:description”,

'thermafill AND patent')'thermafill AND patent')……CONTAINS(“…:description”,CONTAINS(“…:description”,

‘ "thermafill" NEAR "patent“ ')‘ "thermafill" NEAR "patent“ ')

FREETEXT( )FREETEXT( )

Uses advanced probabilistic ranking to Uses advanced probabilistic ranking to return best resultsreturn best results

Intended for natural Intended for natural language querieslanguage queries

Used by dashboard basic searchUsed by dashboard basic search

… … WHERE FREETEXT('extreme sports WHERE FREETEXT('extreme sports customer research')customer research')ORDER BY Rank DESCORDER BY Rank DESC

… … WHERE FREETEXT('extreme sports WHERE FREETEXT('extreme sports customer research')customer research')ORDER BY Rank DESCORDER BY Rank DESC

Ordering And CoercionOrdering And Coercion

RankRank measures the relevance of a measures the relevance of a document document to a predicate or queryto a predicate or query

Coercion functions adjust the rank Coercion functions adjust the rank for predicatesfor predicates

Promote results (e.g., Best Bets)Promote results (e.g., Best Bets)

… … FREETEXT(Subject, 'extreme FREETEXT(Subject, 'extreme sports') sports')

ORDER BY ORDER BY Rank Rank DESCDESC

… … (FREETEXT(*, 'extreme sports')(FREETEXT(*, 'extreme sports')RANK BY RANK BY COERCION(multiply, COERCION(multiply,

0.5))0.5))RANK BY RANK BY COERCION(add, 500)COERCION(add, 500)

… … FREETEXT(Subject, 'extreme FREETEXT(Subject, 'extreme sports') sports')

ORDER BY ORDER BY Rank Rank DESCDESC

… … (FREETEXT(*, 'extreme sports')(FREETEXT(*, 'extreme sports')RANK BY RANK BY COERCION(multiply, COERCION(multiply,

0.5))0.5))RANK BY RANK BY COERCION(add, 500)COERCION(add, 500)

New Search FeaturesNew Search Features

Attribute weightsAttribute weights Group aliasesGroup aliases Quoting for phrase matchingQuoting for phrase matching COALESCE_TABLECOALESCE_TABLE Accent foldingAccent folding Subscriptions (persistent queries)Subscriptions (persistent queries)

Attribute WeightsAttribute Weights

Adjusts the contribution of properties Adjusts the contribution of properties to a predicate’s rankto a predicate’s rank

Tunes queries by emphasizing some Tunes queries by emphasizing some properties over othersproperties over others

……FREETEXT((“…#Title" FREETEXT((“…#Title" WEIGHTWEIGHT(0.8), (0.8), Contents Contents WEIGHTWEIGHT(0.5)), (0.5)),

'thermafill')'thermafill')

… … FREETEXT((FREETEXT((*:1.0*:1.0, , "DAV:href":0.0"DAV:href":0.0), ), 'wolverine gloves')'wolverine gloves')

……FREETEXT((“…#Title" FREETEXT((“…#Title" WEIGHTWEIGHT(0.8), (0.8), Contents Contents WEIGHTWEIGHT(0.5)), (0.5)),

'thermafill')'thermafill')

… … FREETEXT((FREETEXT((*:1.0*:1.0, , "DAV:href":0.0"DAV:href":0.0), ), 'wolverine gloves')'wolverine gloves')

Group AliasesGroup Aliases

Groups properties and weights into Groups properties and weights into composite propertiescomposite properties

Syntactic shortcut to ease Syntactic shortcut to ease writing of querieswriting of queries

. . .WHERE . . .WHERE WITH (“…#Subject", “…WITH (“…#Subject", “…#Title“)#Title“)

AS #RelevantPropertiesAS #RelevantPropertiesFREETEXT(FREETEXT(#RelevantProperties#RelevantProperties, ,

'eagle competitive outlook')'eagle competitive outlook')

. . .WHERE . . .WHERE WITH (“…#Subject", “…WITH (“…#Subject", “…#Title“)#Title“)

AS #RelevantPropertiesAS #RelevantPropertiesFREETEXT(FREETEXT(#RelevantProperties#RelevantProperties, ,

'eagle competitive outlook')'eagle competitive outlook')

Phrase MatchingPhrase Matching

FREETEXT can rank results based FREETEXT can rank results based on phraseson phrases

Indicate phrase with beginning and Indicate phrase with beginning and ending double-quote (“)ending double-quote (“)

……WHERE FREETEXT(Contents, WHERE FREETEXT(Contents, ‘ "eagle 99 competitive outlook“ ')‘ "eagle 99 competitive outlook“ ')

……WHERE FREETEXT(Contents, WHERE FREETEXT(Contents, ‘ "eagle 99 competitive outlook“ ')‘ "eagle 99 competitive outlook“ ')

RankRank= 1000= 1000

RankRank= 999= 999

RankRank600600toto00

RankRank999999toto

500500

Portal Search QueryPortal Search Query

Exact match on Exact match on BestBets valueBestBets value

CONTAINS CONTAINS predicate on predicate on

BestBets valueBestBets value

FREETEXT FREETEXT predicate on predicate on

BestBets valueBestBets value

FREETEXT FREETEXT predicate on predicate on

weighted weighted propertiesproperties AttributeAttribute

WeightingWeighting

RankRankCoercionCoercion

BestBestBetBet

BestBestBetBet

10001000toto

500500

10001000toto

500500

Subscription ObjectSubscription Object

Enhances search by adding “push” Enhances search by adding “push” model for resultsmodel for results

Server-side APIServer-side API Adds or deletes subscriptions on:Adds or deletes subscriptions on:

CategoriesCategories Files or foldersFiles or folders SQL predicatesSQL predicates

Immediate or periodic notificationImmediate or periodic notification

Subscription SampleSubscription SampleSet oSubMgr = Set oSubMgr =

CreateObject("PKM.SubscriptionManCreateObject("PKM.SubscriptionManager")ager")

strSubscriptionUrl = strSubscriptionUrl = oSubMgr.CreateSubscription(_oSubMgr.CreateSubscription(_

strWorkspace,_strWorkspace,_ ‘‘workspace workspace URLURL

strQuery, _strQuery, _ ‘‘SQL predicateSQL predicate4, _4, _ ‘‘Type = SQL Type = SQL

predicatepredicate1, _1, _ ‘‘ImmediateImmediatestrEmail, _strEmail, _ ‘‘Send to Send to

addressaddressstrDesc, _strDesc, _ ‘‘DescriptionDescription……))

Set oSubMgr = Set oSubMgr = CreateObject("PKM.SubscriptionManCreateObject("PKM.SubscriptionManager")ager")

strSubscriptionUrl = strSubscriptionUrl = oSubMgr.CreateSubscription(_oSubMgr.CreateSubscription(_

strWorkspace,_strWorkspace,_ ‘‘workspace workspace URLURL

strQuery, _strQuery, _ ‘‘SQL predicateSQL predicate4, _4, _ ‘‘Type = SQL Type = SQL

predicatepredicate1, _1, _ ‘‘ImmediateImmediatestrEmail, _strEmail, _ ‘‘Send to Send to

addressaddressstrDesc, _strDesc, _ ‘‘DescriptionDescription……))

Document FiltersDocument Filters

Out-of-box support for text, Office, Out-of-box support for text, Office, HTML, TIFFHTML, TIFF

Third-party filters available for PDF, Third-party filters available for PDF, WordPerfect, . . .WordPerfect, . . .

IFilter interface reference in IFilter interface reference in Platform SDKPlatform SDK

Filters work for Indexing Services and Filters work for Indexing Services and SharePoint Portal ServerSharePoint Portal Server

Protocol HandlersProtocol Handlers

Out-of-box support for file system, Out-of-box support for file system, HTTP, Exchange and Lotus NotesHTTP, Exchange and Lotus Notes

Working with DM vendors on third-party Working with DM vendors on third-party implementationsimplementations

Resource Kit will ship sample PH for Resource Kit will ship sample PH for SharePoint Team ServicesSharePoint Team Services

Interface reference and sample included Interface reference and sample included in SharePoint Portal Server SDKin SharePoint Portal Server SDK

SummarySummary

Customizing Search is easy!Customizing Search is easy! Powerful syntax for property and Powerful syntax for property and

full-text searchesfull-text searches Choice of API to match application and Choice of API to match application and

programming preferenceprogramming preference Callable from within Web PartsCallable from within Web Parts Methods to manage subscriptionsMethods to manage subscriptions

ResourcesResources

SharePoint Portal Server Web SiteSharePoint Portal Server Web Sitehttp://www.microsoft.com/Servers/Sharepoint/http://www.microsoft.com/Servers/Sharepoint/

SharePoint News GroupsSharePoint News Groupshttp://www.microsoft.com/servers/sharepoint/http://www.microsoft.com/servers/sharepoint/newsgroup.htmnewsgroup.htm

Digital Dashboard Web SiteDigital Dashboard Web Sitehttp://www.microsoft.com/digitaldashboardhttp://www.microsoft.com/digitaldashboard

Microsoft Training and CertificationMicrosoft Training and Certificationhttp://www.microsoft.com/trainingandservices/http://www.microsoft.com/trainingandservices/default.aspdefault.asp

http://www.microsoft.com/Servers/Sharepoint/

ResourcesResources

MSDNMSDN SharePoint Portal Server 2001 as a SharePoint Portal Server 2001 as a

Collaborative Solutions PlatformCollaborative Solutions Platformhttp://msdn.microsoft.com/library/techart/tahoe.htmhttp://msdn.microsoft.com/library/techart/tahoe.htm

SharePoint Portal Server – Document SharePoint Portal Server – Document Management and Much MoreManagement and Much Morehttp://msdn.microsoft.com/library/periodic/period00/http://msdn.microsoft.com/library/periodic/period00/MSTahoe.htmMSTahoe.htm

PartnersPartners Digital Dashboard Solution ProvidersDigital Dashboard Solution Providers

http://www.microsoft.com/business/digitaldashboard/http://www.microsoft.com/business/digitaldashboard/ddpartners.aspddpartners.asp

Microsoft Direct AccessMicrosoft Direct Accesshttp://www.microsoft.com/directaccesshttp://www.microsoft.com/directaccess