bringing solr to drupal a general and library specific use case peter kiraly

8/8/2019 Bringing Solr to Drupal a General and Library Specific Use Case Peter Kiraly

1/46

Bringing Solr to DrupalA General, and a Library-Specific Use Case

1
http://mail.google.com/mail/?ui=1&view=att&th=11e8ab72613d00d3&attid=0.1&disp=attd&zw


2/46

Two ways of using Solr in Drupal

2

General solution: Apache Solr Search Integration and related

modules: Stats, Autocomplete, Multisite, Ajax, Biblio,

-commerce integration), Views,Multilingual, Geospatial and many othersFits for the overall needs, uses predefined fields

Library specific solution: eXtensible Catalog modules

Fits for library needs, uses dynamic fields


3/46

Two ways of using Solr in Drupal I.

3



-commerce integration), Views,Multilingual, Geospatial and many othersFits for the overall needs, uses predefined fields



This part of the presentation is based on the works and previous presentations of

Robert Douglass.


4/46

Drupal architecture

4

Relational database

for storageSolr index for search


5/46

Purposes

5

Creating a general insfrastructure which is usable in every

Drupal installation

Core module, and additional module for covering specific Solr

functionalities (statistics, autocomplete etc.)

Replace the original (and still default) MySQL-based search

feature


6/46

6

sort

Facet 1

Facet 2


7/46

7

categories

API version

module search

sort

List of modules page


8/46

Advanced search: issues

8


9/46

Whitehouse.gov on Drupal & Solr

9

sort

facets


10/46

Boosting by Drupal specific properties

10


11/46

Boosting and ignoring by document type

11


12/46

Boosting by fields / HTML tags

12


13/46

More like this implementation

13


14/46

Solr in Views integration

14

Views is a very popular module, helps creating interactive DB queries and

result pages. Now it can handle Apache Solr as data source.


15/46

Part of the Views admin page

15

You can specify fields,

sorting, filters, layout,

arguments,

behaviours and more


16/46

Using Tika: file search

16


17/46

Indexing/searching multiple sites

17


18/46

Search in comments

18


19/46

CCK date searching

19

Content Construction Kit: popular module to create document and

field types. CCK date is a special field type handling dates.


20/46

statistics

20

impressive numbers


21/46

Statistics of facet usage

21


22/46

autocomplete

22


23/46

Future plans

23

Crawling with Nutch

Geospatial search

eDismax (Solr 1.5)

Drupal 7

API changes

Improving documentation


24/46

People behind these modules

24

Robert Douglas (DE) http://drupal.org/user/5449

Alejandro Garza (MX) http://drupal.org/user/153120

Peter Wolanin (US) http://drupal.org/user/49851

James McKinney (CA) http://drupal.org/user/472460

Scott Reynolds (US) http://drupal.org/user/60009

Mike O'Connor (US) http://drupal.org/user/104525

Markus Kalkbrenner (DE) http://drupal.org/user/124705


25/46

Links

25

apachesolr http://drupal.org/project/apachesolr (this is the best starting point)

content recommendation patch http://drupal.org/node/372767

views integration http://drupal.org/project/apachesolr_views,http://acquia.com/node/911667

file search http://drupal.org/project/apachesolr_attachments,http://acquia.com/node/1129446

date facet for CCK field http://drupal.org/node/558160

statistics http://drupal.org/project/apachesolr_stats

multisite http://drupal.org/project/apachesolr_multisitesearch

autocomplete http://drupal.org/project/apachesolr_autocomplete


26/46

Two ways of using Solr in Drupal II.

26



-commerce integration), Views,Multilingual, Geospatial and many others

Fits for the overall needs, uses predefined fields




27/46

About eXtensible Catalog

27

an FRBR-based metadata platform

started in 2007

driven by new theories of library science, cultural anthropology andthe practice of web 2.0, library 2.0

Universities of Rochester, Notre Dame, Cornell, North Carolina andCARLI consortium


28/46

Architecture

28

Drupal CMS

Integrated Library System Repository

XC Metadata

Services Toolkit

XC OAI ToolkitXC NCIP Toolkit

MARC Normalization

DC Normalization

Aggregation

Transformation

XC Drupal

Toolkit

circulation data


29/46

Purposes of XC Drupal Toolkit

29

integrate library data into a popular content management system

customizable functionalities

customizable interface(s)

internalization, localization

5000+ custom modules, 20+ library specific modules

wide range on mashup options all features are available through user interfaces


30/46

Search results

30

highlighted terms

availability information

bibliographical data

cover images

facets


31/46

Customized interface (Kyushu University)

31


32/46

Similar documents

32


33/46

XML attribute handling

33

History

subject=History and subject_OCLC=Historysubject_OCLC=Historysubject=Historynone


34/46

Mapping schema fields to Solr types

34


35/46


36/46

Custom PHP code for displaying title

36
http://mail.google.com/mail/?ui=1&view=att&th=11e8ab72613d00d3&attid=0.1&disp=attd&zwhttp://mail.google.com/mail/?ui=1&view=att&th=11e8ab72613d00d3&attid=0.1&disp=attd&zw


37/46

Getting large amount into Drupal: harvesting

37

Harvest is running

List of scheduled harvests


38/46

Data flow between components

38

OAI-PMH

providerDrupal

MySQL Solr

batch delete/insert documents

creating nodes


39/46

39


40/46

Creating highlighter

40


41/46

Setting up field and date facet properties

41

Date facet properties

Field facet properties


42/46

42

List of facets, andtheirs type

General properties


43/46

Reordering facets

43

Just drag and drop


44/46

Using facet term list in search form

44

dropdown

fullfilled with

language

facet termsdropdown definition


45/46

Adding widgets to UI: navigation bar

45

definition of navigation bar

navigation bar in action


46/46

Links Project page http://eXtensibleCatalog.org

XC Drupal Toolkit http://drupal.org/project/xc

Metadata Services Toolkit http://code.google.com/p/xcmetadataservicestoolkit

OAI Toolkit http://code.google.com/p/xcoaitoolkit

NCIP Toolkit http://code.google.com/p/xcnciptoolkit

Developers:

Mlen-Too Wesley (GH) http://drupal.org/user/318924

(H) http://drupal.org/user/352587, http://twitter.com/kiru

bringing solr to drupal a general and library specific use case peter kiraly

Documents