evaluation of an ecommerce search indexing system: solr as a search solution in a leading digital...

7/21/2019 Evaluation of an eCommerce Search Indexing System: SOLR as a Search Solution in a Leading Digital Retail Brand

http://slidepdf.com/reader/full/evaluation-of-an-ecommerce-search-indexing-system-solr-as-a-search-solution 1/22

Evaluation of an eCommerce Search

Indexing System: SOLR as a Search

Solution in a Leading Digital Retail Brand

Author: 9621548

Course: MSc Management and Information Systems: Change and Development

Module: IPDM70001 Fundamentals of Information and Informa tion Systems

University of Manchester, 2015

Tutor: Richard Heeks

Word Count: 3,691

I’d like to extend my gratitude to Paul Jackson, for allowing me to use the company and this specific

system as a subject for academic study. I’d also like to thank Tom Ki ddle for supporting my studies,

b th ith ti d fl ibilit t k Finally I’d like to thank Chris Herbert for putting me into contact



both with time and flexibility at work Finally I’d like to thank Chris Herbert for putting me into contact

9621548

Contents1.0 Company Background ............................................................................................................. 2

2.0 Description of the System ....................................................................................................... 2

A Software Component .............................................................................................................. 2

B People ..................................................................................................................................... 3

3.0 SOLR as a tool within the business process ............................................................................ 33.1 Capture ................................................................................................................................ 4

3.2 Input .................................................................................................................................... 4

3.3 Processing ........................................................................................................................... 4

3. 4 Storage ................................................................................................................................ 4

3. 5 Output ................................................................................................................................. 4

3.6 Description of Business Requirements ............................................................................... 5

4.0 Is SOLR a successful IS? ........................................................................................................... 5

4.1.1 Information Quality ......................................................................................................... 6

4.1.2 Systems Quality ............................................................................................................... 6

4.1.3 Service Quality ................................................................................................................ 7

4 2 Intention to use and actual use 8



9621548

storage where WCS can access it. The Management Centre tool is a user interface which allows

business users to set up rules which define the logic by which SOLR creates the index. It has side

functions, including creating promotions, and managing ‘eSpot’ promotional content on the website.

Finally, a metrics dashboard collects data on the customer search and browse journeys, including

conversion to sales. The customer channels which use SOLR are the UK website, the R2 apps, and the

New Store Browser (NSB) – an app for use within physical stores.

B People

The system has a bi-directional nature (customers & business users). The customer interacts

with SOLR (albeit indirectly) when they enter a search term into the website, or click through a browse

journey, and are presented with a response. Within the business, the Merchandising team create the

search indexing rules, the Content team place promotional content on the website, the Analytics team

collate and report the results of the metrics collected, and the Operations team make decisions on the

results.

3.0 SOLR as a tool within the business process

Maintenance of an effective search tool is vital to eCommerce (Zwass, 2005). Consumers make



9621548

3.1 Capture

Data is fed into this system from other systems. The Product Information System exports product and

category data to the WCS storage database. The user interaction passes first through WCS before it

calls the SOLR index. As such, the information captured by SOLR is in the form of messages from WCS:

from the database, the website’s java code, or API calls from other channels.

3.2 Input

User workstations are the only node of business input into SOLR. This input is in the form of business

rules entered into the Management Center UI – which only functions in Internet Explorer 8. Customer

input is at a more remote stage, comprising users entering search terms, or clicking through browse

journeys on the site. This takes place across many channels and devices, including desktop and laptop

computers, Android and iOS phones, and tablets.

3.3 Processing

There are two stages of SOLR indexing. The first is pre-processing, which extracts the data stored in

the WCS tables, changes its format into a flat file (a file with no structured relationships), and stores

it temporarily. From here, it builds an index based on the logic it has constructed from the business



9621548

which uses the reference points created by the index to prompt a quick response with the necessary

group of information in the payload to be handled by the end channel.

3.6 Description of Business Requirements

The ‘war room’ team (combination of design, build, stakeholders and project management) came up

with 21 user cases which they believed to be ‘must-have’ functionality. This was created from

workshops with stakeholders (in this case Content and Merchandising teams) to cover all the

functionality they needed from the system to do their jobs. A detailed description is included in

Appendix 1, but the basic functions are listed below:

1.

Modify Order of Search Results

2.

Specify Top Search Results

3. Removing Products from Search Results

4.

De-prioritise Products from Search Results

5.

Add or Replace Search Criteria (Search Results)

6.

Assign Replacements (Search Term Associations)7. Assign Synonyms (Search Term Associations)

8.

Create a Landing Page Re-Direct (Search Term Associations)

9.

Reviewing Site Search Statistics

10.

Create Canned Results Set

11. Providing a URL Link Associated with a Set of Products for a Marketing Campaign

12.

Changing the Display Sequence of Products on a Browse Lister

13.

Hide a Product from Search and Browse

14.

Create a Pseudo Attribute Category (not currently used )

15 S if W R d P d t (L di d Li t )



9621548

4.1.1 Information Quality

CARTA is often a good way to assess information quality (completeness, accuracy, relevance,

timeliness, and appropriateness). DeLone and McLean suggest a slightly different set of measures,

based on the academic literature. Rather than appropriateness, they instead include consistency. They

conclude that from the studies they looked at, these factors had a positive correlation with individual

impact, which was measured by decision-making performance, job effectiveness, and quality of work.

Completeness – SOLR provides users with a wide set of metrics. As a result, the merchandising team

are able to make decisions about which rules to set up in Management Center. However, during the

implementation, tagging was not applied to the Apps, and the business lost the ability to collect

metrics on these Apps. Considering that in peak4 2014, for the first time, sales from tablets and mobile

devices rose above 50% of all total sales – much of this from the Apps – this is a significant flaw in the

Completeness of the information.

Accuracy – It is difficult to measure the accuracy of the information when it is analysed from millions

of visits. There is some confidence that the numbers are accurate, as the operations team often report

that it matches the content of calls received to the Contact Centres (customer services).

Relevance – Users can filter the information they receive in many ways, and set up charts with the

i bl th h A th h di i t h k f ti t d ‘ lt ’



9621548

facets set in Management Center is not applied to the SOLR index. There also exists a defect where

search rules to remove products from search is are not effective.

4.1.3 Service Quality

This looks at up-to-date hardware and software, dependability, prompt service to users, user

knowledge to do their job, and whether the users’ best interests are at the heart of the system.

Up-to-date hardware and software – The hardware and software business users have to access the

system are, for the most part, desktop computers running Windows XP, and using Internet Explorer

8. This operating system and browser are supported by Management Center (indeed, it won’t run on

other browsers), and therefore it suits the needs of the business. Other systems which are commonly

used by the business also fit these conditions, so there’s an integration with the business’ ability to

use the software.

Dependability – the SOLR index, once created, is completely stable6. However, there have been

instances where the index has fallen over in creation. These have often been due to environment

instability and poor data feeding into SOLR from back end systems (such as the Mainframe, and MMDB

[the company’s old product data management system7]). Whilst issues such as infrastructure have

caused instability, rather than the SOLR & Management Centre system itself, this does impact users’

ti f d l ti hi ith th t



9621548

Users’ best interests at heart – There can often be a perception that IS does not have users’ best

interests at heart, not exclusive to this company or system, but more widely within business. The

change management team worked closely with users to ensure them that this system was built with

end use in mind, and that the teams involved would be enabled much more by it. A section of a

communications pack delivered to and shared with the teams using the system is included as Appendix

3.

4.2 Intention to use and actual useThe DeLone and McLean model shows that information quality, systems quality and service quality

have a causal relationship with intention to use (DeLone and McLean, 2003). However, this is not

completely relevant in the company, as the merchandising team’s role can only be performed by using

SOLR. Now the system has been implemented, the search functionality of the site can only be enabled

by the system. Whatever the complaints, criticisms, and downfalls of the system, there are no plans

in the pipeline (to the author’s knowledge) to replace it. There are no documented examples of people

consciously objecting to using the system, but there are some user complaints about the system – and

the necessity for colleagues to use it feeds into low level discontent towards it.

4.3 Net benefit

Net benefit for this system can be looked at by both the functional business requirements and whether



9621548

there has been a reduction in the percentage of customers participating in search of approximately

10%.

4.3.2 Browse Visit Participation

Fig 2.2 shows the percentage of customers who visit the site, but this time those who participate in a

browse journey (clicking through the category structure), comparing before and after SOLR was

introduced. Again, we see a reduction in the percentage of customers participating in this journey,

again, of approximately 5%8.

Fig. 2.2



9621548

4.0 So is SOLR a successful IS?

SOLR can be called a partial failure as a results of 3 things. First, its inability to meet the

business functional requirements which were the reason this OOTB software was selected above

others. Then, the complete lack of metrics for Apps, which don’t allow information on user journeys

for a key part of the company’s sales to be analysed by the Analytics team , for decision making from

the Merchandising team. Finally, the poor statistics showing decreased Search and Browse journey

participation, and increased ‘no results’ pages when comparing before and after the system’s

deployment. It is certainly not a complete failure, as the system was not abandoned, and there are no

plans in the short term to replace the system. It is used, and does allow the basic functionality required

of it in providing an index for the WCS site to interact with. Its failure areas all lie in the need from the

business to optimise sales through Search and Browse, and the poor ability of the system to

accommodate this, for the reasons mentioned above.

5.0 Causes of the partial failure of SOLR as an information system

It has been established that SOLR and the search implementation project was a partial failure.

The reasons this occurred and the system does not perform as required can be looked at using the

factor model of IS success and failure. This takes a series of factors and for each compares the main

causes of failure and the main causes of success relating to it. A generic diagram of the model can be

f d i di 4



9621548

was kept on track. Decision making was for the most part hybrid, incorporating the needs of the

business as a priority, and using all the available information. This was as a result of the cross-party

steering groups – with representation from all key areas – meeting regularly to inform the rest of the

group on each area, and contribute to the decision making process. It was rare that decisions were

made hierarchically, and not as a result of agreement from this group.

The OOTB software chosen met the majority of the key business requirements. Those it did

not meet were accounted for, and workarounds were developed (i.e. running manual SQL commands

after the index to restructure categories with no set sequence to alphabetical). What was a problem

was the ability of SOLR to meet those designs within the specific context of the company’s expansive

digital estate. Integration with the highly customised WCS platform was an issue, and the rate of

change in other areas – such as the new apps and the new product management system – meant that

the technical design was out of date by the time the system was in use.

There are two competencies in question: the first of the internal colleagues to perform their

daily tasks. This does not suffer from a lack of knowledge and skill, as users are able to execute their

roles within the confines of the system. The other area of competency is that of the third-party to

implement the system per design, making the suitable customisations from the OOTB package to suit

th l di it l t t Th i t f t i i th bilit f th thi d t t b i



9621548

ReferencesBocij, P., Chaffey, D., Greasley, A., Hickie, S. (2015), Business Information Systems, 5th ed., Pearson

Education, London.

Cleary, T. (1998), Business Information Technology , FT Prentice-Hall.

DeLone, W. H. & McLean E. R. (2003), The DeLone and McLean Model of Information Systems

Success: A Ten Year Update, Journal of Management Information Systems, Spring 2003, Vol. 19. No.

4, pp. 9-30.

The Group (2013), Annual Report and Financial Statements [accessed on 27/02/2015].

Heeks, R. B. & Morgan, S. (2015), Course unit content , Fundamentals of Information and Information

Systems, IDPM, University of Manchester, Manchester.

Zwass, V. (2005), ‘Editor’s Introduction’ , International Journal of Electronic Commerce, Fall 2005, Vol.

19, No. 1, pp. 5-7.

Internal Documents

Company Transformation Plan Summary , created 02/01/2013, last ed. 27/01/2014.

Search Replacement Stakeholder Comms Workshop Write Up, based on workshop held 26.07/2013.

‘Setting the Scene’: Management C entre & Search Replacement , presented 12/03/2014.

S l ti O ti P d t S h A li ti V d S l ti d 12

th

O b 2012



9621548

Appendices

Appendix 1: To-Be Business Functions as emerged from the business stakeholder design meetings.

Appendix 2: Further list of Business Functions incorporating technical scope.

Appendix 3: Section of Change Management Communications Pack to Key Stakeholders.

Appendix 4: Factor model of IS success and failure.



9621548

ii | P a g e Appendix 1: To-Be Business Functions as emerged from the business stakeholder design meetings.



9621548

iii | P a g e



9621548

iv | P a g e



9621548

v | P a g e

Appendix 2: Further list of Business Functions incorporating technical scope.



9621548

vi | P a g e



9621548

vii | P a g e

Appendix 3: Section of Change Management Communications Pack to Key Stakeholders.



9621548

viii | P a g e



9621548

ix | P a g e

Appendix 4: Factor model of IS success and failure.