search technologies assessment · web viewupdated content based on “drcf_nac r1p2 combined...

National Archives and Records Administration

National Archives Catalog (The Catalog)

NARA Catalog Web Sites Data Model Design– Catalog Perspective –

Status-FinalVersion 1.4

July 24, 2015

National Archives & Records Administration

NARA Catalog Web Sites Data Model Design

Avi Rappoport, Madhu Koneni, Kristen Martin, Terri Hobbs, Paul Nelson

Version 1.43

Contract Number GS-35F-0541U

Order Number NAMA-13-F-0120

July 24, 2015

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=owCr1QpR6RHqlM&tbnid=Nzjz_YVq0miSxM:&ved=0CAUQjRw&url=http://blogs.archives.gov/aotus/?attachment_id=1236&ei=51H6Ub_GHdO64APu34H4Aw&bvm=bv.50165853,d.dmg&psig=AFQjCNEMljwv59HAB11EckpVc6CXxxOUkA&ust=1375445858498456

NARA Catalog Web Sites DMD

Contents

1 Overview................................................................................................................21.1 NARA Web Site Content.....................................................................................................2

1.2 What is a DMD?.................................................................................................................3

1.3 Document Conventions.....................................................................................................4

2 Web Sites and Content Extraction...........................................................................52.1 Web Sites and Sub-Sites....................................................................................................5

2.2 Standard Metadata............................................................................................................8

2.3 Metadata Parsing for the Archives.gov Site.......................................................................9

2.4 Metadata Parsing for NARA Blogs....................................................................................10

2.4.1 Parsing Date for AOTUS Blog...................................................................................11

2.4.2 Metadata Parsing for other National Archives Blogs...............................................11

2.5 Presidential Libraries.......................................................................................................12

2.6 Processing Non-HTML content........................................................................................13

2.6.1 Text Extraction........................................................................................................13

2.6.2 Metadata Mapping for Non-HTML Content............................................................13

3 File Processing......................................................................................................14

4 Mapping to Index Fields........................................................................................154.1 Field Metadata Mappings................................................................................................15

4.2 Mapping to Keywords Relevancy Model..........................................................................16

5 Search Fields.........................................................................................................17

6 Search Results Presentation..................................................................................186.1 Search Results (aka “Brief Results” Display)....................................................................18

7 Content Details Presentation................................................................................19

Version Control

Version Date Reviewer Summary Description

0.1 2013-10-25Paul Nelson and Madhu Koneni

Initial Outline

0.2 2013-12-30 Rhea Mandavilli Updates to Sections 1, 6.1.1, 7

0.3 2014-1-2Archana Ballur Nagaraj

Updates to Section 2, 3, 4, 5, 6.1, 6.2(ongoing)

0.4 2014-1-3 Avi RappoportAdding blogs, presidential libraries, updates to all sections

0.5 2014-02-07 Madhu KoneniFeedback from NARA on 02/04/2010 addressed

0.6 2014-02-22 Elizabeth Hobbs Reformatted

0.7 2014-02-23 Elizabeth HobbsRemoved websites not being considered. Updated index mappings and search results.

1.0 2014-02-24 Paul Nelson Top to bottom review and cleanup.

1.1 2014-03-19 Lisong LiuUpdated based on NARA review and feedback in DCRF of 3/11/14

1.2 2014-11-14 Kristy MartinRemoved “Confidential to Search Technologies” text from the footer.

1.3 2015-06-11Jose Hernandez, Kristy Martin

Added the remaining sites specified in DE-60; section updated: 2.1

Changed branding for system name throughout document.

1.4 2015-07-24 Kristy Martin

Updated content based on “DRCF_NAC R1P2 combined 1b_Increment 2 Design_IQS_Consolidated_7_21_15V1 (1).docx”

1

1 Overview

This is the Data Model Design (DMD) for NARA Web Sites data source, which includes archives.gov (and sub-sites such as blogs) and the presidential library web sites.

This document aims at providing detailed documentation of all fields which come from these web sites and how they will be processed through the National Archives Catalog system. The DMD identifies and defines the following for the Web Sites data source:

Metadata elements parsing

Mapping of the metadata elements to the Search Engine Index fields

How the search results are formatted (brief results)

URLs indexed into the Catalog from web sites are unusual (when compared to other Catalog data sources) in that they do not participate in many aspects of the Catalog:

Web pages cannot be annotated (tags, comments, translations, or transcriptions)

There is no “content detail” for web pages.

o Instead the user is simply taken directly to the web page.

There are no custom advanced search fields for web pages.

Therefore, these sections will not be required in this DMD.

1.1 NARA Web Site ContentThe main archives.gov web site is extremely valuable for introductory and reference material about the many resources of the National Archives, from detailed databases to exhibitions. These pages answer common questions of the general public and beginning searchers. In addition, blogs within the archives.gov domain also provide useful information and insights into current issues and activities of the National Archives and Records Administration.

Presidential Libraries are archives for preserving and making accessible papers, records, and other historical materials of U.S. Presidents and their administration for public study and discussion without regard for political considerations or affiliations. Presidential Libraries are important sources for historians, researchers and anyone studying our presidents and history.

Web pages indexed from these resources are presented as part of search results, allowing users easy access into this information.

2

1.2 What is a DMD?The purpose of a Data Model Design document (DMD) is to document and describe all relevant data fields from a data source necessary to support all Catalog functionality. This metadata includes all data fields and their structure (nesting, type, number of values, etc.).

The DMD further describes how metadata values are transformed and stored within the Catalog. This careful accounting of data processing is require to gain a complete understanding of how every field is handled through the Catalog system.

Finally, the DMD also describes how metadata values are presented to the user, in the brief results, on the content detail page (aka the “full results”), from API calls, and in various metadata downloads.

The following diagram shows how metadata is processed and mapped for the Web Sites DMD:

WebCrawler

Extract Content

(section 2)

Map toSearch Engine

Index (section 4)

SearchEngine

Index

ApplicationServer

Map toBrief Results(section 6)

BriefResults

Note that the major section numbers are maintained across all DMDs. So even though sections 3 and 5 are not required in this DMD, empty place-holders will remain so that numbering is consistent when comparing the Web Sites DMD with other DMDs for other data sources.

Specifically, this DMD includes the following:

Extracting metadata and text content from web pages.

o This includes key metadata fields (title, web area) as well as text content.

Index Representation (section 4)

o How the web site metadata fields are represented in the search engine indexes

Brief results presentation (section 6)

o Identifies how index fields are mapped to show the brief results for web pages.

3

1.3 Document ConventionsSince there are many different metadata fields for different purposes and from different systems, field mappings will be used throughout this document to clearly identify the originating source for every field, as follows:

Abbrev Description

WEB This abbreviation will be used for any metadata from a web page.For example: WEB/title will represent the <title></title> field extracted from the web page.

I Fields from the search engine index.For example, “I/title” will represent the title as it is stored in the search engine index for the record.

4

2 Web Sites and Content Extraction

This section covers the web sites crawled by the Catalog web crawler and the metadata extraction and content processing/parsing required for each site.

2.1 Web Sites and Sub-SitesThe web sites and sub-sites to be crawled are listed below.

Root URL Web Area

Archives.Gov Web Site

http://www.archives.gov National Archives: Archives.gov

Major Areas within Archives.gov Web Site

http://www.archives.gov/about National Archives: About

http://www.archives.gov/calendar/ National Archives: Calendar

http://www.archives.gov/dc-metro/events/ National Archives: Calendar

http://www.archives.gov/contact/ National Archives: Contact

http://www.archives.gov/contracts/ National Archives: Doing Business with NARA

http://www.archives.gov/espanol/ National Archives: En Español

http://www.archives.gov/eeo/ National Archives: Equal Employment Opportunity Program

http://www.archives.gov/faqs/ National Archives: FAQs

http://www.archives.gov/fed-employees/ National Archives: Federal Employees

http://www.archives.gov/frc/ National Archives: Federal Records Centers

http://www.archives.gov/federal-register/ National Archives: Federal Register

http://www.archives.gov/forms/ National Archives: Forms

http://www.archives.gov/research/genealogy/ National Archives: Genealogy

http://www.archives.gov/grants/ National Archives: Grants

http://www.archives.gov/careers/ National Archives: Jobs, Internships & Volunteering

http://www.archives.gov/legislative/ National Archives: Legislative Branch

http://www.archives.gov/locations/ National Archives: Locations

http://www.archives.gov/congress/ National Archives: Members of Congress

http://www.archives.gov/research/military/ National Archives: Military Records

http://www.archives.gov/oig/ National Archives: Office of the Inspector General (OIG)

http://www.archives.gov/exhibits/ National Archives: Online Exhibits

http://www.archives.gov/open/ National Archives: Open Government at the National Archives

5

http://www.archives.gov/open/

http://www.archives.gov/exhibits/

http://www.archives.gov/oig/

http://www.archives.gov/research/military/

http://www.archives.gov/congress/

http://www.archives.gov/locations/

http://www.archives.gov/legislative/

http://www.archives.gov/careers/

http://www.archives.gov/grants/

http://www.archives.gov/research/genealogy/

http://www.archives.gov/forms/

http://www.archives.gov/federal-register/

http://www.archives.gov/frc/

http://www.archives.gov/fed-employees/

http://www.archives.gov/faqs/

http://www.archives.gov/eeo/

http://www.archives.gov/espanol/

http://www.archives.gov/contracts/

http://www.archives.gov/contact/

http://www.archives.gov/dc-metro/events/

http://www.archives.gov/calendar/

http://www.archives.gov/about

http://www.archives.gov/

Root URL Web Area

http://www.archives.gov/preservation/ National Archives: Preservation

http://www.archives.gov/presidential-libraries/ National Archives: Presidential Libraries

http://www.archives.gov/press/ National Archives: Press/Journalists

http://www.archives.gov/publications/prologue/ National Archives: Prologue Magazine

http://www.archives.gov/publications/ National Archives: Publications

http://www.archives.gov/records-mgmt/ National Archives: Records Management

http://www.archives.gov/research/ National Archives: Research

http://www.archives.gov/shop/ National Archives: Shop

http://www.archives.gov/education/ National Archives: Teacher’s Resources

http://www.archives.gov/veterans/ National Archives: Veterans

http://www.archives.gov/nae/ The National Archives Museum

http://www.911commission.gov National Commission on Terrorist Attacks upon the United States

http://www.9-11commission.gov National Commission on Terrorist Attacks upon the United States

http://www.fcic.gov Financial Crisis Inquiry Commission

http://www.federalregister.gov Federal Register

http://www.nara.gov National Archives

http://www.ofr.gov The Office of the Federal Register

http://www.ourdocuments.gov The Our Documents Initiative

http://www.presidentialtimeline.org The Presidential Timeline

Blogs

http://blogs.archives.gov National Archives: Blogs

http://blogs.archives.gov/nhprc National Archives Blog: Annotation / NHPRC

http://blog.archives.gov/aotus National Archives Blog: AOTUS Blog

http://blogs.archives.gov/carter-chronicle/ National Archives Blog: The Carter Chronicle

http://blogs.archives.gov/education/ National Archives Blog: Education Updates

http://blogs.archives.gov/foiablog/ National Archives Blog: FOIA Ombudsman

http://blogs.archives.gov/Innovation National Archives Blog: Inside Innovation

http://blogs.archives.gov/mediamatters/ National Archives Blog: Media Matters

http://blogs.archives.gov/online-public-access/ National Archives Blog: NARAtions

http://blogs.archives.gov/ndc/ National Archives Blog: National Declassification Center

http://blogs.archives.gov/prologue/ National Archives Blog: Prologue: Pieces of History

http://blogs.archives.gov/records-express/ National Archives Blog: Records Express

http://blogs.archives.gov/blackhistoryblog/ National Archives Blog: Rediscovering Black History Blog

6

http://blogs.archives.gov/blackhistoryblog/

http://blogs.archives.gov/records-express/

http://blogs.archives.gov/prologue/

http://blogs.archives.gov/ndc/

http://blogs.archives.gov/online-public-access/

http://blogs.archives.gov/mediamatters/

http://blogs.archives.gov/Innovation

http://blogs.archives.gov/foiablog/

http://blogs.archives.gov/education/

http://blogs.archives.gov/carter-chronicle/

http://blog.archives.gov/aotus

http://blogs.archives.gov/nhprc

http://blogs.archives.gov/

http://www.presidentialtimeline.org/

http://www.ourdocuments.gov/

http://www.ofr.gov/

http://www.nara.gov/

http://www.federalregister.gov/

http://www.fcic.gov/

http://www.9-11commission.gov/

http://www.911commission.gov/

http://www.archives.gov/nae/

http://www.archives.gov/veterans/

http://www.archives.gov/education/

http://www.archives.gov/shop/

http://www.archives.gov/research/

http://www.archives.gov/records-mgmt/

http://www.archives.gov/publications/

http://www.archives.gov/publications/prologue/

http://www.archives.gov/press/

http://www.archives.gov/presidential-libraries/

http://www.archives.gov/preservation/

Root URL Web Area

http://blogs.archives.gov/TextMessage/ National Archives Blog: The Text Message

http://blogs.archives.gov/transformingclassification/ National Archives Blog: Transforming Classification

http://blogs.archives.gov/hoover-blackboard/ National Archives Blog: The Hoover Blackboard

Presidential Libraries

http://www.hoover.archives.gov Herbert Hoover Presidential Library & Museum

http://www.fdrlibrary.marist.edu Franklin D. Roosevelt Presidential Library and Museum

http://www.trumanlibrary.org Harry S. Truman Library & Museum

http://www.eisenhower.archives.gov Dwight D. Eisenhower Library, Museum, and Boyhood Home

http://www.jfklibrary.org John F. Kennedy Presidential Library and Museum

http://www.lbjlibrary.org LBJ Presidential Library

http://www.nixonlibrary.gov Nixon Presidential Library & Museum

http://www.fordlibrarymuseum.gov Gerald R. Ford Presidential Library & Museum

http://www.jimmycarterlibrary.gov Jimmy Carter Library & Museum

http://www.reagan.utexas.edu Ronald Reagan Presidential Library & Museum

http://bush41library.tamu.edu George Bush Presidential Library and Museum

http://www.clintonlibrary.gov/ William J Clinton Presidential Library & Museum

http://www.georgewbushlibrary.smu.edu George W. Bush Presidential Library and Museum

Web Sites Not Included:

The following web sites are not included because they not a presidential library and they are not contained within the “archives.gov” web domain:

The Federal Register Blog: http://www.federalregister.gov/blog/

Our Archives Wiki: http://www.ourarchives.wikispaces.net/

2.2 Standard MetadataThe Catalog crawls the www.archives.gov website, the associated blogs and presidential libraries mentioned above, starting with the site root or specified start page. A set of metadata is extracted from each of the pages. Then the each page plus its metadata is indexed into the Catalog.

7

http://www.archives.gov/

http://www.ourarchives.wikispaces.net/

http://www.federalregister.gov/blog/

http://www.georgewbushlibrary.smu.edu/

http://www.clintonlibrary.gov/

http://bush41library.tamu.edu/

http://www.reagan.utexas.edu/

http://www.jimmycarterlibrary.gov/

http://www.fordlibrarymuseum.gov/

http://www.nixonlibrary.gov/

http://www.lbjlibrary.org/

http://www.jfklibrary.org/

http://www.eisenhower.archives.gov/

http://www.trumanlibrary.org/

http://www.fdrlibrary.marist.edu/

http://www.hoover.archives.gov/

http://blogs.archives.gov/hoover-blackboard/

http://blogs.archives.gov/transformingclassification/

http://blogs.archives.gov/TextMessage/

Note:

The “WEB/” prefix is used to identify all metadata fields which come from the Web crawler or are extracted from the HTML page. See section 1.3 above for more details.

The metadata to collect from each web page is listed below:

Field Type From Description

WEB/type string Mapping Table

“archivesWeb” or “presidentialWeb”

WEB/url string Crawler URL of the web page.

WEB/mime string Text extraction or HTML page

Either “text/html” for all web pages, or the mime type as determined by text extraction.

WEB/title string HTML Page This is the title of the webpage or the blog post. This metadata will be displayed to the user. If no title is found, the web area may be used instead, or for a blog, the blog title (title tag of blog web page).

WEB/area string URL Parsing and Mapping Table

The area of the web site to which a web page belongs. For example, with presidential libraries, this is the title of the library web site. For a blog, it is the title of the blog (AOTUS).Also, some sub-areas within archives.gov (such as “National Archives: Teacher’s Resources”) will be separately identified.

WEB/areaUrl string Mapping Table

The URL for the web area, such as the home page for the library.

WEB/date string URL or HTML Page

The date of the web page, if identifiable.

WEB/content string HTML Page Content of the web page. Note that some web pages may have a descriptive tag to identify content. If this does not exist, it will be all of the non-tag text.

Note that not every web site or web page will have all the above metadata. At a minimum the following are needed:

WEB/type WEB/title WEB/area WEB/areaUrl WEB/content

2.3 Metadata Parsing for the Archives.gov SiteThis section provides information on how the standard metadata may be obtained, followed by some examples. For the title, different tags may be appropriate if the main <title> tag is either

8

missing or not informative for a subsection of archives.gov.

Field Tags / Mapping / Parsing Instructions

WEB/type “archivesWeb”

WEB/url URL of web page (from crawler).

WEB/mime “text/html”

WEB/title The title will be extracted from the any of the following tags: <title> <meta name=”description”> <h1> URL file name

The tags are in priority order. For example, if no title can be extracted from <title>, it will be extracted from <meta name=”description”>.

WEB/area This will come from the “Web Area” column of the largest URL from the table in section 2.1 above which matches the prefix of the WEB/url field from the crawler. For example, the URL “http://www.archives.gov/oig/investigations.html” matches

“http://www.archives.gov/oig/” from the table in section 2.1. Therefore, the “WEB/area” field will be set to:

“National Archives: Office of the Inspector General (OIG)”.WEB/areaUrl This will come from the “Web Area” column of the largest URL from the table in section

2.1 above which matches the prefix of the WEB/url field from the crawler.

WEB/date From the <meta name=”date”> tag. Parsed as YYYY-MM-DD.

WEB/content If both  and  exist: Index all text content between the two tags

If only  exists: Index all text content from startindex to the end of the web page

Otherwise Index all text content on the web page

Notes:1. Only index text content. Do not index HTML tags, HTML attributes, or HTML

attribute values.2. Ignore all content found between <script> and </script>3. Do not index the content of 

See below for an example.

Example of startindex & stopindex:

This example is from the top level landing page at http://www.archives.gov/education.

<head> <title>Teachers' Resources</title> <meta name="description" content="Resources for Teachers" /> <meta name="date" content="2013-08-21" /> ...<body><div id="container"> <div id=”header”>..

9

http://www.archives.gov/education

.<div id="container2"><div id="content">

.

. Content to index occurs in here

.

<div class="menu connect"> ...</body></html>

2.4 Metadata Parsing for NARA BlogsThe majority of the NARA blogs have a consistent format. The exception is the AOTUS blog. The following table shows how to obtain the standard metadata.


WEB/type “archivesWeb”



WEB/title For all blogs, the <title> tag will supply the title. However, all blog titles contain both the title of the blog as well as the title of the article. For example:

AOTUS: Collector in Chief | Calling All Walt Whitman FansTo improve accuracy, we will remove the title of the blog from the above, leaving only “Calling All Walt Whitman Fans”. The title of the blog itself will be captured in the WEB/area field below. For the AOTUS blog, extract the blog article title from the text after the “|” pipe

character. For all other blogs, extract the blog article title from the text after the “»”

character.If the delimiter specified above does not occur in the title, take the entire <title> content as the title.

WEB/area This will come from the “Web Area” column of the largest URL from the table in section 2.1 above which matches the prefix of the WEB/url field from the crawler.

WEB/areaUrl This will come from the “Web Area” column of the largest URL from the table in section 2.1 above which matches the prefix of the WEB/url field from the crawler.

WEB/date Parse the date from the text of the blog: AOTUS: <p class="meta">Written on February 14, 2014 | Other blogs: <span class="gray">on February 3, 2014</span>

See examples.

WEB/content There appears to be no standard for marking the content area of blogs. Therefore, the Use the entire text content of the HTML page will be indexed as the content.Notes:

1. Only index text content. Do not index HTML tags, HTML attributes, or HTML

10


2.4.1 Parsing Date for AOTUS Blog

Dates for this blog can be identified with <p class="meta"> as shown below.

<h2><a href="..">Happy Valentine’s Day</a></h2><p class="meta">Written on February 14, 2014 | <a href="…"><p><a href="..."><p>This 1918 valentine refers to the World War I effort…</div>-->

2.4.2 Metadata Parsing for other National Archives Blogs

Dates for other blogs occur after the <span class="gray"> tag.

<div class="post"><div><img alt="..."><h2><a href="... title=”National Declassification Center Completes Quality Assurance of Backlog Final">National Declassification Center Completes Quality Assurance of Backlog Final</a></h2><div class="small">by <a href="...">Nancy Soderberg</a> <span class="gray">on February 3, 2014</span></div> </div></div></br /><br />

2.5 Presidential LibrariesThe page format of these libraries is unfortunately not consistent. The following strategy provides a set of tags to check to obtain the metadata.

In some cases, the library web pages’ title tag is just the official name of the library and is the same for the various pages. In this case, if possible, a different tag should be used to obtain the title of the specific page.


WEB/type “presidentialWeb”



WEB/title The title will be extracted from the any of the following tags: <title> <h1> <h2> <meta name=”description”> <meta name=”keywords”> URL file name

The tags are in priority order. For example, if no title can be extracted from <title>, it will be extracted from <meta name=”description”>.

11

WEB/area This will come from the “Web Area” column of the largest URL from the table in section 2.1 above which matches the prefix of the WEB/url field from the crawler. For example, the URL “http://www.trumanlibrary.org/whistlestop/wedding.htm”

matches “http://www.trumanlibrary.org/” from the table in section 2.1. Therefore, the “WEB/area” field will be set to:

“Harry S. Truman Library & Museum”.WEB/areaUrl This will come from the “Web Area” column of the largest URL from the table in section

2.1 above which matches the prefix of the WEB/url field from the crawler.

WEB/date From the <meta name="date"> tag. Parsed as YYYY-MM-DD.If the tag does not exist, then leave the date blank.

WEB/content If both  and  exist: Index all text content between the two tags

If only  exists: Index all text content from startindex to the end of the web page

Otherwise Index all text content on the web page

Notes:1. Most presidential libraries do not use this convention.2. Only index text content. Do not index HTML tags, HTML attributes, or HTML


2.6 Processing Non-HTML contentAll non-HTML files will be processed as follows:

All media files (video, images, audio) – Skipped

Document files (PDF, MS-Word, etc.) – Will be processed through text extraction

o See below

2.6.1 Text Extraction

All document files (PDF, MS-Word, etc.) will be run through text extraction to extract metadata and text content.

The tool for text extraction (depending on the results of the Analysis of Alternatives, currently in progress) will likely be Apache Tika, see https://tika.apache.org/ for more information.

12

https://tika.apache.org/

2.6.2 Metadata Mapping for Non-HTML Content

The output of Apache Tika will be mapped to the WEB/ fields as follows:


WEB/type If the URL contains “archives.gov” this will be “archivesWeb”.Otherwise it will be “presidentialWeb”.


WEB/mime From <meta name="Content-Type" content=". . ."/>

WEB/title The title will be extracted from the any of the following tags: <title> <h1> File name

The tags are in priority order. For example, if no title can be extracted from <title>, it will be extracted from <h1>

WEB/area This will come from the “Web Area” column of the largest URL from the table in section 2.1 above which matches the prefix of the WEB/url field from the crawler.

WEB/areaUrl This will come from the “Web Area” column of the largest URL from the table in section 2.1 above which matches the prefix of the WEB/url field from the crawler.

WEB/date The date will come from any of the following tags: <meta name="Last-Modified" content=". . ."/> <meta name="date" content=". . ."/> <meta name="Creation-Date" content=". . ."/>

The tags are listed in priority order. The date will be found in the @content attribute. The date will be in ISO 8601 format.If no date exists, then leave the date blank.

WEB/content All text content produced by Apache Tika (minus XHTML tags and XML comments) will be indexed as the text content.

13

3 File Processing

Beyond the metadata parsing described in section 2, there is no file processing required for the Web Sites data source.

This section is retained to ensure that major section numbers are consistent across all Data Model Design (DMD) documents.

14

4 Mapping to Index Fields

The following table maps the metadata to the appropriate index field. The metadata elements are not given here because in some cases they differ based on the particular web site they originate from. See the mappings of document tags to metadata given in section 1.3.

4.1 Field Metadata Mappings

Index Field WEB Metadata Name PurposeI/url WEB/url results

I/source “web” search

I/type WEB/type search

I/oldScope If WEB/type == “archivesWeb”: set to “archives.gov”

If WEB/type == “presidentialWeb”: set to “presidential”

search, facet

I/iconType WEB/mime (normalized) results

I/fileFormat WEB/mime (normalized) results

I/originalMimeType WEB/mime results

I/tabType “all,online,web” results

I/materialsType “web” results

I/title WEB/title results, search

I/titleSort WEB/title with articles and prepositions from the start removed.

sorting

I/allTitles WEB/title results, search

I/webArea WEB/area results

I/webAreaUrl WEB/areaUrl results

I/content WEB/content results, search, teasers

I/teaser The first 500 characters of WEB/content results

I/titleDate WEB/date results

I/dateRangeFacet Choose the appropriate date range facet which covers the WEB/date value. Leave blank if WEB/date is blank.

facets

I/productionDate WEB/date results, sorting

I/productionDateQualifier “YMD” results

Refining by location should only return archival records, not web page results.

15

4.2 Mapping to Keywords Relevancy ModelWeb page metadata will be mapped to the Catalog relevancy model as follows. Note that all of the fields specified (grank1, grank2, grank3, and content) will be searched by all “q=” parameters.

When multiple WEB/ fields are mapped to the same relevancy field, all of their content will be concatenated together into the same field and searched together.

WEB/ field Relevancy Field

WEB/title I/grank2

WEB/url I/grank3

WEB/webArea I/content

WEB/content I/content

16

5 Search Fields

There are no special fields for advanced search over web sites.

17

6 Search Results Presentation

This section details the search results presentation for web site results.

For simple keyword searches, the top three web sites matches will be returned. They are returned as a group in the search results list, under “National Archives website pages”.

6.1 Search Results (aka “Brief Results” Display)As shown in this example, only the top 3 website results are shown as grouped under the heading “National Archives website pages”. The globe icon is used.

This table describes the 3 lines that make up each web site within the group.

Lin

e

SoLR Fields & Pattern

1 {I/title}

LINK: I/url

2 {I/webArea}

LINK: I/webAreaUrl

3 {Highlighted teaser from the search engine, or I/teaser }

[Note: Display first 200 characters as snippet in the result]

Notes:

1. “National Archives website pages” links to a tab with all the web sites search results.

2. The globe icon is a link to the same tab.

18

7 Content Details Presentation

The links in the search results go directly to the external website and therefore there is no content detail page for Archives.gov within the Catalog.

19