incontrol™ operations center guide - amazon s3 · •setup is used to setup your case and...

CONFIDENTIAL

InControl™ Operations Center Guide For System Administrators

Version 7.0

InControl™ 7.0 Operations Center Guide

Confidential

2

Contents Introduction ............................................................................................................................................ 6

Operations Center Control Panels ........................................................................................................ 6

Ops Center Navigation ..................................................................................................................... 6

SETUP ...................................................................................................................................................... 7

Creating a New Case ............................................................................................................................ 7

Creating a New Company .................................................................................................................... 7

Creating a Client/Firm .......................................................................................................................... 8

Entering Case Information ................................................................................................................... 9

Templates.......................................................................................................................................... 10

Setting Up Fields ................................................................................................................................ 11

Introduction .................................................................................................................................. 11

Custom Fields ................................................................................................................................ 13

Field Mapping .................................................................................................................................... 17

Profiles .......................................................................................................................................... 17

Mapping ........................................................................................................................................ 19

Load File ....................................................................................................................................... 19

Field Mapping ................................................................................................................................ 20

Preview ......................................................................................................................................... 25

Users ..................................................................................................................................................... 27

Users by Case .................................................................................................................................... 27

Adding Users ................................................................................................................................. 28

Deleting Users ............................................................................................................................... 29

Change Password........................................................................................................................... 30

Reset Password ............................................................................................................................. 30

Enable\Disable User....................................................................................................................... 30

Groups ........................................................................................................................................... 31

Cases by User .................................................................................................................................... 32


Confidential

3

DATA ..................................................................................................................................................... 32

Importing Data .................................................................................................................................. 33

Language Options .......................................................................................................................... 34

Test Mode ..................................................................................................................................... 34

Build Email Threads ....................................................................................................................... 34

Build Document Folders ................................................................................................................. 34

Loading Errors................................................................................................................................ 36

Pop-Up Loading Errors ................................................................................................................... 42

Change Document Path ................................................................................................................. 45

Updating Data ............................................................................................................................... 46

Import Tags ....................................................................................................................................... 47

Indexing Data .................................................................................................................................... 49

Index Data – ISYS ........................................................................................................................... 49

Indexing Options ............................................................................................................................ 53

Perceptive Index Logs .................................................................................................................... 58

ISYS Indexing Max Limits ................................................................................................................ 61

Indexing Data – Elastic Search ............................................................................................................ 62

Indexing Overview ......................................................................................................................... 62

Documents Not Indexed ................................................................................................................ 62

Documents Not Searchable ........................................................................................................... 63

Image Extensions ........................................................................................................................... 64

Preparing for Indexing ................................................................................................................... 64

Indexing Process ............................................................................................................................ 65

Indexing Options ............................................................................................................................ 66

Index Logs...................................................................................................................................... 72

Bulk Update ....................................................................................................................................... 76

Create Alerts ..................................................................................................................................... 77


Confidential

4

Indexing Transcripts........................................................................................................................... 77

Images................................................................................................................................................... 79

Bulk Image ......................................................................................................................................... 79

Loading Images .................................................................................................................................. 80

Images Overview ........................................................................................................................... 80

Image Sets ..................................................................................................................................... 80

Image Load Files ............................................................................................................................ 81

Use Load File Path ......................................................................................................................... 81

Overwrite Existing Images .............................................................................................................. 81

Keep Images In-Place ..................................................................................................................... 82

Analytics ................................................................................................................................................ 83

Email Threads .................................................................................................................................... 83

Entities .............................................................................................................................................. 84

Entities - ISYS ................................................................................................................................. 84

Entities – Elastic ............................................................................................................................. 85

Near Duplicates & Clusters ................................................................................................................ 86

UTILITIES ............................................................................................................................................... 88

Querying Data ................................................................................................................................... 88

Querying Hits ..................................................................................................................................... 89

Manage Lists ..................................................................................................................................... 90

Near Dupe Export .............................................................................................................................. 93

Build Document Folders..................................................................................................................... 95

Rebuild Document Data ..................................................................................................................... 97

Reports .............................................................................................................................................. 97

Support Functions ................................................................................................................................. 99

System Settings ................................................................................................................................. 99

Working with Application Settings ..................................................................................................... 99

Password Rules ................................................................................................................................ 100

DB Updater ...................................................................................................................................... 101

Index Check - ISYS ............................................................................................................................ 102


Confidential

5

Email Users ...................................................................................................................................... 102

Email Error Logs ............................................................................................................................... 103

Email Logs .................................................................................................................................... 103

Entering License Information ............................................................................................................... 104


Confidential

6

Introduction The InControl ™ Operations Center is used to create InControl databases, load data, and other actions related to InControl database management.

Operations Center Control Panels When you first open the InControl™ Operations Center software, you will have four panes on the left: Setup, Data, Utilities, and Support Functions.

• Setup is used to setup your case and database. • Data is used to load, update, and index data. • Images is used to load and create images. • Analytics is used to build email threads, entities, near duplicates, and clusters. • Utilities is used to query the data, create hit reports, manage lists, build document folders,

export near duplicates, and view reports. • Support Functions is used to edit system settings, change password rules, use the database

updater, and email Reveal support your system logs.

Ops Center Navigation As you use the Operations Center, areas that you open and select on the left will be highlighted in orange in the control panel and added as tabs on the right. Once opened, the areas will stay open and as tabs on the right. To go back to an already opened area you may click on the menus left or tabs.


Confidential

7

SETUP Setup is used to create your database, map your fields, and add users. The first step in Setup is to add your case related information and create your case.

Creating a New Case

Below are the steps to setting up a new case:

1. Create the Company. If you’ve already created the company for a previous Case you can skip. 2. Create the Client/Firm. If you’ve already created the Client\Firm for a previous Case you can

skip. 3. Go to Case create the Case. This will insert the case details and build the database. 4. Make any Field changes or additions. If preferred you may use an existing template to do this. 5. Map fields to a load file to complete database setup and prepare for data import. 6. At any point after database creation you can add users

Creating a New Company


Confidential

8

The “Company” is the end client. If you are working for a law firm, the “Company” is usually your client’s client and usually a corporation. It you are working at a law firm or for a corporation this is usually your client’s name or perhaps the parent company. This field is purely informational, but required to setup a database.

After selecting the Company link on the left, the Companies tab appears in the work area.

All existing Companies are listed in the top window. The Edit/ Delete tab is the default tab. To enter a new Company, switch the tab to Add New, enter the information and then press the “Insert” button. The only required field of information is the company name.

Creating a Client/Firm


Confidential

9

The Client/Firm is the name of your client and their company/firm information.

Existing Clients are listed in the top window, and the “Edit/Delete” tab is the default tab. To create a new Client, select “Add New”.

The required fields have a red * asterisk. They are listed below with a brief description.

• Client Name – the name of the client. • Client Number – a unique number or name that you assigned to this client. All data is organized

by client number, so it is important this is unique.

Once you add these, click ‘Insert’ to add the new client.

Entering Case Information


Confidential

10

Existing Cases are listed in the top window, and the “Edit/Delete” tab is the default tab.

To create a new Case, select “Add New.” This will add the case and create the case database.

There are a number of required fields which have a red * asterisk. They are listed below with a brief description.

• Case Name – the name of the case/project which will display in the Operations Center and InControl for the end user.

• Project ID – the project number which will be used to organize the data and identify the project. • Company Name – is your connection to the company information already created. • Client Name – is your connection to the client information already created. Your database will

be named by client and project ID. • If a case is a template case you would like to copy objects from later, click ‘Case is Template’ • DB Template – A user can select either a standard InControl database, or if they are using a

direct connection option with another ingestion tool, then they can switch to that template if available.

• Search Engine – select either Elastic or Perceptive for the Search and Index options.

Templates


Confidential

11

If desired you can copy objects from a template to save time.

To use Templates:

1. Select the case you want to copy from (You will only see template cases list). 2. Select the case you want to copy to 3. Select the objects you would like to copy 4. Click Copy 5. Review the Log for Errors

Setting Up Fields

Introduction InControl has a pre-built set of fields. Additional information on these fields and their data requirements can be found in the InControl Data Requirements Guide and InControl Fields List. We highly recommend you review this guide and list prior to using the Fields are in the Operations Center.


Confidential

12

Once you area familiar with InControl’s requirements and have your first load file, go to Operation Center’s Fields to prepare your database for data loading from a load file.

In Fields you will be able to create new and edit existing fields

Fields has the following columns commonly used during Field setup:

• Table Name – The SQL column name in the table storing the fielded information • Display Name – The default display name you would like users to see when using this field • Maximum Length – The amount of data the field can currently accommodate • Data Type – The type of data the field will hold • Description – A description of the field • MultiValue – Whether or not the field is multivalue


Confidential

13

Custom Fields You may find that there are fields you need which are not present or that an existing field may need to be modified in order to accommodate your needs.

Creation & Editing is often step number 5 in your workflow below:

1. Select the case name from the drop down list 2. Select your import profile (delimiters)

a. Select an existing profile from the drop-down b. Setup a new profile by clicking Setup

3. Select a load file 4. Setup your Import criteria for each field (field mapping) 5. Create\Edit Fields

Field Creation New/Custom fields can be created using the “New Field” button.

This will launch the new field window.


Confidential

14

Rules for new/custom fields:

• Field Name - The SQL column name. Cannot include spaces.

• Field Display Name - The name displayed to the user throughout InControl when they are looking at the fields. The Display Name can be modified here and also managed per field profile in the web tool Admin\Fields\Manage fields.

• Field Data Type – The InControl field types are:

o Boolean - True/False or Yes/No values

o Date - Multiple date formats supported (Example: DD/MM/YYYY)

o Date\Time - Date and Time combined (Example: DD/MM/YYYY HH:MM:SS)

o Float - Number with decimal point and unlimited precision after decimal

o Integer - Whole numbers, no decimal

o Numeric - Number with decimal point to 17 digits

o Text - Alpha numeric values

o Time - Time (Example: HH:MM:SS)

• Max length - is the length of the field. Any data beyond a field’s max length will get truncated during data loading.

• Is Searchable – indicates whether the field should be indexed for searching

• Is Updateable - indicates whether the field should be updateable by default.

• Is Document Number Field – indicates if field shall be used as a key field when loading images or updating data during Production Update.

• Is MultiValue Field – indicates that the field may contain multiple values. This setting allows the field to be parsed.

• Tally Enabled – Indicates whether a field can be used for the Summary function on the Review Folder View.

• Is Transcript Lookup - designates a field a transcript reference field and will allow the field to be one of the selections in the Document Link Lookup By dropdown menu. The user will be able to query this field and retrieve documents using a unique document ID.


Confidential

15

Field Editing Once created, a field can be edited by selecting the field you would like to edit and clicking Edit Field.

This will launch the edit field window:


Confidential

16

The following attributes can be modified:

• Field Display Name - The name displayed to the user throughout InControl when they are looking at the fields. The Display Name can be modified here and also managed per field profile in the web tool Admin\Fields\Manage fields.

• Max length - is the length of the field. Any data beyond a field’s max length will get truncated during data loading.

• Is Searchable – indicates whether the field should be indexed for searching.

• Is Updateable - indicates whether the field should be updateable by default.

• Is Document Number Field – indicates if field shall be used as a key field when loading images or updating data during Production Update.

• Is MultiValue Field – indicates that the field may contain multiple values. This setting allows the field to be parsed.


Confidential

17

Field Mapping

Items to know before you begin:

• A load file is required to create database records. InControl does not offer ‘New Record’ capability through the web tool.

• The Fields area in the Operations Center is used to prepare the database for data loading. • In Field Mapping you will setup your fields, map them to your load file, and save your mapping.

Only once these steps are taken will you then be able to effectively load your first set of data. • It is important to open Fields, confirm or setup your field mapping, and preview before trying to

load subsequent data. Previewing in advance will help ensure your fields are mapped correctly and your loading will be successful.

In Field Mapping you will do the following steps:


a. Select an existing profile from the drop-down OR Setup a new profile by clicking Setup 3. Select an existing field mapping or create a new one 4. Select a load file 5. Setup or modify your Import criteria for each field (field mapping) 6. Save the Mapping 7. Save the Mapping as a Database Template to be used in other databases.

The documentation below covers each step.

Profiles Profiles are used to save your delimiter settings for later use. The default profile is the ‘Concordance’ profile. It utilizes Concordance delimiters.

To Setup your profile:


Confidential

18


a. Select an existing profile from the drop-down b. OR Setup a new profile by clicking Setup

Delimiters Should you need a new profile you will need to specify your profile name and delimiters.

Quick Intro to Delimiters Supported database load files contain single document records on each line and field and text delimiters which tell the database program where a new field of information begins and ends. This enables the software to load the information into columns for searching and viewing. InControl has default delimiters which match a common industry standard (Concordance), but any unique delimiters which can be typed or copy/pasted by the end user will work with InControl.

Concordance Example:


Confidential

19

The InControl delimiters are: • Text Qualifier - This delimiter is used to indicate the beginning and end of text.

o Common alternatives are: ^ and “ • Field Delimiter - This delimiter is used to separate fielded data.

o Common alternatives: | and , • Multi -value Delimiter - This delimiter is used to indicate the separator between different values

in the same field. o Common example is: ; (semi-colon).

• Nested Value - This delimiter is used to indicate the separator between nested values in a single or multi-value field.

o Common alternative is: \ o Note: By default InControl’s Document Folders feature requires that this delimiter is a

back slash (\). There is a system setting which allows this to be changed by system or project. There are currently no features which use the nested delimiter at field data import. This is only used by the tag loader.

Mapping Once you map your fields you can save your mapping and select it here. If you’ve not already saved a mapping or need a new mapping, there is no need to change this in the form. When set to default ‘Select’ this will create a new mapping.

Load File Selecting your load file is the fourth step in Field setup. Load file selection is necessary before you map fields.



3. Select Mapping if already saved 4. Select a load file


Confidential

20

Once you select the file, you can begin Field Mapping.

Field Mapping The field mapping process connects the fields in the load file you’ve selected to the fields in the InControl database. Load file selection is done either automatically or manually in the ‘Import Field Name’ column in Field Mapping. In addition to mapping fields and loading data from a load file, this column allows you to generate field data using other fields (auto generation) or typed in values (manually entered).

To map fields:



Confidential

21


3. Select a load file 4. Setup your Import criteria for each field in Load File Field column 5. Save your mapping

The documentation below includes information on setting up your import criteria. There are two ways to do this: Automatically and/or Manually.

Automatic Matching To automatically match fields based on Table Name and field header click Match All Fields:

Manual Matching Instead of using automatic matching or as a supplement to automatic matching you may need to manually inform the software how to populate data for each field. While you do this you have three options: Match to Load File (Field Map), Manually Enter, or Auto-Generate. Your options are set in the ‘Import Field Name’ column in Fields.

Import Field Name Options The Import Field Name column in InControl is used to tell the software how to populate the fielded data during data loading.


Confidential

22

This process is called Field Mapping and requires a load file. However, field mapping is not your only option for every field. Some fields allow you to manually enter or auto-generate the information at load time.

Mapping Options Overview:

• Load from a load file o This requires mapping the field to a field in your load file. o The value must be valid and meet the requirements of the field.


Confidential

23

• Manually entered

o Upon data loading, this will bulk update the field with the value you enter.

• Auto-generate o This method uses another field of data to generate the value. o The value you are using to generate must be valid and meet the requirements for the

field.

More in-depth information on Manually Entered and Auto-generate can be found below.

Import Field Name Options Detail:

Load File Mapping Once you’ve responded to the auto-mapping message, either with a yes or a no, you will need to QC the mapping and, in most cases, make some manual changes. Manual mapping is done in the ‘Load File Field’ column. This drop-down you encounter when clicking each field will hold a list of fields in your load file. The dropdown is filtered by the Hide Fields that are not imported checkbox at the top of the screen.


Confidential

24

There are certain fields required for loading your data and it is important that you ensure that these are mapped or addressed, and that any fields you want to load are mapped or added as new fields.

Auto-Generation Auto-generation is an option if you do not have ITEMID and/or PARENTID fields or data for these fields which does not meet InControl’s data requirements. This will utilize other fields to auto-generate the ITEMID/PARENTID information.

You can auto-generate just ITEMID if you have no parent family relationships. If you do have family relationships, you must auto-generate both ITEMID and PARENT_ITEMID.

1. In your field mapping, choose auto-generate from the drop-down instead of selecting a field.

This looks like this:

2. Later, when previewing or loading data, specify the field you want to use to auto-generate. An example would be:


Confidential

25

If you are auto-generating an ITEMID, you will need to ensure that the field you use to auto-generate is populated for all records and is not zero or empty.

If you are auto-generating a PARENTID, you will need to ensure that the field you use to auto-generate references the data in the field you are using to auto-generate ITEMID.

Manually Entered Manually entering is an option if you don’t have data for a required field such as BATCHID or CUSTODIAN_NAME. Both BATCHID and CUSTODIAN_NAME require that data exist for each and every record. The option ‘Manually Entered’ allows you to bulk update the field at load time with a single value, such as the load file name (example ENE001) for BATCHID or None for CUSTODIAN_NAME. t field attributes with a load file selected.

Preview Using preview can assist you in quality checking your field mapping.

Your steps are as follows:



3. Select a load file 4. Setup your Import criteria for each field (field mapping) 5. Save Mapping 6. Preview

To preview, click the Preview button on the right.


Confidential

26

Next, you will be asked to save your settings and input any information required to generate your preview. This will include any items you are auto-generating or manually entering.

Once you’ve input the necessary information, click OK. Then your preview window will launch.


Confidential

27

Users

There are two ways to modify users.

• Users by Case o Provides a Case dropdown and a table by user o Allows you to:

Add new or existing user(s) to a case Delete user(s) from a case Change a user’s password Reset a user’s password Enable\Disable user accounts Change a user’s group

• Cases by User

o Provides a User dropdown and a table by case o Allows you to:

Delete user from case Change a user’s group

Users by Case ‘Users by Case’ enables you to filter all users by case by using the ‘Select Case’ dropdown. Next you can modify by adding additional users, deleting users, changing passwords, resetting passwords, disabling users, or changing user groups.


Confidential

28

Adding Users To add a new user to the case, click the Add button lower right.


Confidential

29

This will launch the user list, which allows you to add a current or new user. To add an existing user, filter as needed, and select them from the list. To add a new user, click the ‘Create New’ button and complete the user detail.

Deleting Users Clicking the X in the user’s row in the Delete column will remove a user from the selected case.


Confidential

30

It will not delete the user’s account, only remove them from the case.

Change Password To change a user’s password, click the ‘Change’ button in the ‘Change PW’ column for their row. For password help, click the help (?) button.

Reset Password To reset a user’s password, click the ‘Reset’ button in the ‘Reset’ column for their row. This will force them to login using their current password and then create a new password.

To both change their password to a temporary password and then require reset to a new password, click change password (per above) and change their password. Next, click Reset.

Enable\Disable User To enable or disable a user’s account, click the Enabled checkbox. A check indicates the account is Enabled. Empty indicates the account is Disabled.


Confidential

31

Groups To add a user to a group, click on the Groups column dropdown and select the group from the list. To add new groups, use the Web Tool’s Admin\Users area.

For more information see the InControl V7.0 Admin Guide section 2 on Users.


Confidential

32

Cases by User ‘Cases by User’ allows you to see a user’s cases by using the ‘Select User’ dropdown. After selecting a user, you can remove them from a case (delete) or change their user group (change).

DATA The Data section is where you will import data and tags, index your documents for HTML view and searching in either Perceptive or Elastic Search, Bulk Update fields, Create Alerts, and Manage Transcripts.


Confidential

33

Importing Data

To import data, perform the following steps:

1. Setup the fields in the “Fields” section. 2. Create, save, and preview your field mapping in Field Mapping 3. Go to Data-Import Data. 4. Select ‘Import Data’ radio button. 5. Select Import Profile 6. Select previously saved Mapping. 7. Browse out to the “Import Data File” to find your load file. 8. Select your language. This will ensure proper handling of your date data. 9. If desired, click ‘Test Mode Only.’ This will perform a test load and help quality check for

problems in your load file. Reveal recommends performing a test load first. 10. Build email threads will be checked by default. If you do not want to build threads, uncheck. 11. Build document folders will be unchecked by default. If you want to build folders, check.


Confidential

34

12. Click Import Data. With test mode on this will perform a test load. With test mode off this will perform an actual load.

13. Review your error logs. If you performed a test load, resolve any issues in your load file and then perform the test load again. If you performed an actual load your next step will be determined by the errors received.

Language Options Your language options inform the software on your date format.

Your options are:

• English (US) – Short Date Example: MM\DD\YYYY • English (United Kingdom) - Short Date Example: DD\MM\YYYY • Russian – Short Date Example: DD\MM\YYYY

Test Mode Test mode will test your data load by copying the data into a temporary table and providing any errors you would receive at load time without actually loading the data. When this is checked, the load will be a test. When unchecked, the load will actually update the database.

Build Email Threads Select build email threads to automatically build the threads once the data has been loaded. This will use whichever of the following fields are available. For the most complete threads, load in all of these fields.

Email Threading MESSAGE_ID SENT_DATE SENT_TIME THREAD_IN_REPLY_TO THREAD_REFERENCES SUBJECT_OTHER

Build Document Folders Administrators can build document folders at import by selecting the ‘Build Document Folders’ checkbox in the Import Documents screen.


Confidential

35

Build document folders will build document folders under the ‘Documents’ and ‘My Views’ based on the field and delimiters specified in the System Settings: ‘DocumentFoldersDatabaseField’, ‘DocumentFoldersMultiValueDelimiter,’ and ‘DocumentFoldersDelimiter’.

The settings are found in System Settings\Operations Center and work as follows:

• Document Folders Database Field o This contains the folder path information used to build the folders.

This can be a single value or multi-value field. Multiple values must be delimited with the Document Folders Multi Value delimiter mentioned in more detail below.

Any subfolders must be delimited with the Document Folders Delimiter mentioned in more detail below.

The default is the SOURCE field, but this can be changed system wide or by project.

• Document Folders Multi Value Delimiter o This is used to parse multiple values (paths) in the Document Folders Database Field. o The default is ; (semi-colon), but this can be changed system wide or by project.

• Document Folders Delimiter o This is used to parse the folders in a value o The default is \ (back-slash), but this can be changed system wide or by project.


Confidential

36

Notes:

• See the Support Functions\Systems Setting section of this guide for more information. • If you do not want to create this at data load, you may also use the Build Document Folders

Utility.

Loading Errors When you import data either in Test Mode or Regular, you will be notified progress and any errors in the Status area.

The types of errors are:

• Load File Errors o Data Conversion Errors – Load file data type does not match field o Truncation – Data being loaded is too large for field o Duplicate Records in Load File – There are records in load file with duplicate ITEMIDSs

and\or BEGDOCs. • Duplicate Records in Database – Once the data is loaded to the temporary table it is then

compared to the database to ensure that.

The “View Log File” and “View Data Errors” buttons can be selected after loading either in test or normal mode to insure everything loaded properly and to help support when troubleshooting issues.

View Error Log File Your error log file will provide a summary and additional detail on any errors. To view, click ‘View Log File.’


Confidential

37

There are two types of errors: Load File Errors and Duplicate Records in Database errors.

Load File Errors will provide the ERROR TYPE, DATABASE COLUMN (FIELD), LOAD FILE FIELD, and DETAILS.

Duplicate Records in Database errors will provide the error item’s ITEMID, BEGDOC, and NATIVE FILE location.


Confidential

38

View Data Errors

For any load file errors, your view data errors log file will provide the problems lines from your load file with load file header.

Error Log Examples

Example 1 In the example below, there are a number of problems with the load file mapping.

• The ITEMID and PARENTID are mixed up. • The CREATION DATE and TIME are mixed up. • The BEGDOC and EXPORTED_NATIVE_FILE location are mixed up.

In Preview, there were no errors flagged as all required fields were mapped to fields with existing data.


Confidential

39

However, at load time, the following results are seen while using ‘Test Mode’:

The error log file mentions Truncation and Duplicate key errors.

Data Truncation

In the example, data was truncated when the Native File Path information of 100+ characters was accidentally loaded into BEGDOC field with max length of 100 characters.

This issue is presented in the Error Log with the starting phrase ‘Data truncated for [InControl Field] [truncated data in its entirety]….


Confidential

40

…field [InControl Field] from [data size] to [max field size], data line [in load file (excludes header)], error file [in View Data Errors (includes header)]

If we go back to preview and view the BEGDOC (Begin Number) field we see the wrong data was mapped.

If we look at the field detail we see that the BEGDOC field has a max length of 100. If we check the data we see the path is 139 characters long. This explains the truncation. Our solution here is to go in and change the Field to BEGDOC instead of EXPORTED_NATIVE_FILE in our field mapping.

If this were a simple truncation, another possible solution would be to change the Maximum length to something which will accommodate the data size.

Duplicate keys

This example also has Duplicate Key Values in ITEMID because the user accidentally loaded Parent ID (attachment information) into that field. This results in 4130 load file duplicates as Parent ID is not unique and InControl requires a unique identifier in ITEMID.

When there are duplicate key values, the software will reference their line in the load file (excluding header). In this instance, errors ‘Duplicate key value 74933 for column ITEMID on line 2 and ITEMID on line 3’ in the error above, reference lines 3 and 4 in the load file and preview below.


Confidential

41

View of Original Log File

Preview View

View Errors There is nothing in View Errors for these items as they are skipped at load time and only the first item (original) is loaded.

Example 2 In this example, we have tried to load a duplicate load file. InControl requires unique BEGDOCs. Loading duplicates will result in an error and InControl will not load the documents. If we have duplicate records in the database, InControl will notify on the Status screen.


Confidential

42

Error Log

Pop-Up Loading Errors In addition to the error log errors, you may encounter pop-up errors while loading data which inform when there is a data issue.

No row at position 1

Error Message:

‘An Error Occurred. The error description is: There is no row at position 1.


Confidential

43

Cause: Unable to auto-generate Parent_ItemID, disconnect between fields used to generate the itemid and parent_itemid values. Example: Your BegDoc and BegAttach are different and you are using these to auto-generate: BEGDOC - ENE000001, BEGATTACH – ENRON000001

Fix:

Modify load file data so that the BEGDOC and BEGATTACH are the same.

String Conversion Error

Error Message:

‘An error occurred: The given value of type String from the data source cannot be converted to type nchar of the specified target column’. Cause: The field type and the data are incompatible. Example:


Confidential

44

Error converting a TIME column. Format should be HH:MM am/pm (e.g., 09:32 am; Timezone indicators cannot be included). This is the record that errors: þ2:28:00 (GMT) PMþ

Fix:

For the example above, you would remove the time zone.

Header Synch Issue

Error:

‘Data error for column [Field Header] on line 1, data error file line 2 Error’. Cause: This usually shows up with DAT file header not being in synch with data. Example:

‘Data error for column CC_ADDRESSES on line 1, data error file line 2 Error. Fix: This error had a DAT with BCC header but no data for BCC in the data.


Confidential

45

Change Document Path If immediately after loading you have your load file still selected in the data load section you will be able to click Document Path preview to preview the EXPORTED_NATIVE_FILE data mapping to see what was placed into that field.

If you find an issue, to find\replace the data in the ‘EXPORTED_NATIVE_FILE’ field you may use the ‘Rename the Import Document Path’ feature.

Once the data has been loaded the document path will allow you to replace sections of the native file path.

For example, if your load file loaded the native file path for all the records as X:\NATIVE\0001\ABC001.doc and this is not an InControl accessible and\or actual location of your data you will need to replace it. In this example, replace X:\ with UNC and volume path.

Follow the same steps to update the OCR Path if desired.

Note: Change Document Path has a Preview which if clicked will show you a preview of the mapped ‘EXPORTED_NATIVE_FILE’ data in load file you’ve selected. Please note, this only works if load file is selected and is not displaying the loaded data.

Warning! Change Document Path will globally replace ALL data in the EXPORTED_NATIVE_FILE field, not just the data loaded and previewed. It is very important that you choose text which you would want replaced across all the data or which will only hit on the newly loaded file.


Confidential

46

Updating Data

Updating, also referred to as overlaying, metadata for specific documents is done with the Update Data button on the Import Data screen. The user will select an Import File that will have the BegDoc or ItemId numbers that are to be updated. For example:

^Begdoc^|Âuthor^ ÊNE0000001^|^Peter Brown^ ÊNE0000002^|^Peter Brown^


Confidential

47

ÊNE0000005^|^Nancy White^ ÊNE0000501^|^Nancy White^ ÊNE0000508^|Êd Green^ ÊNE0000514^|^Mary Black^

When the user clicks on the Update Data button they will see a Fields Screen where they will select the metadata columns to be updated. Once ‘OK’ is selected the software will immediately update the database with your selections.

Selecting the Fields to update and clicking the OK button will update the records in the case database.

Warning! If you prefer to perform a load in Text Mode it is very important to select Test Mode before clicking ‘Update Data’. You will not have an opportunity to do so at a later time. As soon as you click OK the data will be updated.

Import Tags


Confidential

48

Import tags can be used to migrate tags/folders from another system to Reveal or to bulk tag/folder using a text file. Import tags will use the entire Import Profile to parse the data, including the mult-value and nested delimiters.

If you have multiple tag sets you will need to run multiple passes to update all.

Note: There is functional limit to the amount of tags/folders which can exist as large amounts of folders can slow down the system. As a result, we recommend no more than 1,000 and do not recommend using folders to mirror Original Document Location.


Confidential

49

Indexing Data

Index Data – ISYS

Indexing Overview InControl uses either Perceptive Search (ISYS) or Elastic Search technology to index. ISYS will create a single index per load. Elastic Search will combine loads into a single index.

After loading the data, it is now ready for indexing. Indexing will do two things:

1. Index native files or text files for Searching 2. Create an HTML View based on those natives or text files

Notes: ISYS has a max limit of 128 indexes. For very large collections it is important to bundle your imports so you do not exceed this limit. For those using ISYS 10 and later you may be able to increase to upwards of 500, but this could cause performance issues.

Documents Not Indexed Documents which are not indexed will appear in the HTML view with the following message and will not be searchable. Documents that are not indexed will have the following error messages in the index error log:


Confidential

50

Documents Not Searchable Documents that not searchable will appear with the following message in the HTML view unless they are ‘Image Extensions’ as defined below.

Image Extensions Native images with the following extensions: .GIF, .JPG, .JPEG,.PNG are displayed in the HTML view in image form, even though they are not indexed by ISYS. OCR indexed by ISYS for these file types is, therefore, not displayed for these documents. As a result, users may hit on text not visible in the HTML or Image views for these documents. To turn off this image handling you can modify the ImageFileExtensionsForDisplay setting in the Operations Center System Settings:

Preparing for Indexing By default, the software will use the path in EXPORTED_NATIVE_FILE and index what it finds there..

To prepare for indexing, it is important to load a valid path into EXPORTED_NATIVE_FILE and, if available, text or OCR path into OCR_PATH. You may also chose to use the OCR_FLAG field to force the software to index the file found in the OCR_PATH. Additional information on these options is below.


Confidential

51

ISYS does not support paths exceeding 255 characters. If a path is included which exceeds 255 characters indexing will fail.

OCR Path OCR/Text path information should be loaded anytime you have document level OCR or extracted text that you may want to use instead of or as a back-up for the native file indexing. This is recommended as it ensures the most complete indexing and viewing.

OCR Flag There is an OCR Flag field which can be used to force the indexer to use the OCR path instead of the native file (EXPORTED_NATIVE_FILE) for indexing. For each document, OCR_FLAG should be blank, or set with a 'Y' value to indicate explicit usage of the OCR_PATH for indexing.

More details on the behavior of this is below.

When OCR Flag is Not Set 1. If the native file is indexable, the indexer will index the native file, even if OCR_PATH is populated. The native document will be displayed in the HTML view. 2. If the native file is unindexable, not searchable, or does not exist the indexer will failover and use the file in the OCR_PATH.* The file referenced in the OCR_PATH will be displayed in the HTML View.** 3. If there is no usable native file and no usable OCR file the indexer will error.

When OCR Flag is set 1. The indexer will use the OCR_PATH file. The file referenced in the OCR_PATH will be dispayed in the HTML view.** 2. If the indexer can't find the document referenced in the OCR_PATH the indexer will log an error.

OCR & PDFs*

InControl will automatically index using the ‘OCR’ when the native file is unindexable for almost all file types. The exception to this is PDF files. InControl can identify searchable and non-searchable PDFs. Searchable PDFs will be indexed and presented in the HTML view. Non-searchable PDFs will be added to the index error log. However, InControl will not automatically index the OCR for these non-seachable PDFs . To ensure these items are searchable and that text appears in the HTML view, make sure to use the OCR_FLAG.

OCR & Images**


Confidential

52

Native images with the following extensions: .GIF, .JPG, .JPEG,.PNG are displayed in the HTML view in image form, even though they are not searchable. OCR indexed by ISYS for these file types is not displayed by default in the HTML view for these documents and will not offer hit highlighting even if the OCR_Flag is set. For addition information on this see ‘Image Extensions’.

Indexing Process The index area will list all data loads. You will be able to index (or later re-index) selected data loads.

To index/reindex all :

1. Confirm your index path and name is correct. The last folder in the path is the name of your index. Index names (folders) cannot contain spaces.

a. To change your index path or index name click the Index Path and edit or use browse to browse to new location and create a new folder.

2. Select the loads you would like to index/reindex in the Index column. 3. Click Index/Reindex button. 4. Once completed, the index will be checked as ‘Enabled’ and ‘Completed.’ Note: The indexing is done in order from the first load to the last.

To index/reindex a sub-set:

1. Create a list of the documents you would like to index/reindex. a. The list may contain BEGDOC or ITEMID numbers for the documents.

2. Select the field matching your list: BEGDOC or ITEMID 3. The software will automatically select all indexes. 4. Click Index/Reindex button.


Confidential

53

5. Once completed, the index will be checked as ‘Enabled’ and ‘Completed.’

Indexing Options Indexing options can be found in a pop-up window once you click ‘Options’. This area has two tabs: General and Common Words.


Confidential

54

Note: If options are changed after indexing has already completed, you must save the index to a new location and index to that location. Reindexing will not create a new index based on the updated options.

General Indexing Options

Add Alerts for Unindexed Documents If the index is unable to complete successfully for an item the document will be flagged with an alert that can be searched for through InControl.

Index All Documents as Plain Text You can check this, if you have a set of data that you want to index all as plain text. The documents will not display a formatted HTML view but show all the text for a document.

Index Specific Files as Plain Text This option allows you to index files as plain text rather than with formatting. This speeds up the indexing process and, in situations where reviewers are reviewing images instead or you don’t have native files, can save time.


Confidential

55

Note: This most often used to index Excel as plain text because sometimes very large Excel files have difficulty indexing. When used, files will not be “formatted” HTML view in the review tool, however the text will be present. The user can then use the “download” button to open these Excel files in Excel. This option allows the very large Excel files to still be loaded and indexed without causing indexing issues.

Significant Characters By default: -0123456789 are all significant characters.

If a character is significant, the punctuated search word or phrase will match only with words or phrases containing matching punctuation characters.

In other words, significant characters are those which are considered an important part of a word and are indexed. For example, if the hyphen is defined as significant, then "nucleic-acid" is indexed as "nucleic-acid", not as "nucleic acid".

Insignificant Characters By default: ‘_` are insignificant characters.

If a character is insignificant, the occurrence of specified punctuation in the text is ignored, meaning that a search will bring up matching words and phrases, whether punctuated or not.

In other words, insignificant characters are those which are not considered an important part of a word and are not indexed. For example, if the hyphen is defined as insignificant, then "nucleic-acid" is indexed as "nucleicacid".

Combined Search Designator By default, ! is the combined search designator.

This is used when searching in the review database, and you wish to combine two searches to be run together.

Example:

!295 AND !389

Typing this in the text search area will run the search numbered 295 and combine it with the search numbered 389.


Confidential

56

Common Words Common words are terms that may appear frequently within a document but add little value to the ability to search documents. Due to the frequency that these words can occur within a document, they can significantly add to the overall size of an index. By removing common words, we reduce the size of the index, thus increasing overall performance of the index. InControl provides a means of maintaining wordlists.

There is a default word list provided which contains a listing of common words as well as individual letters. Users can choose to use the default word list or they can choose to make a custom word list. Or, they can choose to not use a common word list at all.

Custom word lists To create a custom wordlist, the user selects 'Custom' from the word list dropdown. This enables the list of the common words associated with the index for the case.


Confidential

57

Adding a custom term to a word list By selecting 'Add' the user is presented with a dialog to add a new term. Once the user enters the term and clicks save, the new term will be added to the wordlist for the case. In order for the new term(s) to be added to the common word list and no longer be indexed, the user must delete the existing index and run a new index.

Deleting a term from a wordlist By selecting one or more terms from the word list and clicking the delete button the user can remove terms from the common word list. This means that, if those terms appear in the document set, they will now be indexed and returned during searches.

File Size Limits InControl provides an easy means of forcing the indexing of certain document types as text, or skipping indexing of the files, altogether, if they are above a configurable file size threshold. This can be a helpful tool if there are certain documents/document types that are holding up indexing. The user can move quickly past those documents and complete the indexing. Documents can then be re-indexed at a later point in time, or be reviewed natively.

For indexing as Plain Text, the threshold can be set for Microsoft Excel, Word, and Power Point. The threshold can also be set for Adobe PDFs. The default value for the threshold is 1000KB (approximately


Confidential

58

1 MB). For any files above this threshold, they will have their extracted document text indexed and displayed in the HTML view. This will remove any graphics or specialized formatting for the document.

If a user wants to skip processing completely for a document that is larger than a certain size, they can configure the Do Not Index threshold. This threshold can be set for Microsoft Excel, Word, and Power Point. The threshold can also be set for Adobe PDFs. There is also an 'Other Files' threshold that can be configured. Other files can be specified in the 'Other File Types' text box. Items must contain the period for the extension and multiple document types can be included by separating them with a semi-colon. The default value for the Do Not Index threshold is 10000KB (approximately 10MB).

Perceptive Index Logs InControl errors can be viewed and/or exported once indexing completes.


Confidential

59

Below are the steps to follow:

1. Index/Reindex the documents you would like to index/reindex.

2. View totals to see if there are any errors.


Confidential

60

3. If there are errors or items logged click view log.

4. Review log to determine if there are any errors.

5. If desired, you may export the log by clicking ‘Export Log’. This will export the log as a csv and launch in the application you’ve associated with csv.

Example launching in TextPad:


Confidential

61

Note: This feature is only available after indexing completes and only for the last index(es) created. Should you forget to view or export an error log you can find them at the location specified in System Settings/Shared/OpsToolLog.

Logs are named by case. The most previous log has .OLD appended it to it and older index logs have date & time appended.

ISYS Indexing Max Limits The following chart lists each item with its limitation:

Maximum number of different words: 2 billion

Maximum number of different common words: 65,535

Maximum number of occurrences of a given word: 2 billion

Maximum number of words per index: 32 billion

Maximum number of documents per index: 64 million

Maximum number of directories: 1 million

Maximum significant word length*: default 20 characters, maximum 64

Maximum file path name length: 255 chars

Maximum number of paragraphs per document: 65,535

Maximum number of words per paragraph: 65,535

Maximum number of words per document: 2 billion

Maximum number of indexes in a chain (case): 128

Maximum number of indexes (systemwide): unlimited

Maximum index size: 384 gigabytes (GB)


Confidential

62

Indexing Data – Elastic Search

Indexing Overview Starting in version 7.0, InControl has incorporated Elastic search as an alternative tool to index data. Elastic will create a single index for all documents loaded to a case.

There is a new System Setting called SearchEngineTypeDefault that allows a user to set their indexing/searching environment to support Perceptive only, Elasticsearch only, or a mixed environment. Depending on the setting used, there will be different options available for Case Creation, Indexing, Entities, and Checking the Health of the indicies.

After loading the data, it is now ready for indexing. Indexing will do two things:

1. Index native files or text files for searching 2. Create an HTML View based on those natives or text files

Index names will be automatically generated based upon the case database name. An Elasticsearch index name must meet the following two criteria:

1. The index name cannot contain any spaces 2. Any alphabetic characters within the name must be lower-case

Documents Not Indexed Documents which are not indexed because the native file is not locatable and there is no OCR text provided will not be indexed and, therefore, their document text will not be searchable. The HTML view will show the following error:


Confidential

63

Documents that are not indexed will have the following error messages in the index error log:

Documents Not Searchable Documents that not searchable will appear with the following message in the HTML view unless they are ‘Image Extensions’ as defined below.

Documents that not are not searchable due to containing no usable text will appear with the following message in the error log.


Confidential

64

Image Extensions Native images with the following extensions: .GIF, .JPG, .JPEG,.PNG are displayed in the HTML view in image form, even though they are not indexed by Elastic. OCR indexed by Elasticsearch for these file types is, therefore, not displayed for these documents. As a result, users may hit on text not visible in the HTML or Image views for these documents. To turn off this image handling you can modify the ImageFileExtensionsForDisplay setting in the Operations Center System Settings:

Preparing for Indexing By default, the software will use the path in EXPORTED_NATIVE_FILE and index the native file that is referenced in that location.

To prepare for indexing, it is important to load a valid path into EXPORTED_NATIVE_FILE and, if available, text or OCR path into OCR_PATH. You may also chose to use the OCR_FLAG field to force the software to index the file found in the OCR_PATH. Additional information on these options is below.

InControl does not support paths exceeding 255 characters. If a path is included which exceeds 255 characters indexing will fail.

OCR Path OCR/Text path information should be loaded anytime you have document level OCR or extracted text that you may want to use instead of, or as a back-up for, the native file indexing. This is recommended as it ensures the most complete indexing and viewing.

OCR Flag There is an OCR Flag field which can be used to force the indexer to use the OCR path instead of the native file (EXPORTED_NATIVE_FILE) for indexing. For each document, OCR_FLAG should be blank, or set with a 'Y' value to indicate explicit usage of the OCR_PATH for indexing.

More details on the behavior of this is below.

When OCR Flag is Not Set 1. If the native file is indexable, the indexer will index the native file, even if OCR_PATH is populated. The native document will be displayed in the HTML view.


Confidential

65

2. If the native file is unindexable, not searchable, or does not exist the indexer will failover and use the file in the OCR_PATH.* The file referenced in the OCR_PATH will be displayed in the HTML View.** 3. If there is no usable native file and no usable OCR file the indexer will error.

When OCR Flag is set 1. The indexer will use the OCR_PATH file. The file referenced in the OCR_PATH will be displayed in the HTML view.** 2. If the indexer can't find the document referenced in the OCR_PATH the indexer will log an error.

OCR & PDFs*

InControl will automatically index using the ‘OCR’ when the native file is unindexable for almost all file types. The exception to this is PDF files. InControl can identify searchable and non-searchable PDFs. Searchable PDFs will be indexed and presented in the HTML view. Non-searchable PDFs will be added to the index error log. However, InControl will not automatically index the OCR for these non-seachable PDFs . To ensure these items are searchable and that text appears in the HTML view, make sure to use the OCR_FLAG.

OCR & Images**

Native images with the following extensions: .GIF, .JPG, .JPEG,.PNG are displayed in the HTML view in image form, even though they are not searchable. OCR indexed by Elasticsearch for these file types is not displayed by default in the HTML view for these documents and will not offer hit highlighting even if the OCR_Flag is set. For additional information on this see ‘Image Extensions’.

Indexing Process The index area will list all data loads. You will be able to index (or later re-index) selected data loads.

To index/reindex all :

1. Confirm your index path and name is correct. a. Index names (folders) cannot contain spaces or upper-case characters. b. Index names can be changed by double-clicking in the Index Path box.

2. Select the loads you would like to index/reindex in the Index column. 3. Click Index/Re-Index button. 4. Once completed, the index will be checked as ‘Enabled’ and ‘Completed.’ Note: The indexing is done in order from the first load to the last.


Confidential

66

To index/re-index a sub-set of documents:

1. Create a list of the documents you would like to index/re-index. a. The list may contain BEGDOC or ITEMID numbers for the documents.

2. Select the Doc List checkbox. 3. Navigate to your file containing the document list to index/re-index by using the 'Browse To'

button. 4. Select the key field matching the value in your list from the dropdown: BEGDOC or ITEMID. 5. The software will automatically select all indexes. 6. Click Index/Re-Index button. 7. Once completed, the indexes will be checked as ‘Enabled’ and ‘Completed.’

Indexing Options Indexing options can be found in a pop-up window once you click ‘Options’. This area has three tabs: General, Common Words, and File Size Limits.


Confidential

67

Note: If options are changed after indexing has already completed, you must delete the existing index and re-index all batches within the case to see the changes take effect.

General Indexing Options

Add Alerts for Unindexed Documents If the index is unable to complete successfully for an item the document will be flagged with an alert that can be searched for through InControl. The flag will be displayed in the results grid and the error text gets written to the Alerts Detail field for the document. Defaults to disabled.

Index All Documents as Plain Text You can check this, if you have a set of data that you want to index all as plain text. The documents will not display a formatted HTML view but show all the text for a document. Defaults to disabled.


Confidential

68

Use HD Mode for Html This options allows for a higher definition rendering of the HTML view for documents that aren't indexed as text. This may result in better fidelity to the original native document. This will result in a slower processing time and larger index size. Defaults to disabled.

Remove HTML Metadata HTML extraction may produce some document metadata that is normally present in the HTML but hidden in the rendered view in the reivew page. This removes the hidden metadata and decreases the size of the index however this can slow down indexing. Defaults to disabled.

Skipped Failed Documents When Reindexing This option allows for the skipping of documents that have failed to index during a previous index run. The failed documents will keep any alerts that were previously set. If there are a large number of documents that failed during previous indexing, enabling this option may speed up indexing as it will not have to retry that set of documents. Defaults to disabled.

Use Fast Vector Highlighting A highlighter is used in Elasticsearch to be able to highlight search results from a query. The fast vector highlighter allows for better performance when indexing and highlighting of search results because word locations are stored with the document text data. InControl requires the use of the fast vector highlighter to allow for the highlighting of phrases. Users can disable this option to potentially reduce the index size, but phrase highlighting behavior will be compromised. Defaults to enabled.

Use Stemming This option allows for language specific stemming of words reducing words to word stem, base or root form. For example, if stemming were enabled and the user searched on the term 'profits'. The stem of this term is 'profit'. As a result documents that contain variations on the term that reduce to the same stem would also be returned. For example, documents that contained 'profit', 'profitability', and 'profitable' would all be returned as they all contain the stem 'profit'. InControl's hit legend will display the actual term returned and not just the stem of the word. If the user is interested in exact term matches only, then stemming should be disabled. Defaults to disabled.

Analyzer This option instructs Elasticsearch on how to interpret the language for the documents to be indexed. The analyzer divides text into terms on word boundaries, as defined by the Unicode Text Segmentation algorithm for the given language. It removes most punctuation, lowercases terms, and supports


Confidential

69

removing stop words. The analyzer is applied to the entire index, so you cannot have mixed analyzers for different batches contained within an index. If the user wishes to change the analyzer after a batch has been indexed, they must delete the index and re-index all batches. Defaults to "English".

Text Processing This option tells the indexing services where to store the text that is extracted from the documents. This needs to be set to "Index only" for normal indexing.

Html Processing This option tells the indexing services where to store the HTML that is extracted from the documents. The needs to be set to "Index only" for normal indexing.

Html Image Processing This option tells the indexing services where to store the HTML images that are extracted from the documents. The needs to be set to "File only" for normal indexing.

Overwrite Existing Files This tells the indexing services to overwrite HTML image files when saving if they already exist. Defaults to enabled.

File Naming This option tells the indexing service how to name the HTML image files when saving. Defaults to Begdoc.

Folder Naming This option tells the indexing service hot to name the folders where the HTML image files are saved. Defaults to Begdoc.

Combined Search Designator This is where the character to be used for Combined Searches is defined. This character is relayed to the search service that a search is to be performed as a combined search by number. Usage is detailed further in the User Guide in the Search area.

Common Words Common words are terms that may appear frequently within a document but add little value to the ability to search documents. Due to the frequency that these words can occur within a document, they can significantly add to the overall size of an index. By removing common words, we reduce the size of the index, thus increasing overall performance of the index. InControl provides a means of maintaining wordlists.

There is a default word list provided which contains a listing of common words as well as individual letters. Users can choose to use the default word list or they can choose to make a custom word list. Or, they can choose to not use a common word list at all.


Confidential

70

Custom word lists To create a custom wordlist, the user selects 'Custom' from the word list dropdown. This enables the list of the common words associated with the index for the case.

Adding a custom term to a word list By selecting 'Add' the user is presented with a dialog to add a new term. Once the user enters the term and clicks save, the new term will be added to the wordlist for the case. In order for the new term(s) to


Confidential

71

be added to the common word list and no longer be indexed, the user must delete the existing index and run a new index.

Deleting a term from a wordlist By selecting one or more terms from the word list and clicking the delete button the user can remove terms from the common word list. This means that, if those terms appear in the document set, they will now be indexed and returned during searches.

File Size Limits InControl provides an easy means of forcing the indexing of certain document types as text, or skipping indexing of the files, altogether, if they are above a configurable file size threshold. This can be a helpful tool if there are certain documents/document types that are holding up indexing. The user can move quickly past those documents and complete the indexing. Documents can then be re-indexed at a later point in time, or be reviewed natively.

For indexing as Plain Text, the threshold can be set for Microsoft Excel, Word, and Power Point. The threshold can also be set for Adobe PDFs. The default value for the threshold is 1000KB (approximately 1 MB). For any files above this threshold, they will have their extracted document text indexed and displayed in the HTML view. This will remove any graphics or specialized formatting for the document.

If a user wants to skip processing completely for a document that is larger than a certain size, they can configure the Do Not Index threshold. This threshold can be set for Microsoft Excel, Word, and Power Point. The threshold can also be set for Adobe PDFs. There is also an 'Other Files' threshold that can be configured. Other files can be specified in the 'Other File Types' text box. Items must contain the period for the extension and multiple document types can be included by separating them with a semi-colon. The default value for the Do Not Index threshold is 10000KB (approximately 10MB).


Confidential

72

Index Logs InControl errors can be viewed and/or exported once indexing completes. Under the Status area there are categories to outline the various types of errors that can occur during indexing. Errors such as native files not found, empty documents, and documents types/formats that aren't supported will have their counts reported here.

For a more detailed look at the logs, the user can click the view log button and they will be presented with a complete list of errors that occurred for the latest index/re-index job.


Confidential

73

Upon completion of an index, or re-index, job the user should look at the status area to determine if any errors occurred during the job.


Confidential

74

If errors did occur and they fall outside of the expected behavior for the batch of documents, the user can use the 'View Log' button to get a detailed list of errors.

Errors are presented in a grid that has scrollbars and page navigation present. Users can export results to CSV or XLSX format.


Confidential

75

Additionally there are filter tools present at the top of each column which provide users with the ability to narrow down the result set presented in the grid.


Confidential

76

Bulk Update

The bulk update section allows the administrator to update fields to a single value. An update of BATCHID would be an example. The update is based upon a SQL query or a file list of documents.


Confidential

77

Create Alerts

Create alerts allows you to add alerts to the ALERT_DETAIL field and turn the alert feature on\off at a document level using the ALERT yes\no field. This can also be done by loading data into those fields at loading time.

Indexing Transcripts


Confidential

78

In order to successfully index ASCII Transcript files, transcripts must conform to the following standards:

• Documents must be in ASCII with a line length of 75 characters or less. • Documents should be double-spaced (two CR/LF pairs at each line ending), or single-spaced with the character code 27 (escape) followed by the character 'S' included in the transcript text. ISYS will adapt its paragraph detection logic accordingly. • Paragraphs should be delimited by three CR/LF's. • Each line should be line-numbered. ISYS will display line numbers when documents are in the Browser, but will ignore them for the purpose of phrase searching. • Page breaks should be hard (character code 12).

ISYS provides an escape code facility to set page numbering other than from 1. To specify a non-sequential page number, insert a character code 27 (escape), followed immediately by the new page number at which to start. It is not necessary to number every page, only where the page numbering deviates from sequential order. For example, a transcript file which begins at page 123 could have the character code 27 followed by 123 at the top of the document. The process by which you insert the character code 27 varies for different text editors and word processors. If your transcript lines are not numbered, include the character code 27 (escape) followed by the character 'U' in your transcript text. ISYS will alter its phrase-searching strategy across line boundaries accordingly. To insert special codes into ASCII documents, you can use the DOS Edit facility. Open the document you wish to edit with the Edit command from the DOS prompt. To begin inserting characters press Ctrl-P then the code you need to use from the following list. Note that for each code you must first press Ctrl-P.

Code Keystroke Symbol 12 Ctrl-Shift-L 27 Ctrl-[

Note: ISYS recommends confirming to this standard if you will be using ISYS paragraph specific operators.


Confidential

79

Images

Bulk Image


Confidential

80

The bulk image utility is used for batch TIFF on the Fly processing. The administrator selects the case, image format (TIFF, JPG, etc), the case’s image folder, and how to handle existing images.

Then the user can either TIFF all large files (based on file size), based on query, or from list of ITEMIDs or BEGDOCs.

Loading Images

Images Overview InControl images can be created by 1)bulk imaging, 2)image-on-the-fly, or 3)loading images in. When you load images in, you will need to inform InControl on which image set the images belong to and use the below process.

Image Load Process 1. Select Case 2. Select\Create Image Set 3. Select\Confirm Image Key 4. Select Load File 5. Indicate if Index Path should be used and, if needed, Select Source Directory.

Image Sets InControl supports multi-version image loading. When you load images you will be able to use an existing or create a new image set to place the documents into.

When reviewers review they are able to select which version to look at based on the image set name and document number.

The only existing image set in all databases is the ‘Original.’ This Original set can be loaded into and/or created through use of image-on-the-fly or Bulk Imaging features. It is the default image set for viewing and production.

New image sets can be created by clicking ‘New Set’ and providing the image set name.


Confidential

81

Image Load Files Pre-processed images can be loaded into the system via an IPRO LFP or Opticon (Concordance Image) OPT file with single page images. The Image Directory will populate when the case is selected with the path to where the images reside for that case. You will select the load file which will populate the Load File Type, and load the images.

The images will then load, and are visible via the Image View.

Use Load File Path If your image load file has a complete path to the images they will load from that location, however if your path in the image load file doesn’t have the complete path and starts with the IMAGES folder you can append the path to that IMAGES folder by un-checking the Use Load File Path box and browsing to the IMAGES folder in the Source Directory field. Make sure there is a slash(\) at the end of the Source Directory Path.

Overwrite Existing Images When Overwrite Existing Images any images that have been created in the IMAGES folder, and any references to those images in the db, will be removed and replaced with the images that the image load file is pointing to.


Confidential

82

Keep Images In-Place Images can reside on any connected device that the Reveal application has access to. This can drastically reduce load time when not copying files from the source to the Reveal Images share. To use this feature, make sure the ‘Copy Images’ checkbox is not checked when loading images.


Confidential

83

Analytics

Email Threads


Confidential

84

This can be used to build threads once data has been loaded. This will use whichever of the following fields are available. For the most complete threads, load in all of these fields.

Email Threading MESSAGE_ID SENT_DATE SENT_TIME THREAD_IN_REPLY_TO THREAD_REFERENCES

Entities

Entities - ISYS

The entities section will populate the entities section of InControl™. For the entities listed below, data is pulled from the document metadata loaded into the database for the field. It is not populated from the


Confidential

85

natives or index. If the data is not populated in the metadata load file, then it entities section will not work properly.

Field->Entity Types in populates AUTHOR ->Document Author SENDER -> Message Originating Domain, Message Sender RECIPIENTS -> Message Receiving Domain, Message Addresses BCC-> Message Receiving Domain, Message Addresses CC_ADDRESSES-> Message Receiving Domain, Message Addresses

For the named entities the entities are created using the index and are therefore created using the OCR or native files. If you don’t want to create indexes for the named entities you can just check the Ignore Isys button and it will only create entities from the metadata in the database.

To create entities, simply click the “Build/Rebuild Tables” button.

Note: if rolling data into the DB, it is recommend building entities after everything is loaded. Otherwise, clients will need to take the necessary precautions to insure they do not miss documents if reviewing by entities.

Entities – Elastic Entities are built in a similar fashion if using an Elastic index.


Confidential

86

Near Duplicates & Clusters To identify near duplicates and cluster documents go to ‘Near Dupe and Cluster.’

Select your case and then click ‘Find Near Duplicates’ to identify near duplicates and build near duplicate groups or ‘Build Clusters’ to cluster documents.


Confidential

87

You have a number of options in the Operations Center Near Duplicate and Clustering area:

You can control your similarity by moving the bar to the right or typing in a percentage. The default is 80%. Near duplicate identification is the first step to clustering, so this impacts both. The SQL Document Filter can be used to filter using the fielded data imported during the load process or created during field coding. This data is stored in the IMPORT_DOCUMENTS SQL table and SQL Document Filter enables you to search that table using SQL operators. If you add query language here, InControl will only identify duplicates or cluster those documents which meet your search criteria.


Confidential

88

Filtering rare words is also an option to speed processing. It ignores words that appear in less than 1/2 percent of the collection. This option is useful for collections of mixed languages or with high quantities of alphanumeric words like serial numbers.

The near duplicate detection process supports multi-threaded processing for both the document reading phase as well as the comparison phase. This can result in a significant reduction in the time required to calculate near-duplicates and clusters for a project. By default the tool will use n-2 processors, where ‘n’ is the number of processors on a given machines. Therefore, the more processors your machine has, the less time it will take to complete the process. (A sample 250K document native dataset on a 16-processor machine, should take about one hour to complete).

The settings interface will log the documents that are skipped, while also stating the reason. Typical reasons are, the document was too large or small, the document contained no text or the document was an unsupported file type.

UTILITIES

Querying Data


Confidential

89

In this section, users familiar with how to write SQL Queries can write custom queries against the SQL database. It is also an area with Reveal’s Support department might utilize via a support call to help troubleshoot.

Querying Hits


Confidential

90

Query hits will give you a record of the number of documents where the term is found and how many total times the term is found in the documents.

Manage Lists

To create a new list:

1. Click Add List.


Confidential

91

2. Put in the list name, description, and sort order. Click Save. Note: Manual order will allow you to order manually after values are added.

3. Next, select your list (if not already selected) and select your value add method. You can click Add Values to manually add values or select one of the radio buttons to Add from a text file or a field.


Confidential

92

4. Once values are added, if your sort is manual, you can change the value order by renumbering in the List Order field or sorting on column headers and then clicking ‘Re-Order’ to save the changes.


Confidential

93

5. Once you’ve added your list(s) and values you can associate the lists to a field or fields in the Web Tool at Admin\Fields\Manage Fields. See the Admin area of this new features guide for more details.

Near Dupe Export You can export near duplicate detail using the Near Dupe Export feature.

There are two options for export:

• Pivot – Each document is placed in a single near duplicate group. Duplicate groups may be modified as new data is loaded and re-exported. It is best to load all data before doing a Pivot export.

o Fields exported are:


Confidential

94

NDID - Group ID shared by all items in near duplicate group – BEGDOC or ITEMID per Group ID selection.

NDPIVOTDOC – Group ID shared by all items in near duplicate group. BEGDOC or ITEMID per Group ID selection. Also considered Pivot Document.

DOCID - Document’s ID – BEGDOC or ITEMID per Document ID selection

NDSIMILARITY – the percentage the document is similar to the pivot

NDISPIVOT – will contain a Y if the item is the pivot

o Pivot Grouping Rules

If doc A is in question, and doc A is 81% similar to Pivot_1 and 80% similar to Pivot_2, then it is assigned to Pivot_1 and not found in Pivot_2’s group.

If similarity to both is the same, say 80%, then it gets assigned to the first created group - Pivot_1.

• Relational – Each document can be part of multiple near duplicate groups. This mirrors the relational grouping utilized in InControl and is an accurate export of its near duplicate groupings.

o Fields exported are:

DOCID – Document ID – BEGDOC or ITEMID per Document ID selection.

NDDOCS – Near Duplicates – BEGDOC or ITEMID per Near Duplicates selection

NDSIMILARITIES – Near Duplicates Similarity percentage with BEGDOC or ITEMID per Near Duplicates selection.

o Relational Grouping Rules

If doc A is in question, and doc A is 81% similar to Pivot_1 and 80% similar to Pivot_2, then it is assigned to both Pivot_1 and Pivot_2’s group.


Confidential

95

Build Document Folders

Build document folders can be used to build folders under the ‘Documents’ and ‘My Views’ folders.


Confidential

96

Build document folders will build document folders based on the selected Document Folder Field and delimiters found in the System Settings: ‘DocumentFoldersMultiValueDelimiter,’ and ‘DocumentFoldersDelimiter’.

• Selected Document Folders Database Field o This must the folder path information used to build the folders.

This can be a single value or multi-value field. Multiple values must be delimited with the Document Folders Multi Value delimiter mentioned in more detail below. By default this delimiter is a semi-colon (;).

Any subfolders (nested values) inside a value must be delimited with the Document Folders Delimiter mentioned in more detail below. By default this delimiter is a back-slash (\).

The default field used is the SOURCE field, but this can be changed system wide or by project by modifying the ‘DocumentFoldersDatabaseField’ system setting.

• Document Folders Multi Value Delimiter o This is used to parse multiple values (paths) in the Document Folders Database Field. o The default is ; (semi-colon), but this can be changed system wide or by project by

altering the DocumentFoldersMultiValueDelimiter System Setting. • Document Folders Delimiter

o This is used to parse the folders in a value o The default is \ (back-slash), but this can be changed system wide or by project by

altering the DocumentFoldersDelimiter System Setting.


Confidential

97

Notes:

• See the Support Functions\Systems Setting section of this guide for more information. • If you do not want to create this at data load, you may also use the Build Document Folders

Utility.

Rebuild Document Data In the event data needs to be updated, you can use this utility re-create things like family relationships. For example, if attachments were not originally associated with their parent documents, you can rebuild the data as long as the correct Parent ID is populated.

Reports The Reports area contains system wide reporting which gives you visibility into cases overall.


Confidential

98

It contains a Currently Logged in Users report organized by Case.


Confidential

99

Support Functions

System Settings

System settings are now configurable in the Operations Center and all support functions such as System Settings and Password Rules have moved to their own area 'Support Functions'.

Working with Application Settings This is a settings area where Reveal Support or an advanced administrator can view system/installation settings for troubleshooting and assistance.


Confidential

100

Within the Web.config file one setting that will be specific to individual installs is the link for Support. To update to a specific URL change this value in the web.config file, <add key="SupportSiteLocation" value="http://support.CLIENTVALUE.com"/> The key here is that changing this setting will boot everyone that is logged on, so it should be done off hours.

Password Rules

This section is where an administrator sets the rules for user passwords. Required fields are: password length, number of attempts until lockout, time for password reset, and number of different passwords before reuse.

http://support.clientvalue.com/


Confidential

101

DB Updater

DB updater allows you to run SQL scripts. It does not allow you to run select and see documents, only update.

Note, all cases are selected by default.


Confidential

102

Index Check - ISYS

For cases that are indexed using Perceptive Search, use this to verify if the indexes are not corrupt. If using Elastic, there is an Index Health option in the Elastic Index screen.

Email Users The Email Users function allows the administrator to send email messages to all users in specific case, or multiple cases. This can be used to send maintenance windows, for example. Attachments can also be included if needed.


Confidential

103

Email Error Logs Reveal and its administrators can use logs to trouble shoot problems. We may request that you send us these logs, either individually or in bulk using the ‘Email Logs’ feature.

Email Logs Email logs allows you to email Reveal support your web server, app server, and ISYS logs. This will only work if you’ve connected InControl to an email system.


Confidential

104

To locate, review, and\or send Reveal the logs manually you can go to their stored locations on your servers at Reveal\Logs.

Entering License Information To enter license information click on ‘Enter License’ button in the lower left of the Operation Center. You will need to cut and paste the license key into the field and it should update the Licensed Users and Expiration Date.