(ATS3-PLAT01) Recent developments in Pipeline Pilot

Ton van DaelenSr. Director, Platform Product Management

ton.vandaelen@accelrys.com

The information on the roadmap and future software development efforts are intended to outline general product direction and should not be relied on in making a purchasing decision.

Introduction

• Target audience– Administrators– Developers

• What will you learn?– What is new that affects me or my users (and me indirectly)

• More of What’s New in– (ATS3-PLAT07) Pipeline Pilot Tips, Tricks and Challenges

• Slides and examples will be made available

Topics

• Release history• Administration

– Performance enhancements– Server monitoring– Usage monitoring – Protocol validation– Security

• Integration– .NET components– Python (prototype)

• SharePoint integration

• Pipeline Pilot 8.0 – Dec 2010– Design mode– Local subprotocols– Data flow monitoring

• Pipeline Pilot 8.5 – Aug 2011– Support growing deployments– Isentris and Notebook integration– Support for building ‘apps’

• Pipeline Pilot 8.5 CU1 – Nov 2011– Protocol comparison– Performance

Pipeline Pilot Release History

9.0Enterprise

Workgroup

Personal1-4

6.0

7.0

8.58.0

5.0

Science and Applications Release History

7.5

8.08.0

8.5

Imaging (HCS)Reporting (Canvas elements)Advanced data modeling

Chemistry (MDL formats)Reporting (Stylesheets)Imaging (DICOM)Direct cartridge

Imaging (Big Image Explorer)Chemistry (Content, NEMA, Cheshire)Next generation sequencing

Isentris and Notebook integrationChemRegBioReg

Database Search

Apps

Collections

Mini-Survey

• What server version(s) are you running?– Pre-7.5?– 7.5– 8.0– 8.5

Administration

Performance Testing

• Performance increasingly critical because of ‘apps’– Both Accelrys and custom

• Performance testing– Track our progress– Spot errors

• How do we do this?

Performance Testing

Threads / Windows

Throughput / Windows

Linux / Threads and Throughput

1 5 10 20 50 100

ODBC 7.5 235 202 192 218 1558 18622

ODBC 8.0 224 166 164 156 482 1015

ODBC 8.0 (PoolID)

122 137 117 122 246 367

10003000500070009000

1100013000150001700019000

Latency of PP 7.5, 8.0 & 8.0 PoolID

Avg

Resp

onse

Tim

e (m

s)

1 5 10 20 50 100

ODBC 8.0 224 166 164 156 482 1015

ODBC 8.0 (PoolID) 122 137 117 122 246 367

100

300

500

700

900

1100

Latency of PP 8.0 & 8.0 PoolID

Avg

Resp

onse

Tim

e (m

s)

Pipeline Pilot Configurations Compared (ODBC Test Case)

ODBC latency is 44% lower than 7.5 and 22% lower than 8.0

w/out pooled servers

Improved scalability under load

Server Maintenance and Monitoring

• Administration portal – Run diagnostics– View and cancel running

jobs– View log files– Monitor CPU and memory

• Pro client– Server status indicator

Server Maintenance and Monitoring

• External tools

Windows Performance Counters Ganglia (Linux)

Usage Monitoring

• Why?– Internal accounting (‘Which dept is using imaging?’)– Justify investment (‘Are we using reporting?’)– Administrative (‘What protocols haven’t been used in 1 yr?’)

• Approaches– Usage report in admin portal– Custom analysis of log files– Protocol database analysis using Component Reader

• More in: (ATS3-PLAT11) Advanced Planning

Admin Portal Usage Report

Usage Report Example

Introduced in 8.5. For pre 8.5 servers, install protocols from ftp://download.accelrys.com/protocols/SummaryReport.zip.

Logfile Folder

Remote logfiles. Allows for manual copying if server is

not connected Local logfiles.

Installation logfiles.

Usage Log Analysis Through Protocols

• Usage protocol in C:\Program Files\Accelrys\PPS\apps\scitegic\admin\dataroot\xml

C:\Program Files\Accelrys\PPS\logs\usage\Pipeline Pilot.log

Analysis Example

Runs by User Runs by ClientType Most Frequent Protocols

Analyzing the XMLDB

• Questions to ask– Non-empty Password parameters?– Empty required parameter field?– File references?– Shortcuts as copies?– Protocol for a particular user?– Recent protocols?– Components of a certain collection?– Deprecated components?

• Combined with log file– What components were used recently?– For a migration, what protocols were run recently?

Analyzing the Xmldb

Protocol info:• Author• Date• Component ID

Component info:• Data source references• Passwords

Component ParametersParamName ParamValue ParamTypeSource data\Images\Cards\README.txt URLTypeMaximum LongTypeSourceTag None StringTypeKeep Properties PropertyPreviewTypeHasColumnNames TRUE BoolTypeDelimiter FigureOut StringTypeMultipleValueDelimiter None StringTypeParseNDimensionalArrayStrings TRUE BoolTypeAdditional Options GroupTypeCommentLines StringTypeUndefinedString StringTypeAllLines FALSE BoolTypeTrimWhitespace TRUE BoolTypeCheckSyntax TRUE BoolTypeAddEmptyProperties FALSE BoolTypeOnGeneralError Halt StringTypeOnDataTypeError Halt StringTypeComponentAttributes ComponentReturnsPass StringTypeRegistrant scitegic/generic StringTypeRegistrationDate Sat Feb 26 16:31:03 2011 StringTypeComponentDisplayName Read README.txt StringTypeDerivedFrom Delimited Reader StringTypeCustomErrorText StringTypeComponentIcon DocumentNode StringTypeComponentDisabled 0 LongTypeComponentGUID {7D2AF7C7-A077-4201-B10B-8E8283A5EADD} StringTypeComponentPoint 90 -95 StringTypeComponentHelp 100 StringTypeComponent Sticky Notes StringTypeRegistrationVersion 8.4.5.18 StringTypeComponentLocalID 0 LongTypeComponent Path Components\Data Access and Manipulation\File Readers\Delimited Text ReaderStringType

Accelrys or custom component?

Caption

Help text

Component Reader Example.xml

Date registered

Gotcha!

Component Parameter Interface

Component Parameter Interface

• Inspect protocols before moving to production• Rules checking:

– Hard-coded passwords– Bad file of database references– Broken shortcuts

• Configurable for different deployment modes – Standard protocol, Reports, Web services, RSS

Feeds, SharePoint, etc.

Protocol Validation

• Validation can be set up as hard requirement for publication• Operation is cancelled on validation errors

Protocol Validation

*Not a standard feature. Request from support

(ATS3-PLAT03) Enterprise Protocol & Package Deployment

• Component Review-o-Matic Protocol– Checks Documentation– Checks Parameter Names

• Create your own custom rules

Automating Component Best Practices

See: Components\Database and Application Integration\Utilities\Prototypes\Usability\Component Review-o-Matic

Security

Windows Integrated Authentication

• Run Web Port without providing username /password

• Supported via Kerberos and NTLM • Requirements

– DOMAIN authentication– Kerberos only works on remote servers.

Negotiation on local servers uses NTLM instead of Kerberos, so some restrictions may apply

– Pipeline Pilot Apache service must have valid Service Provider Name on the DOMAIN server. Running as "Local System" usually fulfills this requirement

– Configuration required on server and client’s browser

Windows Integrated Authentication

• Requires Active Directory settings• Enables Secure Services• Authentication provided by NTLM/Kerberos• Limitations

– Delegation not supported– Accessing network resources (e.g., proxy server authentication with

credentials, UNC paths) – Using PP credentials for components that require authentication (e.g.,

database, readers, integration, etc.) – Running protocols from within a protocol (e.g., RunProtocol PilotScript

function, parallel subprotocols)

Server and Client Configuration

• Run Apache service as a user: – To set up Kerberos service principal name for the Pipeline Pilot server, run the following on the

Domain controller: • setspn.exe -a http/SERVERNAME DOMAIN\apache_username

– On the Pipeline Pilot server, use the Services tool and set the Logon User for the "scitegic_apache_8_5_0" service.

• Client Configuration – Internet Explorer:

• Add the server as a trusted site (Tools > Internet Options > Security > Trusted Sites > Custom Level > User Authentication > Logon).

• Select Automatic logon with current user name and password. • If your server is already part of the Local Intranet, select Automatic logon only in Intranet zone.

– Firefox: • Browse to "about:config" and add the server names to the following preferences:

– network.negotiate-auth.trusted-uris – network.automatic-ntlm-auth.trusted-uris

Pass-through (Impersonation)

Scientist

Pipeline Pilot Web Services

Webform Basic

Database

ODBC

Web Services

Basic

SciServer

User Token

Integration

Pipeline Pilot Integration Points

DBs

Pipeline Pilot Enterprise Server

Protocol Runtime Environment (scisvr)

VB Script (On

Client)

Run Program Java Perl Python

VB Script (On

Server)

Admin Portal Help PortalWeb Apps Web Services API

.NET SOAP & HTTP

Telnet / FTP SSH / SCP ODBC /

JDBC

VB Script Cmd Line Java Classes

Perl Scripts

.NET Classes

REST / HTTP

Service

SOAP Service Cmd Line

Professional ClientRun Protocol

Command Line Client

Web Browser

Web Port

JavaScript Client SDK

JavaScript Client

.NET Client SDK

.NET Client

Java Client SDK

Java Client

Web Services API

SOAP Client

.NET

• .NET On Server– Points to a compiled .NET Assembly

• Dynamic .NET– Embed your .NET code directly into

the Server Component

• Leverage .NET supported languages– C#– VB.NET– IronPython

Component Development Update

A look at building a .NET Server Component

.NET Components Demo

• Hello world using dynamic C#

• Hello world using compiled C#

Python (Prototype)

• Requires– ActiveState Python– .NET runtime

• Windows only• Prototype now available

– Looking for testers

properties = component_datarecord.GetRoot().GetProperties()

queryString = properties.GetByName(u"queryString").GetValue().GetString();params = urllib.urlencode({'corpID': queryString})f = urllib.urlopen("http://vm-discoweb-dev.corp.mpi.com/rest/Compound/lookupCorpID/getCompoundProperties?%s" % params)properties.Define("Response", f.read());

SharePoint

Pipeline Pilot & SharePoint

Document Library

SharePointPipeline PilotLists

Pipeline Pilot SharePoint Bridge

Web Parts (3)Web Services

Read Only

Read/Write

Run Job

Protocols

User Created

IT Created

ISV Created

PP Scientifically Enables SharePoint

• Problem/Need: Want to create Pipeline Pilot-based Web Parts and deploy to organization

• Solution: Create packaged (“standalone”) Web Parts (Chapter 4, SharePoint Bridge Admin Guide)

Deploying Custom Web Parts

• Custom Web Parts are available in Web Part Gallery

• Provides a much simpler mechanism for end-users

Using Custom Web Parts

SharePoint Document Library

• Read from / write to document library– Use standard Reader/Writer

components– With Domain Authentication,

R/W will simply work– Alternatives

• Copy between document repository and temporary local version

• Add authentication options to Reader/Writer/Copy components

Summary

• Reasons to upgrade– Administrative– Development– Science and applications

• Plan your upgrade– Engage Accelrys support and services– (ATS3-PLAT11) Advanced Planning

• Resources– Component development guide– Accelrys Community: http://community.accelrys.com/

• Other sessions of interest– (ATS03-PLAT07) Pipeline Pilot Tips, Tricks and Challenges

The information on the roadmap and future software development efforts are intended to outline general product direction and should not be relied on in making a purchasing decision.

For more information on the Accelrys Tech Summits and other IT & Developer information, please visit:https://community.accelrys.com/groups/it-dev