Development of a Web Compute Server for a Bioinformatic Tool,

the Story of www.ConfBuster.org

Gabriel Bégin

Laboratory of Pr Patrick Lagüe

October 2018

Gabriel Bégin

1. Full-Stack Architecture

2. UML System Design

3. Web Front-End

4. Web Back-End

5. Queue System Versatility

6. Multi-Node Computing

I

Tables of Contents

1. Parameters Selection

2. Web UI Simplification

3. Data Transport Flexibility

4. Results Delivery

5. Data Privacy Policy

1. What is ConfBuster ?

2. System Architecture

3. Web UI Development

4. Queue Design

5. Compute Process

6. New Software Integration

Server DevelopmentImplementation for

Non-BioinformaticianAppendixA B C

Gabriel Bégin

Server Development

First Part

A

Gabriel Bégin

Server DevelopmentWhat is ConfBuster ?

Open Source - Suite of Tools for

Macrocycle Conformational Search

and analysis

- Search if a more stable conformation for a

given macrocycle is present.

- Search Results are base on :

- Energy

- RMSD

- Bioinformatics knowledge is needed.

Macrocycle

- Molecule with a ring

of at least 8 atoms

- High interest in

drug development

Conformers

Visulalisation

with PyMOL

Still.& Galynker. (1981).

Tetrahedron, 37, 3981–3996.

Result Matrix

with RMSD-Energy relationship

A-1

Adapted from https://www.rcsb.org/ligand/S1A.

Shen et al. (2004). Mol.Cell 16: 881

Yudin, A.K. (2015). Chem. Sci., 6, 30–49. Barbeau et al. (2018). J. Open Res. Softw., 6, 1.Barbeau et al. (2018). J. Open Res. Softw., 6, 1.

Gabriel Bégin

http://ConfBuster.org

Home Page

Gabriel BéginBégin, G., Barbeau, X., Vincent, A.T. & Lagüe, P. ConfBuster Web Server: a free web

application for macrocycle conformational search and analysis (in preparation)

Gabriel Bégin A-2

Server DevelopmentSystem Architecture

Unified

Modeling

Language

White 3D Boxes

=

Physical Node

Gabriel Bégin

Server DevelopmentSystem Architecture

Grey Boxes

=

(Software) Component

Unified

Modeling

Language

A-2

Gabriel Bégin

Server DevelopmentSystem Architecture

Colored Boxes

=

(Package) Layer

Unified

Modeling

Language

A-2

Gabriel Bégin

Server DevelopmentSystem Architecture

Small White Boxes

=

(Class) Module

Unified

Modeling

Language

A-2

Gabriel Bégin

Server DevelopmentSystem Architecture

Dotted Arrows

=

A uses B

Unified

Modeling

Language

A-2

Gabriel Bégin A-3

Server DevelopmentWeb UI Development

1

3

Front-End Technologies

1) Homemade Files

- Homemade Pages & Libraries

2) Bootstrap CSS

- Page Layout Management Libraries

- Language CSS

- Simple Syntax

3) jQuery JS

- UI Logic Library

- Simplifies the usage of Javascript

- Language : (jQuery &) Javascript

2

1

1

Gabriel Bégin A-3

Server DevelopmentWeb UI Development

Bootstrap CSS

- CSS Library

- Used to simplify Web Interface Design

- Layout based on a 12 column grid

- Rows are added manually

- Allow to configure Responsive UI Resizing

- Computer

- Tablet

- Mobile

- Also provide esthetic buttons/controls

- With current template and examples :

- Fast and Easy to Learn (~1h)

https://getbootstrap.com/

Gabriel Bégin A-4

5 / 12 7 / 12

http://ConfBuster.org

Home Page

Gabriel Bégin A-4

12 / 12

Row #1

Row #2

http://ConfBuster.org

Home Page

Gabriel Bégin A-3

Server DevelopmentWeb UI Development

jQuery

- Javascript Library

- Used to simplify Web Interface Logic

- Different Syntax based on Javascript

- Allow to configure Dynamic Animations

- Bouton Fade

- Text Fade

- With current template and examples :

- Fast and Easy to Learn (~1h)

https://jquery.com/

Gabriel Bégin A-3

Server DevelopmentWeb UI Development

1

2

3

Job Submission Flow

1) Home.html

- Data Validation

- Data Submission

2) DataReception.php

- Data Validation

- Data Encapsulation

3) Synchronous HTTP POST Transaction

- Transmission by B2B Transaction

4) Confirmation.html

- Transmission Confirmation for the user

4

Gabriel Bégin A-4

Server DevelopmentQueue Design

Job Submission Flow

3) Data Transaction

- Synchronous HTTP POST

- Web Server to Web Server

- Transmission by B2B Transaction

- B2B : Business-to-Business

- Prevent Socket Management

- Prevent TCP Port Management

- Apache already implements

multi-request management

- Allow Multi-Threaded Requests

3

Gabriel Bégin A-4

Server DevelopmentQueue Design

Job Submission Flow

5) ComputeJobPreparation.php

- Open Data Package

- Send Data to the Database Manager

- Send Data to the Mail Manager

6) DatabaseManager.php

- Send the Data into the Queue

- SQL Language

7) MySQL DBMS

- Database Software

- Store Jobs & Configurations

- Job Queue

8) MailManager.php

- Send a Queue Entry Confirmation

to the user

5

6

7

8

Gabriel Bégin A-5

Server DevelopmentCompute Process

Job Submission Flow

9) QueueManager.py

- Queue Scheduler

- Extract job from database

- Send job to a worker thread

- Runs as an independent server

10) DatabaseManager.php

- Extract the Data from the Queue

- SQL Language

11) ComputeManager.php

- Abstract Structure of a Worker

- Contains Processing Instructions

- Asynchronous Job Processing

12) FileSystemManager.py

- Simplifies File System Management

13) MailManager.py

- Send Results to User

10

911

13

12

Gabriel Bégin A-6

Server DevelopmentNew Software Integration

ConfBuster-Single-Molecule-Minimization.py

ConfBuster-Macrocycle-Linear-Sampling.py

ConfBuster-Analysis.py

Current System

Gabriel Bégin A-6

Server DevelopmentNew Software Integration

Any Commands Supported

by the Operating System of

the Server.

Compute Command Customization

Gabriel Bégin A-5

Server DevelopmentCompute Process Summary

Database Manager

(Queue Interaction)

Queue Manager

(Scheduler)

Compute Manager

(Worker)

File System Manager

(WorkBench Access)

Mail Manager

(Results Delivery)

https://pixabay.com/en/helmet-safety-safety-helmet-work-158268/

https://pixabay.com/en/checklist-task-to-do-list-plan-1295319/

https://pixabay.com/en/box-paper-delivery-box-1252639/

https://pixabay.com/en/workbench-carpenter-woodworker-148737/

https://pixabay.com/en/envelope-postage-letter-mail-158279/

Gabriel Bégin

Implementation for Non-Bioinformatician

Second Part

B

Gabriel Bégin B-1

Implementation for Non-BioinformaticianParameters Selection

https://github.com/patricklague/ConfBuster

Challenges

- Make ConfBuster accessible for chemist

- Selection of the most relevant parameters

Solutions

- No option selection for the first version

- Keep most of the parameters by default

Gabriel Bégin B-2

Implementation for Non-BioinformaticianWeb UI Simplification

Straightforward Design

- No menu

- All content on the same page

- Important content at the left

- Form at the right

- Only 3 operations needed

- 1) Email Input

- 2) Macrocycle file

- 3) Form submission

- Downloadable Examples

- Molecule sketchers

- Privacy policy

Gabriel Bégin B-3

Implementation for Non-BioinformaticianData Transport Flexibility

Usage of Transport Objects

- Increase flexibility for further

changes to the software stack.

- Prevent configuring every

property on each function.

- Allows to rapidly modify the

system according to users

needs.

Gabriel Bégin B-4

Implementation for Non-BioinformaticianResults Delivery

Delivery Method

- Result Sent by Email

- Universal & Accessible

- Zip Package

- Conformations Identified

- PyMOL Visualization Script

- RMSD/Energy Matrix

Gabriel Bégin B-5

Implementation for Non-BioinformaticianData Privacy Policy

Public Privacy Policy

- No Data are kept on the servers after the compute session

- Compute data are deleted

- Result data are deleted

Gabriel Bégin

Appendix

Third Part

C

Gabriel Bégin

=

C-1

AppendixFull-Stack Architecture

Gang of Four- Erich Gamma

- Richard Helm

- Ralph Johnson

- John Vlissides

OOP Design Patterns- Abstract Factory

- Singleton

- DAO/VO (*Composite)

- MVC (*Observer)

* Ensued from …

Multi-Layer

ArchitecturesWeb Stack Models

(Web) Full-Stack

Architecture+

MySQL

APACHE / PHP

APACHE

Examples of Web Stacks

GNU/Linux Windows

GNU/LinuxApacheMySQLPHP

Windows IIS SQL ServerPHP

GNU/LinuxApache TomcatMySQLJavaEE

Windows IIS SQL ServerASP.Net

MVC

DAO

VO

GN

U / L

INU

X

Gamma et al. (1995). Design patterns : elements of

reusable object-oriented software.

Addison-Wesley Professional.

Connolly, R. (2015). Fundamentals of web development. Pearson Education.

https://docs.microsoft.com/en-ca/iis/get-started/introduction-to-iis/iis-web-server-

overview

https://tomcat.apache.org/tomcat-9.0-doc/index.html

Gabriel Bégin C-2

AppendixUML System Design

- Standard for Software Modeling

- Uses Specific Symbols

- Categorized in Diagrams

- This diagram is based on :

- Application Diagram

- Component Diagram

- Package Diagram

- Class Diagram

Ambler, S. W. (2004). The object primer: Agile

model-driven development with UML 2.0.

Cambridge University Press.

Pilone, D., & Pitman, N. (2005).

UML 2.0 in a Nutshell. O'Reilly Media, Inc.

Unified

Modeling

Language

Gabriel Bégin C-3

AppendixWeb Front-End

Flexible Front-End

- Implementation with popular libraries :

- jQuery

- Bootstrap CSS

- Documented HTML / CSS / JS

- Various models of web pages

- Home

- Confirmation

- 404 / 403

https://github.com/patricklague/ConfBusterWebServer

Gabriel Bégin C-4

AppendixWeb Back-End

Customizable Back-End

- OOP PHP Implementation

- Documented PHP

- User Guide for Server Installation

- Step-by-Step

- Available on GitHub

- Possibilities for B2B Only Implementations

- Scripted Submissions

- Batched Transactions

https://github.com/patricklague/ConfBusterWebServer

Gabriel Bégin C-5

AppendixQueue System Versatility

Web UI

MySQL

DBMS Based Queue Access

- DBMS : DataBase Management System

- Usage of SQL Language

- Simplify Data Interaction

- Usage MySQL Connectors

- PHP PDO

- Python MySQLdb

- MySQL DBMS already implements

asynchronous query management

- Simplifies Queue Traffic Management

MySQL Connectors

PHP Python

Compute System

Gabriel Bégin

Web UI Desktop GUI Mobile Appl.

C-5

AppendixQueue System Versatility

Job Queue

Queue Connectors

PHP / Python / Java / C++ / …

Compute System

Gabriel Bégin

Compute SystemJob

Input

System

C-6

AppendixMulti-Node Computing

Job Queue

Job Input System- Shared Node

OR

- Dedicated Node

Compute/Queue System- Shared Node

OR

- Dedicated Node

Current System

Gabriel Bégin

Compute Node

#1

Job

Input

System

C-6

AppendixMulti-Node Computing

Job Queue

Compute Node

#2

Compute Node

#3

Acknowledgments

SMTP Server

Stéphane Larose

- Web Server Hosting

- HTTPS Certificate

- Institutional DNS

confbuster.ibis.ulaval.ca

Director

Pr Patrick Lagüe

Colleagues

Xavier Barbeau

Antony Vincent

Gabriel Bégin

Gabriel.Begin.4@ULaval.ca

Try it now !

www.ConfBuster.org