Clinical decision making with Machine Learning

Oleksii Barash, Ph.D.

Reproductive Science Center of San Francisco Bay Area

Disclosure

We have no financial relationship with any commercial interest related to the content

of this activity

Reproductive Science Center of the SF Bay Area

• Founded in 1983• In top 30 largest IVF (In Vitro

Fertilization) clinics in USA*• In top 20 clinics with the best clinical

outcomes*• Over 2000 treatment cycles (fresh and

frozen) in 2017

* - CDC Report 2015

What is infertility?

WHO - Infertility definitions and terminology

• Failure to conceive within 12 months of regular unprotected intercourse.

• Primary or secondary.

• 84% of couples will conceive within 1 year and 92% within 2 years.

Scope of the problem

• Infertility affects 12% of the reproductive age population in the US (≈12 million people)

• Infertility affects men and women equally

• More than 50% of infertility patients will have a baby with treatment

• Over 1.5M IVF cycles per year worldwide (≈ 200,000 in USA) in 2014

• Cost of one IVF cycle in US: 10K – 100K

Global fertility Market

Equity Research Reports, 2012

Key growth drivers:1. Aging and Infertility2. Increasing prevalence of Obesity3. Cultural shifts (“Celebrities” and LGBTQ)

Unreasonable expectations…

• 59% of childless women aged 35‐39 still planned tohave a baby• 30% aged 40‐45 did too!(Sobotka, Austrian survey data)

• 58% said they wanted 2 children (aged 21‐23)• Only 36% had achieved that by age 36‐38(Smallwood and Jeffries,UK Population Trends)

Biological clock

Speroff, 2004

IVF treatment overview

IVF is essentially manufacturing

• Complex multidimensional process;

• Constant intake flow of the patients;

• Cutting edge labor and equipment;

• Hundreds of contributing factors (Lab + Clinical);

• Every patient is unique – limited standardization

Ultimate goal –

single healthy baby

Manufacturing outcome prediction

IVF produces a lot of data?

• Main shareholders are open to cutting edge technologies

• Wide Electronic Medical Records adoption;

• IoT devices – sensors, incubators, microscopes, lasers

• Morpho-kinetics (time-lapse)

• Preimplantation Genetic Testing

• “Omics”

Transforming data into knowledge

• Increasing number of publications

• Retrospective and small

• Rare RCTs

Evidence based medicine

Conscientious, explicit and judicious use of current best evidence in making decisions about the care of an individual patient.*

* - Sackett. BMJ 1996;312:311-2

Meta-Analysis

Fertility and Sterility 2010; 94:936-945

• Small number of samples

• Diverse experimental conditions

Personalized decisions to be made in each IVF cycle

• Hormonal Stimulation protocol / dosage / duration

• Lutheal support

• How many embryos to transfer (1, 2 or 3)

• Embryo selection for the transfer (morphological and genetic)

• Financial products (risk sharing programs, money back)

Do You Know Your Embryo Biology?

Time-lapse and Machine learning

Embryo selection for the transfer

• From 1 to 30+ embryos per IVF cycle

• Many morphological and kinetic features per embryo

• Critical choice – no second chance

Traditional embryo evaluation

M. Montag, 2014

Time-lapse monitoring

M. Montag, 2014

Non-invasive imaging and predictions

EEVA (Early Embryo Viability Assessment)

• Xtend algorithm:– over 1,000 combinations of potential parameters

– includes egg age, cell count and Post P3 analysis – which measures cell activity after the four cell stage

– Post P3 is the result of a proprietary analysis based on 74 computer-based attributes that are combined into one parameter

– each embryo gets a developmental potential score ranging from 1 (highest) to 5 (lowest).

– 84% specificity vs 52% by traditional assessment

– The odds ratio of predicting blastocyst formation is 2.57 vs 1.67 by traditional assessment

EEVA (Early Embryo Viability Assessment)

Unusual cleavage patterns

EEVA Xtend algorithm

EEVA Xtend algorithm

Preimplantation Genetic Testing

Oocyte aneuploidy and maternal age

Handyside, 2015

• All primary oocytes are formed before baby-girl is born.

Preimplantation Genetic Testing

Handyside, 2015

DNA sequencing

DNA flow cell

Preimplantation Genetic screening

National Human Genome Research Institute, 2014

• Log scale!

Preimplantation Genetic Testing

National Human Genome Research Institute, 2017

• Log scale!

Single Nucleotide Polymorphism (SNP) algorithm

• 300,000 probs per embryo

• Per chromosome confidence

• Highly accurate and comprehensive results

• Parental genomic information

• Cumulative distribution function (cdf) curves

Cumulative live birth rate after SET,PGS, N=1024

# Cycles Live births Total ETs 1-l/n S(t)

1 178 313 0.43131 0.56869

2 22 59 0.627119 0.72952

3 7 15 0.533333 0.85574

4 1 2 0.5 0.92787

5 1 1 0 1

Presented by RSC team at ASRM 2016

Gene expression, stage & multinucleation

ML-based solutions for IVF

Univfy

Univfy algorithm:• Takes patient data

• Predictive model

based on 13,000 IVF

cycles;

• Chances for positive

outcome

• Chances of twins if 2

embryos were

transferred

Celmatix

Celmatix algorithm:

• Incorporated in our EMR

(ARTworks)

• Software as a service

(SaaS model)

• Data analytics platform to

help optimize patient

management and

counseling

Celmatix - Fertilome

Celmatix algorithm:• 25,000 peer-reviewed studies• 1,713 genes• 427 variant/diagnosis combinations• 201 gene-diagnosis combinations• 32 target genes in the kit

Endometrial Receptivity Analysis (ERA) by Igenomix

Patented in 2009: PCT/ES 2009/000386

Customized microarray (238 genes)

Bioinformatic analysis of data obtained by the customized microarray

Classification and prediction from gene expression.

Endometrial Receptivity Analysis (ERA)

Receptive

Model Classifies the Molecular Receptivity

Status of the Endometrium

Post-ReceptivePre-Receptive

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2010 2011 2012 2013 2014 2015 2016

% o

f cy

cles

ETx1

ETx2

ETx3

ETx4

ETx5

～ Average age – 36.0 ± 5.5 y.o.

～ 39.3% of all patients are over 38 y.o.

SET rate in non-PGT cycles(2010-2016), fresh D5 ET, N=3925

Preimplantation Genetic Testing (PGT) at RSC

～ SET frequency in PGS IVF cycles (average age – 37.5 ± 4.29 y.o. ) – 89.9%

FISH SNP – aCGH - NGS

661

1387

4

735

0

200

400

600

800

1000

1200

1400

0

200

400

600

800

1000

1200

1400

1600

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017

Nu

mb

er o

f IV

F cy

cles

Total volume

PGT cases

78

Live birth rate

Maternal age

Number of embryos for

biopsy

Morphology of the

embryos

SET vs eSET

D5 vs D6

BiopsyTotal

gonadotropin dosage

Number of previous

failed cycles

Number of normal

embryos per cycle

Number of eggs

Euploidy rate

Presented by RSC team: ASRM 2016, 2015, 2014; ESHRE 2015, 2016; PCRS 2014, 2015, 2016; PGDIS 2015, 2017

Factors affecting PGT outcomes

Live birth rate

Embryo_Age

Blastulation_rat

eDonor_

eggs Euploidy_rate Number

_of_normal

d5_to_total_rat

io Total_day_5_bx

Total_day_6_bx

Total_for_biosy

Bx_Day

Emb_Expansion

ICM

TE

Gender

Best_Embryo_For_ET

Elective_SET

Cycle_number

Number_of_Foll

icles

Zygotes

Fert_rate

Unfert

M2

M1

GV

ATR

Multi_PN

PN_1

Degenerated

Cleaved

Cleavage_rate

Number_ext_cu

ltureGood_ext_cultu

reNumber_to_blNumber

_CryoGood_d3_rateTVA_M

D

Number_of_tarnsfers_to_deliv

ery

Semen_Source

Fresh_Frosen_s

p

BMI

PATIENTTYPET

EXT

NO_OF_DAYS

SUMSTIM

ASPIRATED_OOCYTES

HCG_DRUG

TOTAL2PN

GRAVIDITY

PREM

TERM

SAB

BIOCHEMICAL

LIFETIME_SMO

KED

PRIORIVF

PRIORFET

PRIORIUI

HEIGHT

WEIGHT

PRIMARYDIAGN

OSIS

SEMENSOURCE

FSHLEVEL

NEAREST_AMH

MED1

Peak_E2

TOTALIUS

FOLLICLES_BIGGER_THAN_14

ASPIRATED_OOCYTES

NO_FROZEN

NO_VIT

INITIALCONSULT_PREM

INITIALCONSULT_GRAV

IDITY

INITIALCONSULT_SAB

INITIALCONSULT_TERM

INITIALCONSULT_BIOCHEMICA

L

Stim protoco

l

Factors affecting PGT outcomes

More factors?

Bias?

Reproducibility of the

results?

Factors affecting PGT outcomes

What if we can evaluate ALL available factors?

What if we can assess ALL available factors?

20 factors:

202 = 400 plots

381 factors

3812 = 145,161

plots

20 x 20

Machine Learning

Algorithm

Timeframe: Jan 2013 – Jul 2017

Retrospective analysis

Number of PGS transfers: 918

Average age: 35.6 ± 4.8

ONLY Single embryo transfers

Machine learning methods:

• GLM (Generalized Linear Models)

• RPART (Classification and Regression Trees)

• GBM (Generalized Boosted Regression Models)

IVF labEmbryo_Age

Blastulation_rateDonor_eggs

Euploidy_rateNumber_of_normald5_to_total_ratioTotal_day_5_bxTotal_day_6_bxTotal_for_biopsy

Bx_DayEmbryo_Morphology

ExpansionICMTE

GenderClinical_Outcome

BEST_ EMBRYO_FOR_ETELECTIVE_SET

Number_of_tarnsfers_to_deliveryBiopsy tech

CYCLE #PEAK E2TVA MD

TVA TECH# Follicles >12 mm

# EGGS# INSEM

# 2PN % FERT

# UNFERT#M2 or mature

# INT# IMM# ATR

# > 2PN# 1PN# DEG

FERT CK TECHICSI TECH

SEMEN SOURCEFRESH/FROZEN SP

CLEAVED% CLEAVED

HATCH TECH# EXT CULTURE

# GOOD EXT CULT# TO BLAST

# CRYO% OF GOOD QUALITY EMBRYOS

…

clinicalBMI

PRIMARY_DXPATIENTTYPETEXT

LUPRONSTIM

GNRHAMED1

SUMSTIMTRANSFER_DATE

HCG_DRUGGRAVIDITY

PREMTERMSAB

BIOCHEMICALPATIENTRACE

LIFETIME_SMOKEDSMOKING_FREQ

PRIORIVFPRIORFETPRIORIUIHEIGHTWEIGHT

STIMPROTOCOLLUPRONPROTOCOLPRIMARYDIAGNOSIS

SECONDARYDIAGNOSISTERTIARYDIAGNOSIS

SEMENSOURCEPATIENTTYPE

FSHLEVELE2LEVEL

NEAREST_AMHAFC

MED1MED2MED3MED4

MAX_E2TOTALIUS

FERT_METHOD_ICSIFERT_METHOD_IVF

INITIALCONSULT_PREMINITIALCONSULT_GRAVIDITY

INITIALCONSULT_SABINITIALCONSULT_TERM

INITIALCONSULT_BIOCHEMICALStim protocol

…

381 variables per SET:

Lab factors, 918 SETs

Pregnant, %Non-Pregnant, %

% of total SETs

Yes No

Lab + Clinical, 918 SETs

Relevant feature selection algorithm* (Lab factors)

*Number of CART trees = 100

Relevant feature selection algorithm* (Lab + Clinical)

*Number of CART trees = 100

Building the model to predict IVF outcome

Only weak predictors are present

Relatively small sample size (10K)

A lot of features (>300)

Accuracy of predictions = 0.73AUC = 0.76 (Sensitivity/specificity balance)

Building the model to predict IVF outcome(PGT only)

• Benchmark AUC – Starting point• Feature engineering• Feature importance• Feature transformations• Non-important features• Model interpretation

Building the model to predict IVF outcome(FETs only)

Relative Importance

Feature Description

0.95784403_NumCatTE_Prior full term_Prior pre-term_TE_0

Out-of-fold mean of the response grouped by: ['Prior full term', 'Prior pre-term', 'TE'] using 5 folds (numeric columns are bucketed into 25 equally populated bins)

0.55907

164_CV_TE_# EXT CULTURE_FACNAME_LUPRON_PGD.1_Retrieval MD_Retrievaltechnician_Thawingtechnician_0

Out-of-fold mean of the response grouped by: ['# EXT CULTURE', 'FACNAME', 'LUPRON', 'PGD.1', 'Retrieval MD', 'Retrieval technician', 'Thawing technician'] using 5 folds

0.35233 217_BIOCHEMICAL BIOCHEMICAL (original)

Ongoing PR after SET with different blastocyst morphology (918 SETs)

Blastocyst morphology

AA AB BA BB B-/-B p-Value

Total SETs 266 292 33 232 95 n/a

Positive hCG 222 240 26 178 61 n/a

Negative hCG 44 52 7 54 34 n/a

Biochemical 25 23 1 36 16 n/a

Miscarriages 18 17 3 14 6 n/a

Ongoing PR per ET, % 67.3 68.5 66.7 55.2 41.1 p<0.05

Birth outcomes (2013-2015)

107 135 8 117 61 n/a

Live births 66 82 4 56 26 n/a

Live birth rate, % 61.7 60.7 50.0 47.9 42.6 p<0.05

http://www.ivfbigdata.com/pgt-calculator/

eSET FUTURESET vs DET in PGS cycles (2013-2016)

ETx1 ETx2 P-value

Total FETs 569 89

Positive HCG 442 78

Negative HCG 127 11

Ongoing pregnancies 335 66

Ongoing PR, % 58.9% 74.2% p<0.00599

Live birth rate,% 53.5% 71.6% p<0.00523

Twins 3 33 (1 triplet)

Twin rate 0.9% 50.0% p<0.00001

Presented by RSC team at ASRM 2016

The 5 Steps Towards Evidence Based Practice

1. Ask the right clinical question: Formulate a searchable question

2. Collect the most relevant publications: Efficient Literature SearchingSelect the appropriate & relevant studies

3. Critically appraise and synthesize the evidence.4. Integrate best evidence with personal clinical expertise, patient preferences and values:

Applying the result to your clinical practice and patient.5. Evaluate the practice decision or change:

Evaluating the outcomes of the applied evidence in your practice or patient.

The 5 Steps Towards Evidence Based Practice

1. Ask the right clinical question: Formulate a searchable question

2. Collect the most relevant DATA:

Efficient Literature SearchingSelect the appropriate & relevant studies

3. Critically appraise and synthesize the evidence.4. Integrate best evidence with personal clinical expertise, patient preferences and values:

Applying the result to your clinical practice and patient.5. Evaluate the practice decision or change:

Evaluating the outcomes of the applied evidence in your practice or patient.

The current problem with the models: A vs B

Conclusion

1. Machine learning is not yet widely used in clinical practice

2. Augmented decision making with machine learning

3. Auto ML for rapid experimentation knowledge discovery

Thank you!

Lab:

K. A. Ivani, Ph.D.

O. O. Barash, Ph.D.

N. Huen

S. C. Lefko

C. MacKenzie

J. Ciolkosz

E. Homen

E. Jaramillo

MDs:

L. N. Weckstein

S. P. Willman

M. R. Hinckley

D. S. Wachs

E. M. Rosenbluth

S. P. Reid

M. V. Homer

E. I. Lewis