Open Access2009Schüffleret al.Volume 10, Issue 10, Article R105SoftwareMethMarker: user-friendly design and optimization of gene-specific DNA methylation assaysPeter Schüffler*, Thomas Mikeska†, Andreas Waha‡, Thomas Lengauer* and Christoph Bock*

Addresses: *Max-Planck-Institut für Informatik, Campus E1.4, 66123 Saarbrücken, Germany. †Molecular Pathology Research and Development Laboratory, Department of Pathology, Peter MacCallum Cancer Centre, A'Beckett Street, Melbourne, Victoria 8006, Australia. ‡Department of Neuropathology, University of Bonn Medical Centre, Sigmund-Freud-Straße, 53105 Bonn, Germany.

Correspondence: Christoph Bock. Email: cbock@mpi-inf.mpg.de

© 2009 Schüffler et al.; licensee BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.MethMarker<p>A software workflow to translate known differentially methylated regions into clinical biomarkers</p>

Abstract

DNA methylation is a key mechanism of epigenetic regulation that is frequently altered in diseasessuch as cancer. To confirm the biological or clinical relevance of such changes, gene-specific DNAmethylation changes need to be validated in multiple samples. We have developed the MethMarkerhttp://methmarker.mpi-inf.mpg.de/ software to help design robust and cost-efficient DNAmethylation assays for six widely used methods. Furthermore, MethMarker implements abioinformatic workflow for transforming disease-specific differentially methylated genomic regionsinto robust clinical biomarkers.

RationaleAberrant DNA methylation is a common event in many can-cers [1,2]. Functionally, cancer-specific hypermethylationimposes condensed chromatin structure upon CpG islandsthat normally exhibit an open and transcriptionally compe-tent chromatin structure [3]. This epigenetic alterationresults in loss of expression at nearby genes, contributing tocancer development when tumor suppressor genes areaffected [4].

For many years, research in cancer epigenetics has focused onthe use of CpG island hypermethylation events of certaingenes as cancer biomarkers, with the aim of improving cancertreatment through more accurate diagnosis, prognosis andtherapy selection [5,6]. Early diagnosis exploits the fact thatCpG island hypermethylation of cancer-related genes is fre-quently detectable in early-stage tumors [7], for which surgi-cal treatment can be highly effective. Prognosis of clinical

outcome uses DNA hypermethylation events to infer whetheror not a tumor is likely to constitute a major threat to thepatient's health, which is particularly relevant for cancers thatwill kill only a subset of patients if left untreated (for example,prostate cancer). Therapy optimization makes use of DNAmethylation differences between patient subgroups in orderto select the most effective treatment, thus contributing topersonalized cancer treatment.

In spite of significant investment in genome-wide screeningand subsequent validation studies, few DNA methylationbiomarkers have been confirmed by clinical trials. This bot-tleneck in the process of translating basic research findingsinto the clinic is partially due to a discontinuity of methodsbetween the discovery phase and the validation phase. Themethods used most commonly in the discovery phase (such astiling microarray and clonal bisulfite sequencing) are tootime-consuming and expensive to be used in the clinical set-

Published: 5 October 2009

Genome Biology 2009, 10:R105 (doi:10.1186/gb-2009-10-10-r105)

Received: 23 March 2009Revised: 19 August 2009Accepted: 5 October 2009

The electronic version of this article is the complete one and can be found online at http://genomebiology.com/2009/10/10/R105

Genome Biology 2009, 10:R105

http://genomebiology.com/2009/10/10/R105 Genome Biology 2009, Volume 10, Issue 10, Article R105 Schüffler et al. R105.2

ting. Hence, candidate biomarkers have to be adapted to highsample-throughput methods such as MethyLight [8],bisulfite pyrosequencing [9-11], COBRA (combined bisulfiterestriction analysis) [12] or bisulfite single nucleotide primerextension (SNuPE) [13,14]. To be effective, this adaptationstep requires substantial bioinformatic optimization and val-idation.

Based on our experience from a pilot study on the O6-methyl-guanine DNA methyltransferase (MGMT) gene [15], we havedeveloped a systematic workflow for design, optimization andvalidation of DNA methylation biomarkers (reviewed in [16]).The six-step procedure outlined in Figure 1 starts from apreselected differentially methylated region (DMR), whichmay have been identified by genome-wide screening experi-ments or through a candidate gene approach. A typical exam-ple would be a CpG island that overlaps with the promoter

region of a tumor suppressor gene. In the first step, thisregion is subjected to high-resolution analysis of DNA meth-ylation in a small number of cases and controls (for example,by clonal bisulfite sequencing). These experimental data pro-vide MethMarker with a representative map of methylationstate within the DMR and inform all subsequent optimizationsteps. Second, using sets of expert rules, technically feasibleDNA methylation assays are designed for each of six robustand cost-efficient experimental protocols (COBRA, bisulfiteSNuPE, bisulfite pyrosequencing, MethyLight, methylation-specific polymerase chain reaction (MSP) and methylatedDNA immunoprecipitation quantitative PCR (MeDIP-qPCR)). Third, the accuracy of all designed assays is compu-tationally assessed, using the DNA methylation map derivedin the first step. Fourth, the most promising candidatebiomarkers are statistically optimized for maximum discrim-ination between cases and controls. Fifth, to reduce the riskthat candidate biomarkers subsequently fail due to technicalproblems or lack of robustness, all high-scoring assays arevalidated with respect to their susceptibility to experimentalnoise, measurement errors and unknown single nucleotidepolymorphisms. Sixth, the most promising assay is selected,experimentally tested and further optimized based on theoutcome of the experimental validation. After completion ofthese six steps, the candidate biomarker is ready for applica-tion and further validation in clinical studies.

Apart from two key experimental analyses - the generation ofhigh-resolution DNA methylation data in step one and assayvalidation in step six - this workflow is essentially bioinfor-matic in nature. We developed the MethMarker software as auser-friendly implementation of the bioinformatic steps,including automatic assay design for six widely used experi-mental methods (COBRA, bisulfite SNuPE, bisulfite pyrose-quencing, MethyLight, MSP and MeDIP-qPCR) andcomputational biomarker optimization. MethMarker inte-grates well with existing bioinformatic tools for analyzingDNA methylation (reviewed in [17]): epigenome analysistools such as Galaxy [18] and EpiGRAPH [19] can be used toselect promising DMRs for optimization with MethMarker,and high-resolution DNA methylation data can be importeddirectly from three widely used software packages, BiQ Ana-lyzer [20], QUMA [21] and EpiTYPER [22], as well as fromcustom tables. Finally, optimized biomarkers can be exportedin the standardized predictive model markup language(PMML) format [23], which facilitates interoperation withmolecular diagnostics software. A typical screenshot of Meth-Marker is displayed in Figure 2.

ApplicationTo illustrate the biomarker development workflow outlinedin Figure 1 and to demonstrate the practical use of Meth-Marker, we describe its application to the MGMT gene pro-moter, highlighting important decisions, necessary validationexperiments and potential stumbling blocks. The raw data for

MethMarker employs a six-step workflow to design, optimize and validate DNA methylation biomarkers for a given differentially methylated DNA region (DMR)Figure 1MethMarker employs a six-step workflow to design, optimize and validate DNA methylation biomarkers for a given differentially methylated DNA region (DMR). In addition to its main purpose as a full-scale biomarker development tool, MethMarker can also be used simply as an assay design software, in which case steps 3 to 6 (yellow boxes) are omitted.

� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� ! " # " ! $ % & " # ' ( " � ) * ! " ( + , , * # & " - , " . * ( " # / 0 ' . + 1 + # ! * ! + / " & " # " + , , . ' + 1 2� 3 ' . " - + ( , 4 " 5 / 2 " , . ' ( ' / " . . " & * ' # ' 6 + / 7 ( ' . 0 7 , , . " 0 0 ' . & " # " 5 2 + . $ ' . * # & + 8 , 9

* 0 4 + # ! 1 4 ' 0 " / ' * / 0 / . + # 0 1 . * , ' # 0 / + . / 0 * / "

: ; < = > = ? @ A B C > = D E F ? G H = I ? J = B DK L M N O P Q R M S R T Q M U P V W T R P M Q M S P Q X T W T Y X Z [ \ ] ^ ] _ ` a b [ c d e c b _ e \ ^ ^ a b d _ f g h i j ] \ f h i k l m n o m pK q T X W P T r N s M S W T s T r N Q X N Q Q M X N M Q P Q S M W U N M Q Z t h c u i d _ b i _ d ` b a [ d v _ w x n x y _ i a [ _ z b a { e _ b pK | } M Q N s ~ � U } M W X M S � � � � � � � � � � � � � � � � Z � c e _ f a i z h � m i c ] � � _ b g � w � m a b � ^ h o � � � � a \ d ^ \ d ] _ e p

� ; < � G ? � D B � < � � @ � > � � I = B D = G G = � G � B C > � � C � � ? B D B � ? D > � C � G >K � � } } M W X T O U T X   M O Y ~ ¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ­ ® ¯ ª ° ± ¦ § ¨ © ª « ¬ ­ ² ³ ´ µ © ­ ¶ ª ­ · ¸ ¨ · ¹ ¦ º ® ° ¦ º ­ ¬ » ³ ¼ ¨ ¹ » ¬ ¦ º ­ ½ ¾ ° ¿ ¶ ° ¡ ¤K À Q M Á Q � Â Ã Y N W T N � X M U N V N s s Ä N r M P O T O X M P Q V W T N Y T N Y Y N Ä W T s P N Å P s P X Ä

Æ ; Ç F = I H = B D B � = G G = � A � C � B C @ = D È � É = G � E B D < � � @ � > � � I = B D A C B I � GK Ê Ë Ì Í Î Í Ï Ð } W T O P V M Q M S Ñ Â Ò U T X   Ä s N M Q W T N O K M � X S M W T N V   N Y Y N Ä N Q O X W N P Q P Q R Y N U } s TK q N Q Ó P Q R Å Ä X   T Ô � Õ Õ � � � � � Ö � � × � � � Ø Ù � Ú � � � � � � � � Ø Ú Õ � � � � � � � � � Û � Õ � � � Ø � � Ü � � Ø Ô � Õ �K � T s T V M Q M S   P R   K Y V M W P Q R N Y Y N Ä V M U Å P Q N M Q Y S M W Å P M U N W Ó T W M } U P Ý N M Q

Þ ; ß A @ ? J = B D B � < � � @ � > � � I = B D É ? B @ = C à � C GK � s N Y Y P V N M Q M S Y N U } s T Y P Q X M V N Y T Y N Q O V M Q X W M s YK � W N P Q P Q R M S � � � Ù Ø Ô Õ � � Õ � Ø Ø Ù � � � � � � � Ø S M W } W T O P V Q R X   T Y N U } s T V s N Y YK Ã T W S M W U N Q V T T r N s � N M Q Å Ä s T N r T K M Q T K M � X V W M Y Y K r N s P O N M Q

� ; � B @ A H > = B D = I F = I ? E = B DK q M Å � Y X Q T Y Y N Y Y T Y Y U T Q X Å Ä Y P U � s N Q R T W W M Q T M � Y O N X NK � N s V � s N M Q M S Y T Q Y P r P X Ä � Y } T V P r P X Ä � N V V � W N V Ä N Q O V M W W T s N M Q

� ; Ç � A � C ? @ � D > = I F = I ? E = B D = D E C � B A @ ? J = B DK � Ø Ø � � Û � � Ù � � � � Å N Y T O M Q T � } T W P U T Q X N s O N X N S M W X   T O T Y P R Q T O N Y Y N ÄK q T Q R M S X   T W T R W T Y Y P M Q U M O T s Å N Y T O M Q X   T T � } T W P U T Q X N s O N X N

� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� Ç � A B C > ? D G > = D E = C E ? J � E � � � � B C @ = >� � A A I ? È = B D B D D � � G = @ A I � G � È I = G G ? � � ? D � > � � @ ? D > B � È = G � G � = D E � È B D > C B I G �

Genome Biology 2009, 10:R105

http://genomebiology.com/2009/10/10/R105 Genome Biology 2009, Volume 10, Issue 10, Article R105 Schüffler et al. R105.3

this case study are taken from a recent experimental study[15] and are included as a demonstration dataset in the Meth-Marker download package.

The MGMT gene encodes a DNA repair protein, whichremoves alkyl groups from the O6-position of guanine, there-fore protecting the DNA from accumulating excessive damage[24]. It has been shown in a number of studies (see [25] andreferences therein) that hypermethylation of the MGMT pro-moter is a frequent event in various cancers (that is, is rele-vant for diagnosis), that it is associated with decreasedsurvival if the cancer is untreated (that is, is relevant for prog-nosis), and that it renders tumors susceptible to alkylatingdrugs such as temozolomide (that is, is also relevant for ther-apy optimization). However, until recently no assay for meas-uring MGMT promoter methylation had been available thatwas robust enough for routine clinical use and fully compati-ble with DNA extracted from formalin-fixed, paraffin-embed-ded samples [26].

For these reasons, the promoter of the MGMT gene is anexcellent target region for demonstrating the systematicdevelopment of a DNA methylation biomarker, such that theresulting assay is accurate, robust and cost-efficient enoughfor clinical use. To start with, we obtain the genomic DNAsequence of the MGMT promoter region from the UCSCGenome Browser [27]. We also obtain 22 glioblastoma sam-ples, a subset of them showing MGMT promoter methylation,as well as three normal brain samples for use as healthy tissuecontrols. Next, bisulfite-specific PCR primers are designed(manually or using a software tool such as Methyl PrimerExpress [28]), and clonal bisulfite sequencing is performedon DNA from all samples according to a widely used protocol[29]. The sequencing data are processed and quality control-led with BiQ Analyzer [20], resulting in 25 high-resolutionDNA methylation profiles that are used as training samples.(Note that it is usually sufficient to have five to ten trainingsamples per class to guide the optimization step. In our case,however, it was not clear a priori how many of the tumor

This figure shows a screenshot of MethMarker's main analysis windowFigure 2This figure shows a screenshot of MethMarker's main analysis window. From top to bottom, MethMarker displays gene annotation data for the region of interest; its genomic DNA sequence as well as the bisulfite converted sequence; automatically generated assays for the supported experimental methods (COBRA, bisulfite SNuPE, bisulfite pyrosequencing, MSP, MethyLight and MeDIP-qPCR); DNA methylation information for the region of interest, which has been loaded into MethMarker (yellow bars correspond to unmethylated CpGs, blue bars to methylated CpGs); a statistical summary of CpG positions within the region of interest; and - at the bottom - a text field providing advice for the user. All views are highly interactive and can be adjusted to control MethMarker's behavior.

Genome Biology 2009, 10:R105

http://genomebiology.com/2009/10/10/R105 Genome Biology 2009, Volume 10, Issue 10, Article R105 Schüffler et al. R105.4

samples would turn out to belong to the methylated cases orto the unmethylated controls, respectively. Hence, a relativelylarge number of samples were subjected to clonal bisulfitesequencing.)

Next, the genome sequence of the target region, the corre-sponding primer sequences and the BiQ-Analyzer processedDNA methylation profiles are imported into MethMarker.The software tool automatically identifies the correct locationof the MGMT promoter on human chromosome 10, visualizesthe position of the first exon and aligns the DNA methylationprofiles of all 25 training samples (Figure 2). We let Meth-Marker classify the training samples into cases and controls,using hierarchical clustering of the DNA methylation profiles.Consistent with previous observations, we obtain a large clus-ter of samples in which the MGMT promoter is unmethylatedand a smaller cluster consisting of tumor samples with meth-ylated MGMT promoters. The former cluster - which we willrefer to as 'controls' - contains the normal brain samples anda subset of tumors that are likely to be resistant to alkylatingagents used for chemotherapy. The latter cluster ('cases')comprises tumor samples only, presumably those that aresusceptible for chemotherapy using alkylating drugs such astemozolomide [30].

Based on this classification, our goal is to find a DNA methyl-ation assay (or a combination of several assays) that providesaccurate, robust and cost-efficient separation between casesand controls. First, we let MethMarker design all feasibleDNA methylation assays for the target region, using COBRA,bisulfite SNuPE, bisulfite pyrosequencing, MethyLight andMeDIP-qPCR. We chose to exclude MSP because severalMSP-based assays for MGMT promoter hypermethylationare already available [26] and because MSP-based assays donot always work well on formalin-fixed, paraffin-embeddedsamples [15]. Next, we let MethMarker score the individualassays in terms of their correlation with the overall DNAmethylation level in each of the training samples (Additionaldata file 1). A Pearson correlation coefficient above 0.9 and aSpearman correlation coefficient above 0.8 indicate a highlyaccurate and predictive assay. Even when a single CpG sitealready provides a highly accurate measurement - as is thecase here - it is highly recommended to use a combination ofat least three to four CpG sites in order to increase robustnessof the DNA methylation assay in the presence of experimentalnoise and rare sequence polymorphisms. To that end, Meth-Marker identifies the optimal combinations of DNA methyla-tion assays for each method, again ranked by their correlationwith the overall DNA methylation level in each of the trainingsamples (Additional data file 1).

From the resulting list, we select several assay combinationsthat appear to provide a suitable balance between accuracy,robustness and cost (higher robustness is usually achieved byincluding more CpG sites, which makes the candidatebiomarker more expensive to use). For each of these assay

combinations, we let MethMarker optimize logistic regres-sion models that predict whether a sample belongs to thecases or to the controls (Figure 3). During this step, weightsare learned for the individual assays in order to maximize theclassification accuracy of the candidate biomarker. Meth-Marker benchmarks the candidate biomarkers in terms ofaccuracy, correlation, specificity and sensitivity. Additionally,the biomarkers' robustness is assessed by comparing falsepositive and false negative rates under increasing error rate,by simulating noisy measurement data. This step accounts forthe fact that not all error sources may be well-represented inthe training data. For example, COBRA, bisulfite SNuPE andbisulfite pyrosequencing are sensitive to rare inherited C-to-T single nucleotide polymorphisms at the assayed CpGs, andMSP as well as MethyLight can give rise to erroneous meas-urements if the DNA methylation profile in the target regiononly partially matches with the designed probe (see Mikeskaet al. [15] and Bock et al. [20] for a more in-depth discussionof potential error sources).

For each candidate biomarker, MethMarker also calculatesan extensive performance evaluation summary (Figure 4).We use the results from this window to compare how well sev-eral top-scoring candidate biomarkers separate between themethylated cases and unmethylated controls. Also, we testthe robustness of each candidate biomarker by artificiallyintroducing noise and observing how much noise it can toler-ate until the first classification errors start to appear. As aquintessence of all performance evaluations of MethMarker,we conclude that the following two candidate biomarkers aremost suitable for assessing promoter hypermethylation of theMGMT gene in routine clinical use: the COBRA biomarkercomprising CpG sites 5/6 and 18, utilizing the Hpy99I andHpyCH4III restriction endonucleases (r = 0.985), and thebisulfite pyrosequencing biomarker comprising CpG sites 13,18 and 20 (r = 0.990). Both biomarkers achieve 100% test setaccuracy during leave-one-out cross-validation. Compared tothe biomarkers that we previously established for the samedataset [15], the biomarkers identified by MethMarkerachieve an identical accuracy and score marginally higher interms of correlation and robustness (data not shown). Never-theless, we recommend that practical studies of MGMT pro-moter methylation continue using the previously publishedbiomarkers [15] because they have been validated experimen-tally, while the two MethMarker-derived biomarkersreported here have not been tested on clinical samples.

Having completed the design, optimization and computa-tional validation of candidate biomarker DNA methylationassay for the MGMT promoter, two key steps remain: experi-mental assay validation and experimental biomarker valida-tion. First, it is essential to make sure that the DNAmethylation assays included in the selected biomarker workwell in the lab and result in roughly the same DNA methyla-tion measurements as predicted based on the high-resolutionDNA methylation profiles. To that end, the assays are applied

Genome Biology 2009, 10:R105

http://genomebiology.com/2009/10/10/R105 Genome Biology 2009, Volume 10, Issue 10, Article R105 Schüffler et al. R105.5

to DNA from the training samples, and each assay's empiricalmeasurement value is compared with the simulated measure-ment value that MethMarker calculated from the high-resolu-tion profiles. Assays showing low correlation or highdeviation should be rejected from practical use as biomark-ers. Second, the most important step for any new DNA meth-ylation biomarker is to validate its sensitivity, specificity andpractical utility in a large number of patients, both by retro-spective studies based on archival material with known clini-cal history and in prospective clinical trials. While severalclinical trials have already confirmed the effect of MGMThypermethylation on chemotherapy resistance in gliomas[31] and glioblastomas [30,32], the MethMarker-optimizedbiomarker may facilitate the clinical confirmation of MGMT'spredictive role in other cancers.

ConclusionsRecent advances in genome-wide DNA methylation mappinghave provided researchers with rapid and cost-efficient waysto contribute to the ever-growing list of genomic regionsreported as differentially methylated in specific cancers and/or patient subgroups. However, a comparable advance for theefficient conversion of DMRs into clinical biomarkers is lack-ing. Thus, the rate with which new DNA methylation biomar-kers are tested and confirmed in clinical trials has remaineddisappointingly low. While it is inevitable that a large per-centage of candidate biomarkers will fail in clinical trials(either because they are not reproducible in different patientcohorts or because their sensitivity and specificity are insuffi-cient for practical use), a more systematic approach to epige-netic biomarker development could help discard many of

This figure displays a screenshot of MethMarker's biomarker performance comparison, assessing the robustness of candidate biomarkers to elevated error ratesFigure 3This figure displays a screenshot of MethMarker's biomarker performance comparison, assessing the robustness of candidate biomarkers to elevated error rates. In this example, CO_30 and CO_16 exhibit the overall best performance, in terms of low false positive/negative rates as well as high levels of accuracy, sensitivity, specificity and correlation.

Genome Biology 2009, 10:R105

http://genomebiology.com/2009/10/10/R105 Genome Biology 2009, Volume 10, Issue 10, Article R105 Schüffler et al. R105.6

these unsuccessful candidates early and at low cost. Con-versely, careful selection and optimization of candidatebiomarkers can reduce the risk of losing effective biomarkersdue to contingencies of the validation process, such as acci-dental selection of DNA methylation assays that measurehighly noisy CpG positions in a promoter region that wouldotherwise provide reliable classification. The workflowdescribed in this paper provides a starting point toward amore systematic way of transforming disease-specific DMRsinto robust and cost-efficient clinical biomarkers. The Meth-Marker software was developed to facilitate the implementa-

tion of this workflow. To enable further refinement andadaptation to local requirements, we are happy to shareMethMarker's source code with interested researchers.

Materials and methodsMethMarker is implemented in Java (version 1.5 or laterrequired). It is platform-independent and can be launcheddirectly from within a web browser. The software comes witha case-study tutorial demonstrating the design, optimizationand validation of a DNA methylation biomarker based on the

A screenshot of MethMarker's performance window, summarizing the evaluation of a bisulfite pyrosequencing-based biomarkerFigure 4A screenshot of MethMarker's performance window, summarizing the evaluation of a bisulfite pyrosequencing-based biomarker. In the upper panel, MethMarker displays the optimized regression formula, which predicts - based on measurement values for CpGs number 5 and 18 - whether a sample belongs to the case (that is, is methylated, indicated by positive score values) or to the control group (that is, is unmethylated, indicated by negative score values). Note that the score value is a measure of the probability with which the sample is a case rather than a control, not an estimate of

the DNA methylation level (in fact, the probability p can be calculated from the score s with a simple formula: . The center panel displays the

results of leave-one-out cross-validation, providing an estimate of the biomarker performance on new data. The diagrams at the bottom visualize the degree of separation between the two classes when plotting the measured level of DNA methylation over the score value of the regression formula (left) and the robustness of predictions in the face of increasing noise levels (right).

pe s

=+ −

1

1

Genome Biology 2009, 10:R105

http://genomebiology.com/2009/10/10/R105 Genome Biology 2009, Volume 10, Issue 10, Article R105 Schüffler et al. R105.7

MGMT gene. MethMarker's user interface reflects the work-flow for biomarker design, optimization and validation out-lined in Figure 1.

Step 1: data importAs the first step, the DMR of interest is imported. Meth-Marker supports several sequence formats, including FASTA,GenBank and EMBL. Typical regions of interest include thepromoters of tumor suppressor genes and CpG islands thatexhibit cancer-specific hypermethylation. However, Meth-Marker imposes no restrictions on the type of region to beanalyzed. MethMarker can thus be applied not only to humancancers, but more generally to epigenotyping in all kinds oforganisms that exhibit CpG dinucleotide methylation.

High-resolution DNA methylation profiles for a subset ofcases and controls are crucial for MethMarker's optimizationprocess, as they provide the training set on which all candi-date biomarkers are optimized and computationally vali-dated. These profiles are usually derived by clonal bisulfitesequencing [33] or mass spectrometry and preprocessed withappropriate tools. MethMarker can directly import DNAmethylation profiles from files generated with BiQ Analyzer[20], QUMA [21] and EpiTYPER [22], and it is easy to convertDNA methylation data from a different source into a formatthat can be read by MethMarker.

On completion of data import, MethMarker displays a high-resolution DNA methylation profile of the region of interest,visualized as lollipop diagrams or as methylation propensitydiagrams. Internally, MethMarker uses Needleman-Wunschsequence alignment [34] in order to correct for incompleteoverlap between the target region and the DNA methylationprofiles. It is thus possible to tile a large target region withseveral bisulfite sequencing amplicons.

Optionally, MethMarker can annotate the region with tran-scription start site and exon positions retrieved from theUCSC Genome Browser [27]. To that end, MethMarker per-forms an automatic BLAT search on the UCSC GenomeBrowser website, obtains the genomic coordinates of theregion and retrieves exon information for overlapping Ref-Gene genes from the UCSC Table Browser. Data on singlenucleotide polymorphisms are acquired in the same way, ena-bling MethMarker to avoid polymorphic sites when designingDNA methylation assays. All annotation data can be manuallyrevised and amended.

Step 2: design of DNA methylation assaysMethMarker implements automatic assay design for sixexperimental methods commonly used for DNA methylationanalysis: COBRA, bisulfite SNuPE, bisulfite pyrosequencing,MethyLight, MSP and MeDIP-qPCR. The first five methodsutilize bisulfite treatment of genomic DNA to detect DNAmethylation indirectly. However, they differ in the way theyinterrogate the amount of DNA methylation, leading to spe-

cific experimental constraints that limit the application ofeach method to assaying a subset of CpG positions. The sixthmethod, MeDIP-qPCR, uses an antibody-based approach toenrich for methylated genomic DNA, which leads to quite dif-ferent experimental constraints [35]. For all methods, assaydesign rules were developed, reviewed by domain experts andimplemented in MethMarker, as described in more detail inthe MethMarker assay design dialogue. However, it is recom-mended that all primers designed with MethMarker arereviewed by the experimenter before ordering, to excludeproblems such as hairpins, self-dimers and cross-dimers,which MethMarker does not automatically check for.

All automatically designed DNA methylation assays can bevisualized, revised or excluded by the user, for example, basedon results of previous experiments. Furthermore, Meth-Marker allows users to define and incorporate custom assays,which enables the software to include experimental methodsthat are not directly supported.

Step 3: scoring of DNA methylation assaysBased on the samples for which high-resolution DNA methyl-ation profiles are available (see step 1), MethMarker scores allDNA methylation assays in terms of their correlation with theoverall level of DNA methylation in each sample. The meas-urement values of the DNA methylation assays are calculateddirectly from the high-resolution DNA methylation profiles,using a set of method-specific rules. For COBRA, bisulfiteSNuPE and bisulfite pyrosequencing, the measurement valueis calculated simply as the average DNA methylation level ofthe assayed CpG site(s), based on the high-resolution DNAmethylation profiles of the respective sample. For MSP,MethyLight and MeDIP-qPCR, the measurement value is cal-culated as the percentage of individual clones in which all par-ticipating CpG sites are simultaneously methylated. To betterresemble real PCR conditions, for MSP and MethyLight a sin-gle CpG position is allowed to have an incorrect methylationvalue. While simulated measurements cannot replace experi-mental validation of the resulting DNA methylation assays(see [36] for a discussion of the limitations of simulating DNAmethylation measurements in silico), they provide a suitableindication for identifying the most predictive DNA methyla-tion assays to be included in the optimization step.

Step 4: biomarker optimizationFrom the list of DNA methylation assays, ranked by their cor-relation with the overall DNA methylation levels of the train-ing samples, the user can select a subset for biomarkeroptimization. MethMarker then scores all possible combina-tions of the selected DNA methylation assays and againassesses the correlation with the overall DNA methylationlevels of the training samples. To allow for fair comparisonbetween assay sets of different sizes, no weight fitting is per-formed at this stage. Rather, the score value of each combina-tion is calculated as the mean measurement value of allcontributing DNA methylation assays. The results of this

Genome Biology 2009, 10:R105

http://genomebiology.com/2009/10/10/R105 Genome Biology 2009, Volume 10, Issue 10, Article R105 Schüffler et al. R105.8

comparison are listed in the order of decreasing correlationcoefficients, and the user can select a subset of the mosthighly scoring combinations of DNA methylation assays foroptimization and computational validation as candidatebiomarkers, a procedure that is performed as follows.

First, the training samples are classified into cases and con-trols. This classification can be performed based on knownsample information (for example, tumor samples versus nor-mal tissue annotation) or based on the DNA methylation pro-files themselves, using one of the following methods: a fixedthreshold on the average DNA methylation level, hierarchicalclustering, or K-means clustering with K = 2. In all cases, theDNA methylation profiles in the subset with the higher aver-age methylation levels are labeled as methylated 'cases' andthe remaining profiles are labeled as unmethylated 'controls'.

Second, logistic regression is used to optimize the weight withwhich the individual measurements contribute to the overallbiomarker score, accounting for the fact that different CpGsvary in their predictiveness of the overall level of DNA meth-ylation. Internally, MethMarker uses the WEKA package [37]to train a logistic regression model for each candidatebiomarker, classifying the training samples into cases versuscontrols based on simulated methylation measurements forall contributing CpGs.

Third, the predictiveness of the logistic regression models isvalidated by leave-one-out cross-validation - that is, the logis-tic regression models are repeatedly trained on all but onetraining samples and their prediction performance isassessed on the remaining sample. The results of the optimi-zation step, including a cross-validation-based estimate of theprediction performance on new data, are displayed in thebiomarker summary window (Figure 4).

Step 5: validation of DNA methylation biomarkersWhile the results of the leave-one-out cross-validation (step4) already provide an important selection criterion for identi-fying the most suitable DNA methylation biomarkers, they donot account for potential errors and experimental problemsthat can occur during practical use. MethMarker thereforeprovides an additional validation step, which assesses therobustness of each candidate biomarker toward noisy data,sequencing errors and unknown single nucleotide polymor-phisms. In this step, the optimal logistic regression model isre-applied to all samples for which high-resolution DNAmethylation profiles are available (this can include samplesthat were not taken into account in the training phase - forexample, because they constitute outliers or borderlinecases), and the biomarker's prediction confidence for a givensample is plotted against its mean DNA methylation level, ascalculated from the DNA methylation profiles. It is thus pos-sible to visually assess how well each candidate biomarkerseparates between the (methylated) cases and (unmethyl-ated) controls. Furthermore, MethMarker assesses the

robustness toward erroneous data - such as sequencing errorsor unknown single nucleotide polymorphisms - by randomlychanging the DNA methylation measurement of a subset ofCpGs. The error rate is varied over a wide range, and theimpact on the prediction accuracy is visualized in the biomar-ker summary window (Figure 4), enabling the user to assesswhether or not a specific candidate biomarker is sufficientlyrobust for clinical use.

Step 6: application of DNA methylation biomarkersBased on the results of the computational assessment, theuser selects a few of the most promising biomarkers forexperimental validation, performs the necessary DNA meth-ylation assays on DNA from the training samples and uploadsthe results into MethMarker. By comparison between thesimulated and actual measurements, MethMarker can evalu-ate the reliability of each candidate biomarker under routineexperimental conditions and re-train its logistic regressionmodels accordingly (for example, down-weighting the contri-bution of a CpG whose DNA methylation assay exhibits a highlevel of experimental noise). This experimental validationstep is important because it corrects for any deviations fromthe theoretically optimal measurement conditions thatunderlie the computational simulation of measurement val-ues.

When the optimization and validation steps are completedand the user is satisfied with the overall performance, one ormore candidate biomarkers are typically selected for furtherdevelopment. MethMarker provides two ways of facilitatingthe steps toward comprehensive clinical testing and wide-spread practical use. First, MethMarker can generate a com-prehensive PDF report describing the key properties of aselected biomarker. This report includes the final sampleclassification formula as well as a summary of the accuracyand robustness assessment. Based on this file, it is straight-forward to apply the biomarker assay to new data, requiringno statistical or bioinformatic tools beyond a pocket calcula-tor. Second, a selected biomarker can be exported in a stand-ardized data format, PMML, which is supported by severalstatistics packages and can be imported into diagnostics soft-ware. PMML has been developed by the Data Mining Group[23] to facilitate data exchange between developers and usersof classification and regression models. All classifiers createdwith MethMarker fulfill the PMML 3.2 standard (see Addi-tional data file 2 for illustration). Third, MethMarker sup-ports multi-center biomarker validation studies. To that end,the PDF and PMML documentation files of the selectedbiomarker are distributed to all participating centers; eachcenter then performs the necessary DNA methylation assaysfor all local samples, loads the PMML file and the measure-ment values into MethMarker and obtains the biomarkerresult for each of their samples; finally, the measurement val-ues from all centers as well as the corresponding clinical dataare combined, loaded into MethMarker and a global assess-ment of biomarker performance is obtained. If the perform-

Genome Biology 2009, 10:R105

http://genomebiology.com/2009/10/10/R105 Genome Biology 2009, Volume 10, Issue 10, Article R105 Schüffler et al. R105.9

ance is not satisfactory, the entire process can be reiteratedand the biomarker re-optimized based on the data obtained inthe previous round of validations.

AbbreviationsCOBRA: combined bisulfite restriction analysis; DMR: differ-entially methylated region; MeDIP-qPCR: methylated DNAimmunoprecipitation quantitative PCR; MGMT: O6-methyl-guanine DNA methyltransferase; MSP: methylation-specificpolymerase chain reaction; PMML: predictive model markuplanguage; SNuPE: single-nucleotide primer extension.

Competing interestsThe authors declare that they have no competing interests.

Authors' contributionsCB initiated the project and conceptualized workflow andsoftware. PS designed and implemented MethMarker, devel-oped the case study tutorial, set up the website and draftedthe paper. TM devised the assay design rules and contributedhis experience with COBRA, bisulfite SNuPE, bisulfite Pyro-sequencing, MSP, MethyLight and MeDIP-qPCR. He alsoprovided experimental data and performed extensive betatesting. AW provided experimental data. TL contributedadvice and ideas throughout the project. All authors wereinvolved in the writing of the paper.

Additional data filesThe following additional data are available with the onlineversion of this paper: a screenshot of MethMarker's perform-ance ranking of DNA methylation assays and candidatebiomarkers (Additional data file 1); the XML-based PMMLmodel that MethMarker uses for exporting, importing andstoring candidate biomarkers (Additional data file 2).Additional data file 1Screenshot of MethMarker's performance ranking of DNA methyl-ation assays and candidate biomarkersScreenshot of MethMarker's performance ranking of DNA methyl-ation assays and candidate biomarkers.Click here for fileAdditional data file 2The XML-based PMML model that MethMarker uses for exporting, importing and storing candidate biomarkersThe XML-based PMML model that MethMarker uses for exporting, importing and storing candidate biomarkers.Click here for file

AcknowledgementsWe would like to thank Jörn Walter and Martina Paulsen for interesting dis-cussions, David Thomas for providing EpiTYPER test data, Joachim Büch aswell as Oliver Schönleben for technical support, and Chelsee Hewitt forcritical reading of the manuscript. TM thanks Alexander Dobrovic for hiscontinued support. Furthermore, we acknowledge advice by severalresearchers involved in the EU Network of Excellence 'The Epigenome',which helped us devise the expert rules for assay design and validation. Thiswork was partially funded by the European Union through the CANCER-DIP project (HEALTH-F2-2007-200620) and by the German Federal Minis-try of Education and Research through the NGFN-Plus Brain TumorNetwork (01GS08187).

References1. Laird PW: Cancer epigenetics. Hum Mol Genet 2005, 14(Spec No

1):R65-76.2. Esteller M: Epigenetics in cancer. N Engl J Med 2008,

358:1148-1159.3. Bock C, Walter J, Paulsen M, Lengauer T: CpG island mapping by

epigenome prediction. PLoS Comput Biol 2007, 3:e110.

4. Baylin SB, Ohm JE: Epigenetic gene silencing in cancer - a mech-anism for early oncogenic pathway addiction? Nat Rev Cancer2006, 6:107-116.

5. Laird PW: The power and the promise of DNA methylationmarkers. Nat Rev Cancer 2003, 3:253-266.

6. Teodoridis JM, Strathdee G, Brown R: Epigenetic silencing medi-ated by CpG island methylation: potential as a therapeutictarget and as a biomarker. Drug Resist Updat 2004, 7:267-278.

7. Feinberg AP, Ohlsson R, Henikoff S: The epigenetic progenitororigin of human cancer. Nat Rev Genet 2006, 7:21-33.

8. Eads CA, Danenberg KD, Kawakami K, Saltz LB, Blake C, Shibata D,Danenberg PV, Laird PW: MethyLight: a high-throughput assayto measure DNA methylation. Nucleic Acids Res 2000, 28:E32.

9. Uhlmann K, Brinckmann A, Toliat MR, Ritter H, Nurnberg P: Evalu-ation of a potential epigenetic biomarker by quantitativemethyl-single nucleotide polymorphism analysis. Electrophore-sis 2002, 23:4072-4079.

10. Colella S, Shen L, Baggerly KA, Issa JP, Krahe R: Sensitive and quan-titative universal pyrosequencing methylation analysis ofCpG sites. Biotechniques 2003, 35:146-150.

11. Tost J, Dunker J, Gut IG: Analysis and quantification of multiplemethylation variable positions in CpG islands by pyrose-quencing. Biotechniques 2003, 35:152-156.

12. Xiong Z, Laird PW: COBRA: a sensitive and quantitative DNAmethylation assay. Nucleic Acids Res 1997, 25:2532-2534.

13. El-Maarri O, Herbiniaux U, Walter J, Oldenburg J: A rapid, quanti-tative, non-radioactive bisulfite-SNuPE-IP RP HPLC assayfor methylation analysis at specific CpG sites. Nucleic Acids Res2002, 30:e25.

14. Gonzalgo ML, Jones PA: Quantitative methylation analysisusing methylation-sensitive single-nucleotide primer exten-sion (Ms-SNuPE). Methods 2002, 27:128-133.

15. Mikeska T, Bock C, El-Maarri O, Hübner A, Ehrentraut D, SchrammJ, Felsberg J, Kahl P, Büttner R, Pietsch T, Waha A: Optimization ofquantitative MGMT promoter methylation analysis usingpyrosequencing and combined bisulfite restriction analysis. JMol Diagn 2007, 9:368-381.

16. Bock C: Epigenetic biomarker development. Epigenomics 2009,1:99-110.

17. Bock C, Lengauer T: Computational epigenetics. Bioinformatics2008, 24:1-10.

18. Blankenberg D, Taylor J, Schenck I, He J, Zhang Y, Ghent M, Veerar-aghavan N, Albert I, Miller W, Makova KD, Hardison RC, NekrutenkoA: A framework for collaborative analysis of ENCODE data:making large-scale analyses biologist-friendly. Genome Res2007, 17:960-964.

19. Bock C, Halachev K, Büch J, Lengauer T: EpiGRAPH: User-friendly software for statistical analysis and prediction of(epi-) genomic data. Genome Biol 2009, 10:R14.

20. Bock C, Reither S, Mikeska T, Paulsen M, Walter J, Lengauer T: BiQAnalyzer: visualization and quality control for DNA methyl-ation data from bisulfite sequencing. Bioinformatics 2005,21:4067-4068.

21. Kumaki Y, Oda M, Okano M: QUMA: quantification tool formethylation analysis. Nucleic Acids Res 2008:W170-175.

22. Ehrich M, Nelson MR, Stanssens P, Zabeau M, Liloglou T, XinarianosG, Cantor CR, Field JK, Boom D van den: Quantitative high-throughput analysis of DNA methylation patterns by base-specific cleavage and mass spectrometry. Proc Natl Acad SciUSA 2005, 102:15785-15790.

23. PMML Version 3.2 [http://www.dmg.org/pmml-v3-2.html]24. Gerson SL: MGMT: its role in cancer aetiology and cancer

therapeutics. Nat Rev Cancer 2004, 4:296-307.25. Jacinto FV, Esteller M: MGMT hypermethylation: a prognostic

foe, a predictive friend. DNA Repair (Amst) 2007, 6:1155-1160.26. Iafrate AJ, Louis DN: "MGMT for pt mgmt": is methylguanine-

DNA methyltransferase testing ready for patient manage-ment? J Mol Diagn 2008, 10:308-310.

27. Karolchik D, Kuhn RM, Baertsch R, Barber GP, Clawson H, DiekhansM, Giardine B, Harte RA, Hinrichs AS, Hsu F, Kober KM, Miller W,Pedersen JS, Pohl A, Raney BJ, Rhead B, Rosenbloom KR, Smith KE,Stanke M, Thakkapallayil A, Trumbower H, Wang T, Zweig AS, Haus-sler D, Kent WJ: The UCSC Genome Browser Database: 2008update. Nucleic Acids Res 2008, 36:D773-779.

28. Methyl Primer Express [http://docs.appliedbiosystems.com/pebiodocs/04370961.pdf]

29. Bisulfite sequencing of small DNA/cell samples (PROT35)[http://www.epigenome-noe.net/researchtools/protocol.php?pro

Genome Biology 2009, 10:R105

http://genomebiology.com/2009/10/10/R105 Genome Biology 2009, Volume 10, Issue 10, Article R105 Schüffler et al. R105.10

tid=35]30. Hegi ME, Diserens AC, Gorlia T, Hamou MF, de Tribolet N, Weller

M, Kros JM, Hainfellner JA, Mason W, Mariani L, Bromberg JE, Hau P,Mirimanoff RO, Cairncross JG, Janzer RC, Stupp R: MGMT genesilencing and benefit from temozolomide in glioblastoma. NEngl J Med 2005, 352:997-1003.

31. Esteller M, Garcia-Foncillas J, Andion E, Goodman SN, Hidalgo OF,Vanaclocha V, Baylin SB, Herman JG: Inactivation of the DNA-repair gene MGMT and the clinical response of gliomas toalkylating agents. N Engl J Med 2000, 343:1350-1354.

32. Hegi ME, Diserens AC, Godard S, Dietrich PY, Regli L, Ostermann S,Otten P, Van Melle G, de Tribolet N, Stupp R: Clinical trial sub-stantiates the predictive value of O-6-methylguanine-DNAmethyltransferase promoter methylation in glioblastomapatients treated with temozolomide. Clin Cancer Res 2004,10:1871-1874.

33. Hajkova P, el-Maarri O, Engemann S, Oswald J, Olek A, Walter J:DNA-methylation analysis by the bisulfite-assisted genomicsequencing method. Methods Mol Biol 2002, 200:143-154.

34. Needleman SB, Wunsch CD: A general method applicable tothe search for similarities in the amino acid sequence of twoproteins. J Mol Biol 1970, 48:443-453.

35. Pelizzola M, Koga Y, Urban AE, Krauthammer M, Weissman S, Hala-ban R, Molinaro AM: MEDME: an experimental and analyticalmethodology for the estimation of DNA methylation levelsbased on microarray derived MeDIP-enrichment. Genome Res2008, 18:1652-1659.

36. Bock C, Walter J, Paulsen M, Lengauer T: Inter-individual varia-tion of DNA methylation and its implications for large-scaleepigenome mapping. Nucleic Acids Res 2008, 36:e55.

37. Frank E, Hall M, Trigg L, Holmes G, Witten IH: Data mining in bio-informatics using Weka. Bioinformatics 2004, 20:2479-2481.

Genome Biology 2009, 10:R105