- Methodology article
- Open Access
An adaptable method using human mixed tissue ratiometric controls for benchmarking performance on gene expression microarrays in clinical laboratories
© Pine et al; licensee BioMed Central Ltd. 2011
- Received: 25 January 2011
- Accepted: 12 April 2011
- Published: 12 April 2011
Molecular biomarkers that are based on mRNA transcripts are being developed for the diagnosis and treatment of a number of diseases. DNA microarrays are one of the primary technologies being used to develop classifiers from gene expression data for clinically relevant outcomes. Microarray assays are highly multiplexed measures of comparative gene expression but have a limited dynamic range of measurement and show compression in fold change detection. To increase the clinical utility of microarrays, assay controls are needed that benchmark performance using metrics that are relevant to the analysis of genomic data generated with biological samples.
Ratiometric controls were prepared from commercial sources of high quality RNA from human tissues with distinctly different expression profiles and mixed in defined ratios. The samples were processed using six different target labeling protocols and replicate datasets were generated on high density gene expression microarrays. The area under the curve from receiver operating characteristic plots was calculated to measure diagnostic performance. The reliable region of the dynamic range was derived from log2 ratio deviation plots made for each dataset. Small but statistically significant differences in diagnostic performance were observed between standardized assays available from the array manufacturer and alternative methods for target generation. Assay performance using the reliable range of comparative measurement as a metric was improved by adjusting sample hybridization conditions for one commercial kit.
Process improvement in microarray assay performance was demonstrated using samples prepared from commercially available materials and two metrics - diagnostic performance and the reliable range of measurement. These methods have advantages over approaches that use a limited set of external controls or correlations to reference sets, because they provide benchmark values that can be used by clinical laboratories to help optimize protocol conditions and laboratory proficiency with microarray assays.
- Microarray Assay
- Array Manufacturer
- Reliable Range
- Reliable Region
- Robust Multichip Analysis
As one of the first 'omic scale technologies in wide-spread use, microarrays provide a test case for addressing issues associated with highly multiplexed assays and high dimensional datasets. During the last decade, community-wide efforts were initiated to address concerns that arose with the increased use of microarray technology in research. These concerns include data comparability, the lack of universal controls for gene expression experiments, the need for data reporting standards, and methods for determining appropriate statistical approaches. The MicroArray Quality Control (MAQC) projects led by the FDA [1, 2], the External RNA Controls Consortium under the leadership of the National Institute of Standards and Technology , and the Microarray Gene Expression Database (MGED) group which developed the MIAME (Minimum Information About a Microarray Experiment) standards  have laid the foundation for establishing confidence in microarray measurements, an essential step for the use of this technology in biomedical applications.
The use of microarray-based clinical tests, although currently limited, is expected to expand with the increasing incorporation of personalized medicine into medical practice. The availability of clinically directed standards would help promote the development of highly multiplexed gene expression assays in laboratory medicine . The use of microarrays in clinical diagnostics will require assurance that clinical laboratories are and remain proficient in using this technology. Microarrays are designed for high throughput assessment of relative mRNA levels, but comparative expression measurements are constrained by limits in the dynamic range of detection . Expression ratios derived from microarray data are compressed in comparison to quantitative real time polymerase chain reaction (qRT-PCR) assay data, especially for probes with high signal intensities . The dynamic range and reliability of measurements on microarray assays can be further compromised by suboptimal laboratory proficiency with this technology. Routine performance assessments using reference samples are highly recommended as a best practice for the generation of high quality microarray data . External RNA controls provided by array manufacturers have utility for evaluating the success or failure of certain sample processing steps but are limited in number and, because they do not span the range of intensities in biological samples, are not representative of all genes on a microarray . The biologically complex samples used in the MAQC-I project to assess microarray data comparability across platforms have been widely used but, unlike samples in a typical analysis, are not closely related. MAQC-I sample A is a universal human reference RNA (UHRR) prepared by pooling RNA from multiple cell lines  and MAQC-I sample B consists of human brain reference RNA (HBRR) prepared from multiple individuals . The difference in mRNA content between MAQC-1 samples A and B hinders the use of MAQC sample titrations to measure the ability to detect known changes between samples . Typically, the MAQC samples have been used for performance comparisons by correlation of results to the reference microarray datasets or to the qRT-PCR data generated for a subset of analytes in the MAQC samples on the TaqMan platform (for examples, see [12, 13]). Correlations to historical reference sets measure the degree of similarity to benchmark data but not necessarily an improvement in performance over reference set levels.
Objective criteria to measure performance on microarrays can be developed for samples which are designed to have known differences in expression. We developed a system for assessing technical performance with microarray assays that involves two biologically complex samples (mixed tissue ratiometric controls (MTRC)) that are representative of experimental samples . MTRC are two samples with three or four RNA components mixed in different proportions. Technical performance is based on quantifiable measures of diagnostic accuracy and the reliable region of measurement. Diagnostic accuracy is the ability of a microarray assay to correctly detect true positive and negative changes in the MTRC samples based on differences in tissue-selective transcripts that are consistent with the ratios of mixed tissue RNA. The area under the curve (AUC) from receiver operating characteristic (ROC) plots summarizes the technical performance in a single benchmark value based on measurements of over 100 true positive and negative analytes in the MTRC. The utility of this approach has been evaluated using rat whole genome microarrays with mixtures of total RNA from four rat tissues (liver, brain, testis, and kidney) known to have distinctly different expression patterns [14, 15]. Translating this approach for laboratories that analyze clinical samples is challenged by quality issues with human tissue RNA. In this manuscript, we describe a method for the design and use of performance standards for human gene expression arrays that use commercially available RNA and include metrics that identify the most reliable region of the dynamic range of measurement.
Selection of components for the human MTRC
Source and quality of human tissue RNA used in testing
Selection of analytes for the human MTRC-4
Performance assessments made using the MTRC are based on tissue-selective analytes. For MTRC analyte selection, tissue-selectivity was defined as an average signal intensity that is ten-fold higher in one tissue than the other tissues in the MTRC. For the MTRC-4, UHRR, HBRR, liver RNA, and skeletal muscle RNA were individually labeled and hybridized to Affymetrix Human Genome U133A 2.0 arrays on separate dates to create three replicate datasets. For each probe set in each tissue, the difference between its log2 signal intensity in that tissue and its maximum value in the remaining tissues was calculated to determine a tissue-selective index (TSI) for each dataset. Using a mean TSI cutoff of 3.22 log2 units, a total of 1035 tissue-selective analytes were identified for the MTRC-4 that included 429 HBRR-, 255 liver RNA-, 197 UHRR-, and 154 skeletal muscle RNA-selective probe sets (see Additional file 1 - Lists of MTRC-4 analytes for Affymetrix HG-U133A 2.0 arrays).
Evaluation of components for the human MTRC-4
Differences in performance between MTRC-4 batches labeled with the same method
Evaluation of diagnostic performance
Comparison of AUCs derived with the MTRC-4 and MTRC-3 using different labeling protocols
Variations in method
For each labeling method, a p-value was calculated for each MTRC-4 analyte using a paired t-test comparison of normalized log2 signals from array data for three replicate assays of Mix1 and Mix2. A ROC-plot was generated for each of the true positive subsets in the MTRC-4 datasets (for 1.5-, 2-, and 4-fold changes) compared to the true negative (1-to-1) subset and the corresponding AUCs calculated as described previously . No significant difference in the accurate detection of changes of two-fold or greater was observed between labeling methods (Table 3). However, there was a statistically significant difference in 1.5-fold AUC between datasets generated with aRNA targets and datasets generated with ss-cDNA targets (P < 0.01). Of the ratios in the MTRC-4, 1.5-fold changes are more sensitive to noise  which can be contributed by factors such as variation between replicates. A lower degree of reproducibility between analyte ratios for technical replicates was observed for data generated using Method 3 compared to Method 1, which may be due in part to the greater familiarity of our laboratory with Methods 1 and 2. A Spearman's rank correlation coefficient (ρ) between each possible comparison of technical replicates was calculated using the log2 ratios of 1035 tissue-selective analytes. The mean ρ (± standard deviation) for datasets 1A-4, 1B-4, 3A-4, and 3B-4 was 0.897 ± 0.02, 0.928 ± 0.004, 0.861 ± 0.019, and 0.852 ± 0.003, respectively. No significant difference in 1.5-fold AUC was found between two versions of a manufacturer's kit (datasets 1A-4 and 2A-4) or between variations in the aRNA protocol in the time of labeling (datasets 1A-4 and 1B-4). There was also no significant difference in the AUCs when the amount of ss-cDNA hybridized to arrays was changed within a protocol (dataset 3A-4 vs. 3B-4).
Determination of reliable range of measurement
where n is the number of analytes per bin. The standard deviation (s) in the log2 ratios of all probe sets is used as the variance estimate and a comparison to the χ2 distribution at α = 0.01 is used to evaluate the goodness of fit within each bin. The set of contiguous bins that pass the χ2 test is identified and used to define the upper and lower limits of a reliable range of measurement using the maximum and minimum intensities from the "passing" bins, respectively (see Figure 1B). By binning LRD values by signal intensity, the LRD-plot demarcates regions of the dynamic range where ratios significantly deviate from their target values due to noise or ratio compression.
Optional design for the human MTRC
Diagnostic performance measured with the 1.5-fold AUC was similar between MTRC-3 and MTRC-4 datasets that were generated using the same labeling methods. No significant difference was observed between 1.5-fold AUCs for MTRC-3 dataset 2A-3 and MTRC-4 dataset 2A-4 that were both labeled using Method 2A or between MTRC-3 dataset 3C-3 and MTRC-4 datasets 3A-4 and 3B-4 that all used variations of Method 3, despite differences in the identity and number of analytes between the two MTRC designs (Table 3). Similar to the MTRC-4, significant differences were observed between MTRC-3 datasets generated with different types of targets (aRNA vs. ss-cDNA). A statistically significant difference in diagnostic performance was observed between dataset 2A-3 that used the MTRC-3 with a linear isothermal amplification method and dataset 3C-3 that used the MTRC-3 with the T7 RNA polymerase based amplification method to generate target for hybridization to arrays (Table 3). A longer reliable range and dynamic range of measurement seen for Method 3 compared to Methods 1 and 2 with the MTRC-4 and in published reports  were also observed with MTRC-3 dataset 3C-3 compared to MTRC-3 dataset 2A-3 (Figure 5).
The use of human MTRC samples which contain known changes in expression along the dynamic range of measurement with the reliable range metric introduced in this manuscript provide a mechanism for clinical laboratories conducting microarray assays to benchmark performance, assess laboratory proficiency, and measure process improvement. Process drift or process improvement, introduced through automation of procedures or changes in reagents, equipment, operator, or platforms, may involve relatively subtle effects on microarray data. For example, in this study we evaluated the effect of a protocol option with a reduced time for target labeling (Method 1A vs. 1B). Our results indicated that the shorter incubation time could be used without significantly impacting assay performance. For these process improvement applications, we have found that a test for accurate measurement of 1.5-fold changes in comparative expression is a better discriminator of subtle effects on performance than assays for changes that are two-fold or higher. A target ratio of 1.5-fold is included in the MTRC-4 as one of 4 components or in the MTRC-3 as the single true positive change. We recommend using the 1.5-AUC metric, which incorporates a measure of diagnostic accuracy and proficiency in replication of results with the MTRC, as a first tier test for process improvement.
Although correlation to reference sets or within replicates is often used to assess performance on microarrays, this metric is of limited value for measuring improvement in laboratory techniques. Datasets generated using methods 3A and 3B had similar mean Pearson's correlation coefficients (data not shown) and Spearman's rank correlation coefficients for ratio reproducibility between technical replicates. However, a clear improvement in the RI-plot (Figures 3A and 4A) and in the reliable range value (Figure 2) could be observed with a change in protocol that reduced the amount of target hybridized to arrays (Method 3B vs. 3A). Optimal results with microarray assays are best obtained using measurements that avoid the boundaries of the dynamic range . The MTRC contain hundreds of analytes with known target ratios that span the dynamic range of measurement and that serve as input values into reliable range estimates. The reliable range of measurement that is calculated with the MTRC from LRD-plots is a benchmark value for evaluating an acceptable performance level for laboratory protocols.
We demonstrated two options in sample design for mixed tissue ratiometric controls. The same metrics (ROC-plot AUCs, reliable range from LRD-plots) can be calculated with either design and yielded similar results. The MTRC-3 design is easily tunable to measure a target ratio of interest and has an advantage over the MTRC-4 in requiring only 3 different sources of reliably good quality human tissue RNA. The human MTRC-4 is directly comparable to the previously described rat MTRRM design used to illustrate the utility of a ROC-plot AUC to identify outliers in performance in a large set of proficiency testing data [14, 15].
Previously, we have described methodology for using the MAQC samples A and B, and two dilutions of A and B, for performance assessments based on ROC-plot metrics . However, these samples have limitations. The MAQC samples A-D each contain 1 or 2 tissues and the observed differences between total RNA components do not correspond to mixed ratios without adjustment for the projected difference in mRNA content between UHRR and HBRR . The MTRC-4 design tested in this study limits the skewing effect of UHRR on observed ratios by using it in a 1:1 ratio. The qRT-PCR data generated for the MAQC-I project can be used as the basis for selecting limited sets of true positive and true negative analytes . However, this approach yields a low number of analytes with limited statistical power for calculating an AUC from a ROC-plot or determining the reliable range from an LRD-plot. The addition of 1 or 2 different human tissue RNA components to MAQC samples A and/or B creates a two-sample system that is better suited for monitoring process improvement with microarray assays. Although this study is limited to Affymetrix gene expression arrays, we have previously shown that the MTRC samples and metrics can be used on other platforms [14, 15].
While DNA microarrays are widely used in medical research in developing improved assays for prediction or diagnosis of disease or for assessing the efficacy and adverse effects of pharmaceuticals, their use in clinical practice is currently limited. One microarray-based transcriptional profiling assay is currently approved by the FDA for predicting responsiveness to certain therapeutic interventions for breast cancer and several others are under development . The expanded use of microarray technology in clinical medicine would be enabled by the implementation of methods for monitoring and optimizing laboratory performance. The methods for measuring performance on human genome-wide microarrays that are described in this manuscript fit this purpose and can be performed by clinical laboratories using samples prepared from commercially available sources.
Human tissue RNA was obtained from Ambion (Applied Biosystems, Carlsbad, CA) or Stratagene (Agilent Technologies Inc., Santa Clara, CA). The catalog and lot numbers of the products tested are indicated in Table 1. RNA was also prepared from human skeletal muscle tissue using a Qiagen TissueLyser and Qiagen RNeasy miniprep kits (Qiagen, Valencia, CA). The donor tissue was obtained from amputation procedures through the Cooperative Human Tissue Network which is funded by the National Cancer Institute. The protocol for this study was reviewed by the FDA Research Involving Human Subjects Committee.
RNA quantification and assessment of purity was performed on a NanoDrop spectrophotometer (ThermoScientific, Wilmington, DE). RNA quality was assessed using an Agilent RNA 6000 Nano kit, an Agilent Bioanalyzer (Agilent Technologies, Inc.), and the manufacturer's software to assign RINs . The RINs that were measured for different lots of commercial human RNA are in Table 1.
UHRR, HBRR, liver RNA, and skeletal muscle RNA were individually labeled using the 3' IVT Express kit (Method 1A) and hybridized to Human Genome U133A 2.0 arrays (Affymetrix, Santa Clara, CA). Three replicate datasets were created by labeling and hybridizing the four tissue RNA samples on separate dates to introduce technical variation. The commercial lots of RNA used for analyte selection are indicated in Table 1. For each replicate dataset, the log2 signal intensities were derived and quantile normalized using the Robust Multichip Analysis (RMA) algorithm in the Affymetrix Expression Console software. A mean TSI for each probe set was calculated from the three replicate datasets and the threshold for tissue selectivity was a mean TSI greater than 3.22 log2 units. For the MTRC-4, 429 HBRR-, 255 liver RNA-, 197 UHRR-, and 154 skeletal muscle RNA-selective analytes were identified (see Additional file 1 - Lists of MTRC-4 analytes for Affymetrix HG-U133A 2.0 arrays). With omission of the UHRR from the comparison, 586 HBRR-, 384 liver RNA-, and 181 skeletal muscle RNA-selective analytes were identified as analytes for the MTRC-3 (see Additional file 2 - Lists of MTRC-3 analytes for Affymetrix HG-U133A 2.0 arrays).
Four batches of MTRC-4 were prepared using total RNA from four human tissues from different lots (Table 1). Batches 1-3 were tested in singlicate measurements (Trials 1-3) and Batch 4 was used in all of the other MTRC-4 datasets. A 100 μg batch of MTRC-4 Mix1 contains 20 μg UHRR, 30 μg liver RNA, 40 μg HBRR RNA, and 10 μg skeletal muscle RNA. A 100 μg batch of MTRC-4 Mix2 contains 20 μg UHRR, 20 μg liver RNA, 20 μg HBRR RNA, and 40 μg skeletal muscle RNA. The ratio of total RNA contributed by UHRR, liver, HBRR and skeletal muscle between Mix1 and Mix2 in the MTRC-4 is 1:1, 1.5:1, 2:1, and 1:4, respectively. UHRR was used for the 1-to-1 component in the MTRC-4 to minimize the impact on the observed mixed ratio due to the higher fraction of mRNA in the cell line derived total RNA than in sources of tissue RNA like the HBRR . Human liver RNA, which was found to have a higher RIN value across commercial lots than the other components, was used for the ratio more sensitive to performance differences (1.5-fold). Of the remaining two components, the HBRR was selected for measuring 2-fold changes because it contained the larger number of tissue-selective analytes.
MTRC-3 were prepared from total RNA from three human tissues using the lots indicated for Batch 5 in Table 1 and mixed to provide a 1:1 ratio for HBRR and reciprocal 1.5:1 ratios for liver and muscle RNAs. A 100 μg batch of MTRC-3 Mix1 contained 25 μg HBRR RNA, 45 μg liver RNA, and 30 μg skeletal muscle RNA. A 100 μg batch of MTRC-3 Mix2 contained 25 μg HBRR RNA, 30 μg liver RNA, and 45 μg skeletal muscle RNA. With the HBRR as the 1-to-1 component in the MTRC-3, a similar number of true negatives (586 brain RNA-selective analytes) and true positives (565 combined liver RNA- and skeletal muscle RNA-selective analytes) could be used in the ROC-plot analyses.
Labeled target was prepared from total RNA samples using one of the following three reagent kits: (1) 3' IVT Express kit (Affymetrix Part No. 901229), (2) IVT kit (Affymetrix Part No. 900449), or (3) Ovation RNA Amplification System V2 and the FL-Ovation cDNA Biotin module kits (Nugen Catalog Nos. 3100 and 4200). Labeled target was hybridized to Affymetrix Human Genome U133A 2.0 arrays in a GeneChip Hybridization Oven 640. The arrays were washed on an Affymetrix GeneChip Fluidics Station 450 using fluidics protocol FS450-002 and scanned using an Affymetrix GeneChip Scanner 3000 7G. The variations made in protocols for each target labeling method are listed in Table 3. For each method, a pair of MTRC was run in triplicate but processed on different days to create sets of technical replicates. The microarray data from this study are available in the ArrayExpress Archive at the European Bioinformatics Institute through accession number E-TABM-1091 http://www.ebi.ac.uk/arrayexpress.
Each dataset that was created using a different target labeling protocol or different batch of MTRC was processed separately with RMA. The selective analytes for the 1-to-1 component (UHRR for the MTRC-4 and HBRR for the MTRC-3) were used to normalize Mix2 with respect to Mix1. For each technical replicate, the difference in the 10% trimmed mean intensity between the Mix1 and Mix2 data for analytes in the 1-to-1 component was calculated and used to correct the Mix2 signal data.
For ROC-plot calculations using the MTRC, the true negatives are the tissue selective analytes for tissue RNA that is mixed in a 1-to-1 ratio between Mix1 and Mix2. Separate ROC-plots are generated for each true positive subset in the MTRC. For MTRC-4, the true positives are the 1.5-, 2-, and 4-fold changes and for the MTRC-3, the true positives are the 1.5-fold changes in both directions. For singlicate assays, analytes are ranked by log2 ratio . For replicate assays, analytes are ranked by p-values calculated using a paired t-test comparison of the three Mix1 and Mix2 signals. AUCs were calculated using the trapezoidal method. Statistical analyses of differences in AUCs were performed as previously described  using the method of Hanley and McNeil .
For MTRC-4 datasets, the analytes (excluding the 1-to-1 component) were distributed into 19 bins of 42, with a single bin of 40 at the lowest intensity. For MTRC-3 datasets, the analytes (excluding the 1-to-1 component) were distributed into 20 bins of 28 analytes, omitting the 5 lowest intensity analytes. The standard deviation (s) ranged from 0.30 - 0.34 for MTRC-4 datasets and from 0.16 - 0.21 for MTRC-3 datasets.
This project was funded in part by the CDER Regulatory Science and Review Enhancement program. The findings and conclusions in this article have not been formally disseminated by the Food and Drug Administration and should not be construed to represent any Agency determination or policy.
- Shi L, Reid LH, Jones WD, Shippy R, Warrington JA, Baker SC, Collins PJ, de Longueville F, Kawasaki ES, Lee KY, Luo Y, Sun YA, Willey JC, Setterquist RA, Fischer GM, Tong W, Dragan YP, Dix DJ, Frueh FW, Goodsaid FM, Herman D, Jensen RV, Johnson CD, Lobenhofer EK, Puri RK, Scherf U, Thierry-Mieg J, Wang C, Wilson M, Wolber PK, Zhang L, Amur S, Bao W, Barbacioru CC, Lucas AB, Bertholet V, Boysen C, Bromley B, Brown D, Brunner A, Canales R, Cao XM, Cebula TA, Chen JJ, Cheng J, Chu TM, Chudin E, Corson J, Corton JC, Croner LJ, Davies C, Davison TS, Delenstarr G, Deng X, Dorris D, Eklund AC, Fan XH, Fang H, Fulmer-Smentek S, Fuscoe JC, Gallagher K, Ge W, Guo L, Guo X, Hager J, Haje PK, Han J, Han T, Harbottle HC, Harris SC, Hatchwell E, Hauser CA, Hester S, Hong H, Hurban P, Jackson SA, Ji H, Knight CR, Kuo WP, Leclerc JE, Levy S, Li QZ, Liu C, Liu Y, Lombardi MJ, Ma Y, Magnuson SR, Maqsodi B, McDaniel T, Mei N, Myklebost O, Ning B, Novoradovskaya N, Orr MS, Osborn TW, Papallo A, Patterson TA, Perkins RG, Peters EH, Peterson R, Philips KL, Pine PS, Pusztai L, Qian F, Ren H, Rosen M, Rosenzweig BA, Samaha RR, Schena M, Schroth GP, Shchegrova S, Smith DD, Staedtler F, Su Z, Sun H, Szallasi Z, Tezak Z, Thierry-Mieg D, Thompson KL, Tikhonova I, Turpaz Y, Vallanat B, Van C, Walker SJ, Wang SJ, Wang Y, Wolfinger R, Wong A, Wu J, Xiao C, Xie Q, Xu J, Yang W, Zhang L, Zhong S, Zong Y, Slikker W: The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements. Nat Biotechnol. 2006, 24 (9): 1151-1161. 10.1038/nbt1239.View ArticleGoogle Scholar
- Shi L, Campbell G, Jones WD, Campagne F, Wen Z, Walker SJ, Su Z, Chu TM, Goodsaid FM, Pusztai L, Shaughnessy JD, Oberthuer A, Thomas RS, Paules RS, Fielden M, Barlogie B, Chen W, Du P, Fischer M, Furlanello C, Gallas BD, Ge X, Megherbi DB, Symmans WF, Wang MD, Zhang J, Bitter H, Brors B, Bushel PR, Bylesjo M, Chen M, Cheng J, Cheng J, Chou J, Davison TS, Delorenzi M, Deng Y, Devanarayan V, Dix DJ, Dopazo J, Dorff KC, Elloumi F, Fan J, Fan S, Fan X, Fang H, Gonzaludo N, Hess KR, Hong H, Huan J, Irizarry RA, Judson R, Juraeva D, Lababidi S, Lambert CG, Li L, Li Y, Li Z, Lin SM, Liu G, Lobenhofer EK, Luo J, Luo W, McCall MN, Nikolsky Y, Pennello GA, Perkins RG, Philip R, Popovici V, Price ND, Qian F, Scherer A, Shi T, Shi W, Sung J, Thierry-Mieg D, Thierry-Mieg J, Thodima V, Trygg J, Vishnuvajjala L, Wang SJ, Wu J, Wu Y, Xie Q, Yousef WA, Zhang L, Zhang X, Zhong S, Zhou Y, Zhu S, Arasappan D, Bao W, Lucas AB, Berthold F, Brennan RJ, Buness A, Catalano JG, Chang C, Chen R, Cheng Y, Cui J, Czika W, Demichelis F, Deng X, Dosymbekov D, Eils R, Feng Y, Fostel J, Fulmer-Smentek S, Fuscoe JC, Gatto L, Ge W, Goldstein DR, Guo L, Halbert DN, Han J, Harris SC, Hatzis C, Herman D, Huang J, Jensen RV, Jiang R, Johnson CD, Jurman G, Kahlert Y, Khuder SA, Kohl M, Li J, Li L, Li M, Li QZ, Li S, Li Z, Liu J, Liu Y, Liu Z, Meng L, Madera M, Martinez-Murillo F, Medina I, Meehan J, Miclaus K, Moffitt RA, Montaner D, Mukherjee P, Mulligan GJ, Neville P, Nikolskaya T, Ning B, Page GP, Parker J, Parry RM, Peng X, Peterson RL, Phan JH, Quanz B, Ren Y, Riccadonna S, Roter AH, Samuelson FW, Schumacher MM, Shambaugh JD, Shi Q, Shippy R, Si S, Smalter A, Sotiriou C, Soukup M, Staedtler F, Steiner G, Stokes TH, Sun Q, Tan PY, Tang R, Tezak Z, Thorn B, Tsyganova M, Turpaz Y, Vega SC, Visintainer R, von Frese J, Wang C, Wang E, Wang J, Wang W, Westermann F, Willey JC, Woods M, Wu S, Xiao N, Xu J, Xu L, Yang L, Zeng X, Zhang J, Zhang L, Zhang M, Zhao C, Puri RK, Scherf U, Tong W, Wolfinger RD, MAQC Consortium: The MicroArray Quality Control (MAQC)-II study of common practices for the development and validation of microarray-based predictive models. Nat Biotechnol. 2010, 28 (8): 827-38. 10.1038/nbt.1665.View ArticleGoogle Scholar
- External RNA Controls Consortium: Proposed methods for testing and selecting the ERCC external RNA controls. BMC Genomics. 2005, 6: 150-10.1186/1471-2164-6-150.View ArticleGoogle Scholar
- Brazma A, Hingamp P, Quackenbush J, Sherlock G, Spellman P, Stoeckert C, Aach J, Ansorge W, Ball CA, Causton HC, Gaasterland T, Glenisson P, Holstege FCP, Kim IF, Markowitz V, Matese JC, Parkinson H, Robinson A, Sarkans U, Schulze-Kremer S, Stewart J, Taylor R, Vilo J, Vingron M: Minimum information about a microarray experiment (MIAME)--toward standards for microarray data. Nat Genet. 2001, 29: 365-71. 10.1038/ng1201-365.View ArticleGoogle Scholar
- Enkemann SA: Standards affecting the consistency of gene expression arrays in clinical applications. Cancer Epidemiol Biomarkers Prev. 2010, 19: 1000-1003. 10.1158/1055-9965.EPI-10-0044.View ArticleGoogle Scholar
- Sharov V, Kwong KY, Frank B, Chen E, Hasseman J, Gaspard R, Yu Y, Yang I, Quackenbush J: The limits of log-ratios. BMC Biotechnol. 2004, 4: 3-10.1186/1472-6750-4-3.View ArticleGoogle Scholar
- Naef F, Socci ND, Magnasco M: A study of accuracy and precision in oligonucleotide arrays: extracting more signal at large concentrations. Bioinformatics. 2003, 19 (2): 178-84. 10.1093/bioinformatics/19.2.178.View ArticleGoogle Scholar
- Corvi R, Ahr HJ, Albertini S, Blakey DH, Clerici L, Coecke S, Douglas GR, Gribaldo L, Groten JP, Haase B, Hamernik K, Hartung T, Inoue T, Indans I, Maurici D, Orphanides G, Rembges D, Sansone SA, Snape JR, Toda E, Tong W, van Delft JH, Weis B, Schechtman LM: Meeting report: Validation of toxicogenomics-based test systems: ECVAM-ICCVAM/NICEATM considerations for regulatory use. Environ Health Perspect. 2006, 114 (3): 420-9. 10.1289/ehp.8247.View ArticleGoogle Scholar
- Kerr KF: Extended analysis of benchmark datasets for Agilent two-color microarrays. BMC Bioinformatics. 2007, 8: 371-10.1186/1471-2105-8-371.View ArticleGoogle Scholar
- Novoradovskaya N, Whitfield ML, Basehore LS, Novoradovsky A, Pesich R, Usary J, Karaca M, Wong WK, Aprelikova O, Fero M, Perou CM, Botstein D, Braman J: Universal Reference RNA as a standard for microarray experiments. BMC Genomics. 2004, 5 (1): 20-10.1186/1471-2164-5-20.View ArticleGoogle Scholar
- Shippy R, Fulmer-Smentek S, Jensen RV, Jones WD, Wolber PK, Johnson CD, Pine PS, Boysen C, Guo X, Chudin E, Sun YA, Willey JC, Thierry-Mieg J, Thierry-Mieg D, Setterquist RA, Wilson M, Lucas AB, Novoradovskaya N, Papallo A, Turpaz Y, Baker SC, Warrington JA, Shi L, Herman D: Using RNA sample titrations to assess microarray platform performance and normalization techniques. Nat Biotechnol. 2006, 24 (9): 1123-31. 10.1038/nbt1241.View ArticleGoogle Scholar
- Wen Z, Wang C, Shi Q, Huang Y, Su Z, Hong H, Tong W, Shi L: Evaluation of gene expression data generated from expired Affymetrix GeneChip®microarrays using MAQC reference RNA samples. BMC Bioinformatics. 2010, 11 (Suppl 6): S10-10.1186/1471-2105-11-S6-S10.View ArticleGoogle Scholar
- Klevebring D, Gry M, Lindberg J, Eidefors A, Lundeberg J: Automation of cDNA synthesis and labelling improves reproducibility. J Biomed Biotechnol. 2009, 2009: 396808-10.1155/2009/396808.View ArticleGoogle Scholar
- Pine PS, Boedigheimer M, Rosenzweig BA, Turpaz Y, He YD, Delenstarr G, Ganter B, Jarnagin K, Jones WD, Reid LH, Thompson KL: Use of diagnostic accuracy as a metric for evaluating laboratory proficiency with microarray assays using mixed tissue RNA reference samples. Pharmacogenomics. 2008, 9: 1753-63. 10.2217/146224184.108.40.2063.View ArticleGoogle Scholar
- Thompson KL, Rosenzweig BA, Pine PS, Retief J, Turpaz Y, Afshari CA, Hamadeh HK, Damore MA, Boedigheimer M, Blomme E, Ciurlionis R, Waring JF, Fuscoe JC, Paules R, Tucker CJ, Fare T, Coffey EM, He Y, Collins PJ, Jarnagin K, Fujimoto S, Ganter B, Kiser G, Kaysser-Kranich T, Sina J, Sistare FD: Use of a mixed tissue RNA design for performance assessments on multiple microarray formats. Nucleic Acids Res. 2005, 33: e187-10.1093/nar/gni186.View ArticleGoogle Scholar
- Son CG, Bilke S, Davis S, Greer BT, Wei JS, Whiteford CC, Chen QR, Cenacchi N, Khan J: Database of mRNA gene expression profiles of multiple human organs. Genome Res. 2005, 15 (3): 443-50. 10.1101/gr.3124505.View ArticleGoogle Scholar
- Su AI, Cooke MP, Ching KA, Hakak Y, Walker JR, Wiltshire T, Orth AP, Vega RG, Sapinoso LM, Moqrich A, Patapoutian A, Hampton GM, Schultz PG, Hogenesch JB: Large-scale analysis of the human and mouse transcriptomes. Proc Natl Acad Sci USA. 2002, 99 (7): 4465-70. 10.1073/pnas.012025199.View ArticleGoogle Scholar
- Thompson KL, Pine PS, Rosenzweig BA, Turpaz Y, Retief J: Characterization of the effect of sample quality on high density oligonucleotide microarray data using progressively degraded rat liver RNA. BMC Biotechnol. 2007, 7: 57-10.1186/1472-6750-7-57.View ArticleGoogle Scholar
- Ma C, Lyons-Weiler M, Liang W, LaFramboise W, Gilbertson JR, Becich MJ, Monzon FA: In vitro transcription amplification and labeling methods contribute to the variability of gene expression profiling with DNA microarrays. J Mol Diagn. 2006, 8 (2): 183-92. 10.2353/jmoldx.2006.050077.View ArticleGoogle Scholar
- Ach RA, Floore A, Curry B, Lazar V, Glas AM, Pover R, Tsalenko A, Ripoche H, Cardoso F, d'Assignies MS, Bruhn L, Van't Veer LJ: Robust interlaboratory reproducibility of a gene expression signature measurement consistent with the needs of a new generation of diagnostic tools. BMC Genomics. 2007, 8: 148-10.1186/1471-2164-8-148.View ArticleGoogle Scholar
- Kurn N, Chen P, Heath JD, Kopf-Sill A, Stephens KM, Wang S: Novel isothermal, linear nucleic acid amplification systems for highly multiplexed applications. Clin Chem. 2005, 51 (10): 1973-81. 10.1373/clinchem.2005.053694.View ArticleGoogle Scholar
- Barker CS, Griffin C, Dolganov GM, Hanspers K, Yang JY, Erle DJ: Increased DNA microarray hybridization specificity using sscDNA targets. BMC Genomics. 2005, 6 (1): 57-10.1186/1471-2164-6-57.View ArticleGoogle Scholar
- Thompson KL, Pine PS: Comparison of the diagnostic performance of human whole genome microarrays using mixed-tissue RNA reference samples. Toxicol Lett. 2009, 186: 58-61. 10.1016/j.toxlet.2008.08.018.View ArticleGoogle Scholar
- Slodkowska EA, Ross JS: MammaPrint™ 70-gene signature: another milestone in personalized medical care for breast cancer patients. Expert Rev Mol Diagn. 2009, 9 (5): 417-22. 10.1586/erm.09.32.View ArticleGoogle Scholar
- Schroeder A, Mueller O, Stocker S, Salowsky R, Leiber M, Gassmann M, Lightfoot S, Menzel W, Granzow M, Ragg T: The RIN: an RNA integrity number for assigning integrity values to RNA measurements. BMC Mol Biol. 2006, 7: 3-10.1186/1471-2199-7-3.View ArticleGoogle Scholar
- Hanley JA, McNeil BJ: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982, 143: 29-36.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.