Research

Dataset size and composition impact the reliability of performance benchmarks for peptide-MHC binding predictions

Read the article

Abstract

Background: It is important to accurately determine the performance of peptide: MHC binding predictions, as this enables users to compare and choose between different prediction methods and provides estimates of the expected error rate. Two common approaches to determine prediction performance are cross-validation, in which all available data are iteratively split into training and testing data, and the use of blind sets generated separately from the data used to construct the predictive method. In the present study, we have compared cross-validated prediction performances generated on our last benchmark dataset from 2009 with prediction performances generated on data subsequently added to the Immune Epitope Database (IEDB) which served as a blind set. Results: We found that cross-validated performances systematically overestimated performance on the blind set. This was found not to be due to the presence of similar peptides in the cross-validation dataset. Rather, we found that small size and low sequence/affinity diversity of either training or blind datasets were associated with large differences in cross-validated vs. blind prediction performances. We use these findings to derive quantitative rules of how large and diverse datasets need to be to provide generalizable performance estimates. Conclusion: It has long been known that cross-validated prediction performance estimates often overestimate performance on independently generated blind set data. We here identify and quantify the specific factors contributing to this effect for MHC-I binding predictions. An increasing number of peptides for which MHC binding affinities are measured experimentally have been selected based on binding predictions and thus are less diverse than historic datasets sampling the entire sequence and affinity space, making them more difficult benchmark data sets. This has to be taken into account when comparing performance metrics between different benchmarks, and when deriving error estimates for predictions based on benchmark performance.

binding prediction Immune Epitope Database allergy Hypersensitivity (MeSH) immune system disease Primates Mammalia Vertebrata Chordata Animalia (Animals Chordates Humans Mammals Primates Vertebrates) - Hominidae [86215] human common major histocompatibility complex MHC 10060 Biochemistry studies - General 10064 Biochemistry studies - Proteins peptides and amino acids 34502 Immunology - General and methods 34508 Immunology - Immunopathology tissue immunology 35500 Allergy cross-validated analysis mathematical and computer techniques sequence analysis laboratory techniques genetic techniques Biochemistry and Molecular Biophysics Methods and Techniques BIOCHEMICAL BIOTECHNOLOGY MATHEMATICAL T-CELL EPITOPES DATABASE IMMUNOGENICITY IMMUNOLOGY NETMHCPAN MOLECULES SEQUENCE RESOURCE AFFINITY Benchmarking of MHC class I predictors Epitope prediction Sequence similarity Cross-validation