A Paradise Deferred
Reflections on the Dire Need for Curated, Open ADME-Tox Benchmarks
ADME Talks is a newsletter democratizing the cutting edge in virtual ADME-Tox. Pun intended.
Were you forwarded this post? Please subscribe:
An intrepid drug hunter gazes out over the hazy in silico ADME-Tox landscape
The Wanderer Above The Sea of Fog, Caspar David Friedrich
Towards Prediction Paradise?
In their March 2003 article “ADMET in silico modelling: towards prediction paradise?” Han van de Waterbeemd, then Pfizer’s Director of in silico ADMET, and Eric Gifford, an associate research fellow, paint a paradisaical prospect: high-throughput multi-objective small molecule discovery in part enabled by the precise computational modeling of ADME-Tox endpoints. Concluding their discussion, the authors express optimism about the in silico future. They opine:
Driven by the changes in the working paradigm in the pharmaceutical and biotechnology industry, in silico approaches will inevitably find their place.
However, they quickly hedge their positivity:
The ability to continuously adapt and refine the existing models by building on larger and higher-quality data sets will be crucial to the success of in silico approaches.
Resounding from 2003, van de Waterbeemd and Gifford’s clarion call for large, clean, high quality data sets to advance the state-of-the-art in ADME-Tox modeling rings clear and true still today. Indeed, their call is particularly poignant in 2020 given that, despite the proliferation of machine learning methods and the coming of age of deep learning approaches in the last five years, reliable prediction still feels like a far off dream for the majority of ADME-Tox tasks.
Individual academic teams have made praiseworthy efforts in advancing the state-of-the-art for a handful of properties. Many have published painstakingly aggregated and cleaned training and testing data sets to encourage open scientific advancement. Others, however, choose not to publish their data. Others still train models with proprietary data from pharma industry partners that they couldn’t publish even if they wanted to. Forgoing an extended discussion on how we’ve gotten to this point, the reality is that the ADME-Tox benchmarking landscape is fragmented. Because most new algorithms are published and metered against bespoke, opaque, or inaccessible data sets, it is difficult to track progress on in silico tasks. After 17 years, we are still well short of the “prediction paradise” envisioned by van de Waterbeemd and Gifford.
A crisis of comparison
The lack of widely-accepted, open benchmarks to guide in silico ADME-Tox development casts a fog over the research landscape. Comparing model performance across different benchmark test sets is akin to comparing apples to figs, and for most new papers, it is nigh impossible to discern with certainty whether model featurization, algorithmic advances, or enriched training data is responsible for improvements in performance. (Perhaps some combination or all three contribute!) Given the current climate, it is difficult to judge which approaches are promising and where future research should be applied.
Not only does a lack of general benchmarks make it hard to tell where progress is being made on ADME-Tox tasks, it also creates an environment in which commercial entities offering ADME-Tox prediction services (companies not wholly without scientific integrity, but nevertheless with apparent commercial interests) can market products trumpeting inflated performance metrics on tiny sample sizes with close to meaningless applicability domains. What’s more––to complete a vicious cycle––when inflated metrics like these are published and promoted without consequence, they propagate unreasonable expectations for the state-of-the-art in ADME-Tox prediction and set back the responsible adoption of machine learning tools into discovery workflows.
The missing ingredient
The most crucial missing ingredient for one day reaching the ADME-Tox prediction promised land is the curation and hosting of high quality benchmark data sets. Making datasets public, setting up leader boards for model prediction using unhackable performance metrics, encouraging development from intrepid individuals, for-profit companies, and research institutes alike, all out in the open, is––to mix metaphors––the proverbial formation of the knights of the round table that must precede the hunt for the holy grail.
OpenBench strives to be the home for open, high quality ADME-Tox datasets and the gathering place for in silico ADME-Tox discussion online. In the early going, we will signal boost and host links to the richest available public data sets and host models for a selection of ADME-Tox endpoints. Over time, we will begin to publish novel benchmarks and research of our own. We will put on competitions to encourage improvement against established benchmarks and make the best-in-class trained endpoints available for easy use at low marginal cost in the OpenBench Lab. Sign up for a free trial today!
If you believe in our mission and want to contribute, please participate in a brief survey to help guide our efforts in the early stages and subscribe to our newsletter:
You can also reach out via twitter @opnbnch or eMail us directly at founders@opnbnch.com. In the meantime, happy predicting. See you in paradise!