Saturday, 30 August 2014

Ligand efficiency metrics considered harmful

It has been a while since I did a proper blog post. Some of you may have encountered a Perspective article entitled, ‘Ligand efficiency metrics considered harmful’ and I’ll post on it because the journal has made the article open access until Sept 14.  The Perspective has already been reviewed by Practical Fragments and highlighted in a F1000 review. Some of the points discussed in the article were actually raised last year in Efficient Voodoo Thermodynamics, Wrong Kind of Free Energy and ClogPalk : a method for predicting alkane/water partition coefficients.  There has been recent debate about the validity of ligand efficiency (LE) which is summarized in a blog post (make sure to look at the comments as well).  However, I believe that both sides missed the essential point which is that the choice (conventionally 1 M) of concentration that is used to define the standard state is entirely arbitrary.  

In this blog post, I’ll focus on what I call ‘scaled’ ligand efficiency metrics (LEMs). Scaling means that a measure of activity or affinity is divided by a risk factor such as molecular weight (MW) or heavy (i.e. non-hydrogen) atoms (HA).   For example, LE can be calculated by dividing the standard free energy of binding (ΔG°) by HA:
LE  =  (1/HA)´RTloge(Kd/C°)

Now you’ll notice that I’ve written ΔG° in terms of the dissociation constant (Kd) and the standard concentration (C°) and I articulated why this is important early last year.  The logarithm function is only defined for numbers (i.e. dimensionless quantities) and, inconveniently, Kd has units of concentration.  This means that LE is a function of both Kd and C° and I’m going to first redefine LE a bit to make the problem a bit easier to see.   I'll use IC50 instead of Kd to define a new LEM which I’m not going to name (in the LEM literature names and definitions keep changing so much better to simply state the relevant formula for whatever is actually used):


I have a number of reasons for defining the metric in this manner and the most important of these is that the new metric is metric is dimensionless.  Note how I use the number of heavy atoms (NHA) rather than HA to define the metric.  Just in case you’re wondering what the difference is, HA for ethanol is 2 heavy atoms while NHA is 2 (lacking units of heavy atoms).  Apologies for being pedantic (some might even call me a units Nazi) but if people had paid more attention to units, we’d never have got into this sorry mess in the first place.  The other point to note is that I’ve not converted IC50 to units of energy, mainly for the reason that it is incorrect to do so because an IC50 is not a thermodynamic quantity. However, there are other reasons for not introducing units of energy. Often units of energy go AWOL when values of LE are presented and there is no way of knowing whether the original units were kcal/mol or kJ/mol. Even when energy units are stated explicitly, this might be masking a situation in which affinity and potency measurements have been combined in a single analysis.  Of course, one can be cynical and suggest that the main reason for introducing energy units is to make biochemical measurements appear to be more physical. 

So let’s get back to that new metric and you’ll have noticed a quantity in the defining equation that I’ve called Cref (reference concentration).  This is similar to the standard concentration in that the choice of its value is completely arbitrary but it is also different because it has no thermodynamic significance. You need to use it when defining the new LEM because, at the risk of appearing repetitive, you can’t calculate a logarithm for a quantity that has units.  Another way of thinking about Cref is as an arbitrary unit of concentration that we’re using to analyze some potency measurements.  Something that is really, really important and fundamental in science is that your perception of a system should not change when you change the units of the quantities that describe the system.  If it does then, in the words of Pauli, you are “not even wrong” and you should avoid claiming penetrating insight so as to prevent embarrassment later. So let’s take a look at how changing Cref affects our perception of ligand efficiency.  The table below is essentially the same as what was given in the Perspective article (the only difference is that I’m using IC50 in the blog post rather than Kd).  I also should point out that Huan-Xiang Zhou and Mike Gilson made a similar criticism of LE back in 2009 (although we cited their article in the context of standard states, we failed to notice the critique at the end of their article and there really is no excuse for having missed it).  When a reference concentration of 1 M is used, the three compounds are all equally ligand efficient according to the new LEM.  If Cref = 0.1 M, the compounds appear to become more ligand efficient as molecular size increases but the opposite behavior is observed for Cref = 10 M.

Here’s a figure that shows the problem from a different angle. See how the three parallel lines respond the scaling transformation (Y=> Y/X).  The line that passes through the origin transforms to a line of zero slope (Y/X is independent of X) while the other two lines transform to curves (Y/X is dependent on X).  This graphic has important implications for fit quality (FQ) because it shows that some of the size dependency of LE is a consequence of the (arbitrary) choice of a value (usually 1 M) for Cref or C°.


A common response that I’ve encountered when raising these points is that LE is still useful.  My counter-response is that Religion is also useful (e.g. as a means for pastors to fleece their flocks efficiently) but that, by itself, neither makes it correct nor ensures that that it is being used correctly (if that is even possible for Religion).  One occasionally expressed opinion is that, provided you use a single value for Cref or C°, the resulting LE values will be consistent.  However, consistency is no guarantee of correctness and we need to remember that we create metrics for the purpose of measuring things with them.  When you advocate use of a metric in drug discovery, the burden of proof is on you to demonstrate that the metric has a sound scientific basis and actually measures what it supposed to measure.
The Perspective is critical of the LEMs used in drug discovery but it does suggest alternatives that do not suffer from the same deficiencies.  This is a good point to admit that it took me a while to figure out what was wrong with LE and I’ll point you towards a blog post from over five years ago that will give you an idea about how my position on LEMs has evolved.  It is often stated that LEMs normalize activity with respect to risk factor although it is rarely, if ever, stated explicitly what is meant by the term ‘normalize’.   One way of thinking about normalization is as a way to account for the contribution of a risk factor, such as molecular size or lipophilicity, to activity.  You can do this by modelling the activity as a function of risk factor and using the residual as a measure of activity that has been corrected for the contribution made by the risk factor in question.  You can also think of an LEM as a measure of the extent to which the activity of a compound beats a trend (e.g. linear response of activity to HA). If you’re going to do this then why not use the trend actually observed in the data rather than some arbitrarily assumed trend?

Although I still prefer to use residuals to quantify extent to which activity beats a trend, the process of modelling activity measurements hints at a way that LE might be rehabilitated to some extent.  When you fit a line to activity (or affinity) measurements, you are effectively determining the value of Cref (or C°) that will make the line of fit pass through the origin and which you can then use to redefine activity (or affinity) for the purpose of calculating LE. I would argue that the residual is a better measure of the extent to which the activity of a compound beats the trend in the data because the residual has sign and the uncertainty in it does not depend explicitly on the value of a scaling variable such as HA. However, LE defined in this manner can at least be claimed to take account of the observed trend in the activity data. Something that you might want to think about in this context is whether or not you'd expect the same parameters (slope and intercept) if you were to fit activity measured against different targets to MW or HA.  My reason for bring this up is that it has implications for the validity of mixing results from different assays in LE-based analyses so see what you think.   

I’ll wrap up by directing you to a presentation that I've been doing lately and it includes material from the earlier correlation inflation study.  A point that I make when presenting this material is that, if we do bad science and bad data analysis, people can be forgiven for thinking that the difficulties in drug discovery may actually be of our own making.

Wednesday, 4 June 2014

Ligand efficiency metric teaser

Here's a bit of fun. See how these parallel lines transform when you divide Y by X (this is how you calculate ligand efficiency).

Here's some more fun. See how these lines transform when you subtract X from Y (this is how you calculate lipophilic efficiency.

Would you consider activities lying on a straight line when plotted against molecular size or lipophilicity to represent compounds with the same efficiency?

Friday, 23 August 2013

To review or not to review?

I'll admit that I don’t always see eye-to-eye with journal editors although it has been a while since I last blogged on this theme.  Recently we published an article on prediction of alkane/water partition coefficients and, as was noted at the end of the blog post, the article had originally been submitted to another journal which declined to review the submission.  Much is written about peer review and, if your manuscript actually gets reviewed, you’ll get a good idea about why the journal said nein.  Put another way, there is some accountability even if you don’t like the outcome.
It can be very different when the journal declines to review and I see this as a sort of shadow land in the manuscript review world.  You often (usually?) don’t get any useful feedback as to why your submission was not even worthy of review and the explanation will often be that the manuscript lacked broad appeal.  Sometimes the manuscript will be farmed out to a member of the editorial advisory board who will effectively make the decision for the editor but you never get to see what actually got said.   This is why I talk about a shadow land. 
Our alkane/water partition coefficient manuscript was originally submitted to the Journal of Chemical Information and Modeling and I really did think it was a JCIM sort of manuscript.  Those open science enthusiasts reading this might be interested to know that we were providing source code (albeit source that used the proprietary OEChem toolkit from OpenEye) and the data that was used to build and test the model.  I went through the submission process which included creating a graphical abstract and providing email addresses for potential reviewers (whom I contacted to see if it’d be OK).  The response from JCIM was swift:

'Thank you for considering the Journal of Chemical Information and Modeling for your manuscript submission. The Journal is committed to promoting high standards in the fields that it covers, so we will be quite selective. Most ACS journals including JCIM are also experiencing substantial increases in submissions that are burdening the editorial and reviewing systems. Consequently, most editors are now prescreening and returning ca. 30% of submissions without further review.

For JCIM, some typical problems that will lead to return of a manuscript by the Editor include papers that:
(A) fall outside the aims and scope of JCIM, sometimes characterized by few if any references to papers in JCIM or other mainstream journals in computational chemistry,
(B) are routine studies, do not provide significant new results, or promote new methods without demonstration of advantages over prior alternatives,
(C) center on developments and interest in research areas that are tangential to the scope of JCIM, or
(D) are characterized by poor presentation of the work that is inconsistent with the style and quality of typical papers in JCIM. In particular, QSAR/QSPR submissions are welcome, but authors must be careful to avoid problems under heading (B).

In my judgment, your submission is inappropriate for JCIM.  The journal has limited interest in QSAR/QSPR studies; ones that are considered need to adhere to the guidelines published in JCIM 46, 937 (2006).
I regret any inconvenience, but hope that this prompt return will allow you to consider alternative journals without further delay.'
I was genuinely surprised by the decision but take the view that journals can publish anything they want to. Nobody is entitled to have their manuscripts published.  Unless of course they have paid for the service as is the case for Open Access.  I was a bit alarmed that the journal appeared to be using number of references to its own articles ('…fall outside the aims and scope of JCIM, sometimes characterized by few if any references to papers in JCIM or other mainstream journals in computational chemistry') as a criterion for deciding whether to send a manuscript for review. However, life must go on and so I replied as follows:

'Thank you for your prompt response to our manuscript submission. While we are disappointed to learn that JCIM is not interested in prediction of alkane/water partition coefficients, we fully accept your decision and will seek a more appropriate home for the manuscript.' (07-Mar-2013)

The manuscript was duly submitted to JCAMD which accepted it but just as the article went to press, I noticed that JCIM had recently published an article on alkane/water partition coefficients.  I was aware that our article had cited two or three articles from JCIM’s predecessor JCICS but only one from JCIM itself.  Given that that it appeared that JCIM were actually interested in alkane/water partitioning, I sought specific feedback as to why our submission was not considered worth sending for review:
'Although our alkane/water partition article is currently in press at JCAMD, I must now confess to being perplexed by your decision to reject the manuscript without review given the recent publication of "Structural Determinants of Drug Partitioning in n-Hexadecane/Water System" ( ) in JCIM.  The possibility that number of references to JCIM articles might be used as a criterion for deciding whether to send a manuscript for review did worry me (and continues to do so) when you first communicated the decision in March.  I would be interested to know if you are able to comment more specifically on this matter.' (23-May-2013)
JCIM have not yet responded to my query and I am still perplexed.

Literature cited
Kenny, Montanari, Propopczyk (2013) ClogPalk: A method for predicting alkane/water partition coefficient. JCAMD 27:389-402 DOI

Natesan, Wang, Lukacova, Peng, Subramaniam, Lynch, Balaz (2013) Structural determinants of drug partitioning in n-hexadecane/water system. JCIM 53:1424-1435 DOI

Tuesday, 13 August 2013

Malan's Ten Rules for Air Fighting

I'm going to introduce the #SavingPharma hashtag in this post but I'm also going to introduce WW2 fighter pilot (and apartheid opponent)  Adolph Gysbert Malan (better known as Sailor).  Some of what these combat pilots had to say is still relevant and resonant today.  One of my favorites is the cricketer Keith Miller (and former Mosquito pilot) responding to an interviewer's question about the pressures of Test Cricket by noting that, "Pressure is a Messerschmitt up your arse, playing cricket is not".  Malan's Ten Rules for Air Fighting can teach those who would claim to be leading Drug Discovery and I reproduce them here:
1. Wait until you see the whites of his eyes. Fire short bursts of one to two seconds only when your sights are definitely "ON".

2. Whilst shooting think of nothing else, brace the whole of your body: have both hands on the stick: concentrate on your ring sight.

3. Always keep a sharp lookout. "Keep your finger out".

4. Height gives you the initiative.

5. Always turn and face the attack.

6. Make your decisions promptly. It is better to act quickly even though your tactics are not the best
7. Never fly straight and level for more than 30 seconds in the combat area.

8. When diving to attack always leave a proportion of your formation above to act as a top guard.

9. INITIATIVE, AGGRESSION, AIR DISCIPLINE, and TEAMWORK are words that MEAN something in Air Fighting.

10. Go in quickly - Punch hard - Get out

Sunday, 4 August 2013

Presentations on slideshare

I recently uploaded some presentations to slideshare including my recent Gordon conference talk and the slide set used on visits to a couple of Pharmaceutical companies before the conference.  My slideshare profile is pwkenny and there are currently 13 presentations uploaded.  Be warned that there is plenty of overlap between some slide sets and it is fair to say that there are more presentations than themes.  I hope that you'll find the slide sets useful.

Monday, 29 July 2013

Some reflections on computer-aided drug design (after attending CADD Gordon conference)

I’ve just returned to Trinidad where I’ve been spending the summer.  I was in the USA at the Computer-Aided Drug Design (CADD) Gordon Conference (GRC) organized by Anthony Nicholls and Martin Stahl.  The first thing that I should say is that this will not be a conference report because what goes on at all Gordon Conferences is confidential and off-record. This is intended to make discussions freer and less inhibited and you won’t see proceedings of GRCs published.  Nevertheless, the conference program is available online so I think that it’ll be OK to share some general views of the CADD field (which have been influenced by what I picked up at the conference) even though commenting on specific lectures, posters or discussions would be verboten.

The focus of the conference was Statistics in CADD and the stuff that I took most notice of was the use of Baysian methods although I still need to get my head round things a bit more. Although modelling affinity/potency and output of ADMET assays (e.g. hERG blockade) tends to dominate the thinking of CADD scientists, the links between these properties and clinical outcomes in live humans are not as strong as many assume.  Could the wisdom of Rev Bayes be applied to PK/PD modeling and the design of clinical trials?  I couldn’t help worrying about closet Bayesians in search of a good posterior and what would be the best way to quantify the oomph of a ROC curve...

Reproducibility is something that needs to be addressed in CADD studies and if we are to improve this we must be prepared to share both data and methods (e.g. software).  This open access article should give you an idea of some of the issues and directions in which we need to head. Journal editors have a part to play here and must resist the temptation to publish retrospective analyses of large proprietary data sets because of the numbers of citations that they generate.  At the same time, journal editors should not be blamed for supplemental information ending up in PDF format.  For example, I had no problems (I just asked) getting JMC (2008), JCIM (2009) and JCAMD (2013 and 2013) to publish supplemental information in text (or zipped text) format.

When you build models from data, it is helpful to think of signal and noise.  The noise can be thought of as coming from both the model and from the data and in some cases it may be possible to resolve it into these two components.  The function of Statistics is to provide an objective measure of the relative magnitudes of signal and noise but you can’t use Statistics to make noise go away (not that this stops people from trying).  Molecular design can be defined as control of behavior of compounds and materials by manipulation of molecular properties and can be thought of as being prediction-driven or hypothesis-driven.   Prediction-driven molecular design is about building predictive models but it is worth remembering that a much (most?) pharmaceutical design involves a significant hypothesis-driven component. One way of thinking about hypothesis-driven molecular design is as framework for assembling structure activity/property relationships (SAR/SPR) as efficiently as possible but this is not something that statistical methodology currently appears equipped to do particularly well.

The conference has its own hashtag (#grccadd) and appeared to out-tweet the Sheffield Cheminformatics conference which ran concurrently.  Some speakers have shared their talks publically and a package of statistical tools created especially for the conference is available online  

Literature cited and links to talks
WP Walters (2013) Modeling, informatics, and the quest for reproducibility. JCIM 53:1529-1530 DOI

CC Chow, Bayesian and MCMC methods for parameter estimation and model comparison. Link

N Baker, The importance of metadata in preserving and reusing scientific information Link

PW Kenny, Tales of correlation inflation.  Link

CADD Stat Link

Sunday, 14 July 2013

Prediction of alkane/water partition coefficients

Those of you who follow this blog will know that I have a long standing interest in alkane/water partition coefficient and I’d like to tell you a bit about the ClogPalk model for predicting these from molecular structure that we published during my time in Brasil. Some years ago we explored prediction of ΔlogP (logPoct - logPalk) from calculated molecular electrostatic potentials and this can be thought of as treating the alkane/water partition coefficient as a perturbation of the octanol/water partition coefficient.  One disadvantage of this approach is that it requires access to logPoct and I was keen to explore other avenues.  The correlation of logPalk with computed molecular surface area (MSA) is excellent for saturated hydrocarbons and I wondered if this class of compound might represent a suitable reference state for another type of perturbation model.  Have a look at Fig 1 which shows plots of logPalk against MSA for saturated hydrocarbons (green), aliphatic alcohols (red) and aliphatic diols (blue).  You can see how adding a single hydroxyl group to a saturated hydrocarbon shifts logPalk down by about 4.5 units and adding two hydroxyl groups shifts logPalk further still.

The perturbations are defined substructurally using SMARTS notation. Specifically, each perturbation term consists of a SMARTS definition for the relevant functional group and a decrement term (e.g. 4.5 units for alcohol hydroxyl).  The model also allows functional groups to interact with each other.  For example, an intramolecular hydrogen bond ‘absorbs’ some of a molecule’s polarity and manifests itself as an unexpectedly high logPalk value.  Take a look at this article if you’re interested in this sort of thing.  The interaction terms can be thought of as perturbations of perturbations. The ClogPalk model is shown in Fig 2.

The performance of the model against external test data is shown in Figure 3.  There do appear to be some issues with some of the data and measured values of logPalk were found to differ by two or more units for some compounds (Atropine, Propanolol, Papavarine).  Also there are concerns about the self-consistency of the measurements for Cortexolone, Cortisone and Hydrocortisone. Specifically, the logPalk of Cortexolone (-1.00) is actually lower than that for its keto analogue Cortisone (-0.55).
The software was built using OpenEye programming toolkits (OEChem and Spicoli) and you’ll find the source code and makefiles in the supplementary information with all the data used to parameterize and test the models. It’s not completely open source because you’ll need a license from OpenEye to actually run the software.  However, the documentation for the toolkits is freely available online and you may be even able to get an evaluation license to see how things work.  You’ll also find the source code for SSProFilter in the supplemental material and this is an improved (it also profiles) version of the Filter program that I put together with the Daylight toolkit back in 1996. Very useful for designing screening libraries and you might want to take a look at this post on SMARTS from a couple of years ago.

There's some general discussion in the article that is not specific to the ClogPalk model and I'll mention it briefly since I think this is relevant to molecular design. Those of you who believe that the octanol/water partition coefficient is somehow fundamental might like to trace how we ended up with this particular partitioning system.  We also address the question of whether logP or logD is the more appropriate measure of lipophilicity measure and some ligand efficiency stuff from an earlier post makes its journal debut.  
That’s about all I wanted to say for now and I’ll finish by noting that the manuscript was originally submitted to another journal but that's going to be the subject of a post all of its very own...
Literature cited
Toulmin, Wood, Kenny (2008) Toward prediction of alkane/water partition coefficients. J Med Chem 51:3720-3730 DOI
Kenny, Montanari, Prokopczyk (2013)ClogPalk: A method for predicting alkane/water partition coefficient. JCAMD 27:389-402 DOI