PAINS and editorial policy

<< previous || next >>

I have blogged previously (1 | 2 | 3 | 4 | 5 ) on PAINS. In this post, I present the case against inclusion of PAINS criteria in the Journal of Medicinal Chemistry (JMC) Guidelines for Authors (viewed 22-Jan-2017) as given below:

"2.1.9. Interference Compounds. Active compounds from any source must be examined for known classes of assay interference compounds and this analysis must be provided in the General Experimental section. Compounds shown to display misleading assay readouts by a variety of mechanisms include, but are not limited to, aggregation, redox activity, fluorescence, protein reactivity, singlet-oxygen quenching, the presence of impurities, membrane disruption, and their decomposition in assay buffer to form reactive compounds. Many of these compounds have been classified as Pan Assay Interference Compounds (PAINS; see Baell & Holloway, J. Med. Chem. 2010, 53, 2719-2740 and webinar at bit.lyj/mcPAINS). Provide firm experimental evidence in at least two different assays that reported compounds with potential PAINS liability are specifically active and their apparent activity is not an artifact."

The term 'known classes of assay interference compounds' must be defined more precisely in order to be usable by both authors submitting manuscripts and reviewers of those manuscripts. Specifically, the term 'known classes of assay interference compounds' implies the existence of a body of experimental data in the public domain for which specific substructural features have been proven to cause the observed assay interference. The JMC Guidelines for Authors (viewed 22-Jan-2017) imply that any assay result for a compound with 'potential PAINS liability' should necessarily be treated as less informative than would be the case if the compound did not have 'potential PAINS liability'. I shall term this as 'devaluing' the assay result.              

The PAINS acronym stands for Pan Assay INterference compoundS and it was introduced in a 2010 JMC article (BH2010) that is cited in the Guidelines for Authors (viewed 22-Jan-2017). The PAINS filters introduced in the BH2010 study are based on analysis of frequent-hitter behavior in a panel of 6 AlphaScreen assays. Each PAINS filter consists of a substructural pattern and is associated with an enrichment factor that quantifies the frequent hitter behavior. Compounds that quench or scavenge singlet oxygen have the potential to interfere with AlphaScreen assays but individual PAINS substructural patterns were not evaluated for their likelihood of being associated with singlet oxygen quenching or scavenging. For example, the BH2010 study makes no mention of studies ( 1 | 2 | 3 | 4 ) linking singlet oxygen quenching/scavenging to the presence of a thiocarbonyl group which is a substructural element present in rhodanines.  

I argue that a high hit rate against a small panel of assays that all use a single detection technology is inadmissible as evidence for pan-assay interference.  I also argue that the results of screening against this assay panel can only be invoked to devalue the result from an AlphaScreen assay. In a cheminformatic context, the applicability domain of a model based on analysis of results from this assay panel is restricted to activity measured in AlphaScreen assays. Furthermore, it is questionable whether it is valid to invoke the results of screening against this assay panel to devalue a concentration response from an AlphaScreen assay because the results for each assay of the panel were obtained at a single concentration (i.e. no concentration response).

The BH2010 study does present some supporting evidence that compounds matching PAINS substructural patterns are likely to interfere with assays. In a cheminformatic context, this supporting evidence can be considered to extend the applicability domain of PAINS filters. However, supporting evidence is only presented for some of the substructural patterns and much of that supporting evidence is indirect and circumstantial. For example, the observation that rhodanines as a structural class have been reported as active against a large number of targets is, at best, indirect evidence for frequent hitter behavior which is characterized by specific compounds showing activity in large numbers of assays. There is not always a direct correspondence between PAINS substructural patterns and those used in analyses that are presented as supporting evidence. For example, the BH2010 study uses substructural patterns for rhodanines that specify the nature of C5 (either saturated or with exocyclic carbon-carbon bond).  However, the sole rhodanine definition given in the BMS2006 study specifies an exocyclic carbon-carbon double bond. This means that it is not valid to invoke the BMS2006 study to devalue the result of every assay performed on any rhodanine.

The data (results from 6 AlphaScreen assays and associated chemical structures) that form the basis of the analysis in the BH2010 study are not disclosed and must therefore be considered to be proprietary. Furthermore, some of the supporting evidence that compounds matching PAINS filters are likely to interfere with assays is itself based on analysis (e.g. BMS2006 and Abbott2007) of proprietary data. The JMC Guidelines for Authors (viewed 22-Jan-2017) make it clear that the use of proprietary data is unacceptable:

"2.3.5.2 Proprietary Data. Normally, the use of proprietary data for computational modeling or analysis is not acceptable because it is inconsistent with the ACS Ethical Guidelines. All experimental data and molecular structures used to generate and/or validate computational models must be reported in the paper, reported as supporting information, or readily available without infringements or restrictions. The Editors may choose to waive the data deposition requirement for proprietary data in a rare case where studies based on very large corporate data sets provide compelling insight unobtainable otherwise.

2.3.6 QSAR/QSPR and Proprietary Data. The following are general requirements for manuscripts reporting work done in this area:

(3) All data and molecular structures used to carry out a QSAR/QSPR study are to be reported in the paper and/or in its supporting information or should be readily available without infringements or restrictions. The use of proprietary data is generally not acceptable."

Given JMC's stated unacceptability of analysis based on proprietary data, to use such analysis to define editorial policy would appear to contradict that editorial policy.

To sum up:

• Analysis of the screening results for the BH2010 assay panel can only be invoked  invoked to devalue or otherwise invalidate the result from an AlphaScreen assay.

• Additional supporting evidence is only provided in BH2010 for some of the PAINS filters. In these cases, the evidence is not generally presented in a manner that would allow a manuscript reviewer to assess risk of assay interference in an objective manner.

• Most of the analysis presented in the BH2010 study has been performed on proprietary data. To base JMC editorial policy on analysis of proprietary data would appear to contradict the Journal's policy on the use of proprietary data.

I rest my case.

Confessions of a Units Nazi

Regular readers (both of them) of this blog will know that I have an interest, which some might term an obsession, with units. At high school in Trinidad, we had the importance of units beaten into us by the Holy Ghost Fathers and, for some of the more refractory cases, the beating was quite literal. I was taught physics by the much loved, although somewhat highly-strung, Fr. Knolly Knox (aka Knox By Night) who, as Dean of the First Form, used to give 'licks' with a cane of hibiscus (presumably chosen for its tensile properties). You quickly learned not to mess with The Holy Ghost Fathers, especially the Principal, Fr. Arthur Lai Fook (aka Jap), and it was a brave student who responded to the request by Fr. Pedro Valdez to define the dyne by answering, "Fah, it what happen after living". Fr. Pedro was a gentle soul although his brother, Fr. Toba, who taught me Latin, would lob a blackboard eraser with reproducible inaccuracy at any student who had the temerity to doze off during the Second Punic War while Hannibal and his elephants were steamrollering the hapless legions of Gaius Flaminius into Lake Trasimene. At least we didn't have detention at my school. Actually we did have detention only it was called 'penance'. Each and every student also had a Judgement Book in which was entered a mark (out of 10) for each subject each and every week. A mark of 5 (or less) or a failure to return one's Judgement Book, duly signed by parent or guardian, by Wednesday morning earned the transgressor a corrective package of Licks and Penance.  As a thoughtful child, I managed to shield my parents from this irksome bureaucracy and, in any case, it was simply safer that The Holy Ghost Fathers were never given the opportunity to familiarize themselves with the authentic parental signatures.

I used to think that 'Virtus et Scientia' was Latin for 'Licks and Penance'  (17-Feb-2018 update)

What we learned from the Holy Ghost Fathers was that most physical quantities have dimensions and if the quantities on the opposite sides of the 'equal sign' in an equation have different dimensions then it is a sign of an unforced error rather than a penetrating insight. For example the dimensions of force are MLT^-2 (M = mass; L = length; T = time) and you are free to express forces in newtons, dynes or poundals as you prefer. You can think of a physical quantity as a number multiplied by a unit and, without the unit, the number is meaningless. Units are extremely important but at the same time they are arbitrary in the sense that if your physical insight changes when you change a unit then it is neither physical nor an insight. Here's a good illustration of why dimensional analysis matters.

I have blogged ( 1 | 2 | 3 ) about how building the a concentration unit into the definition of ligand efficiency (LE) results in a metric that is physically meaningless (even though it remains a useful instrument of propaganda) and, for the masochists among you, there's also the LE metric critique in JCAMD. The problem can be linked to a lack of recognition of the fact that logarithms can only be calculated for numbers (which lack units). However, LE has another 'units issue' which is connected with the fact that it is a molar energy that is scaled in the definition of LE rather than pIC₅₀ or pK_d. This needn't be an issue but, unfortunately, it is. LE is defined by dividing a molar energy by the number of non-hydrogen atoms in the molecular structure and there is nothing in the definition of LE that says that the energy has to be expressed in any particular unit. This means that you can define LE using any energy unit that you want to. Some 'experts' appear to believe that dividing a molar energy by number of non-hydrogen atoms relieves them of the responsibility to report units. I'm referring, of course, to the practise of multiplying pIC₅₀ or pK_d by 1.37 when calculating LE. You might ask why people do this, especially given that 'experts' tout the simplicity of LE and they don't multiply pIC₅₀ or pK_d by 1.37 when they calculate LipE/LLE. Don't ask me because I'm neither expert nor 'expert'.

Let's take a look at this NRDD article on LE metrics and I'd like you to go straight to Box 1 (Ligand efficiency metrics). Six numbered equations are shown in Box 1 and it is stated towards the end of the first paragraph that "each equation corresponds to a mathematically valid function".  This statement is incorrect because the first equation (1) in Box 1 is not a mathematically valid function. The reason for this is that the logarithm function cannot take as its argument a quantity, such as K_d, that has units. Equation (5), which defines LLE_AT, is mathematically valid although it differs from the mathematically ambiguous equation that was originally used to define LLE_AT. 

To be honest, I think that Box 1 is probably beyond repair by conventional erratum and I'll back this opinion with an example:

"Assuming standard conditions of aqueous solution at 300K, neutral pH and remaining concentrations of 1M,
 –2.303RTlog(K_d/C°) approximates to –1.37 × log(K_d) kcal/mol." 

At my school in Trinidad this would have been called a 'ratch' and, once detected, it would have earned its perpetrator a corrective package of Licks and Penance. I don't think even the Holy Ghost Fathers could have exorcised a concentration unit quite this efficiently. 

In some physical chemistry literature, K_d is defined as a dimensionless quantity by including C° in the definition of K_d. However, in the literature of biochemistry, biophysics and medicinal chemistry,  K_d  is usually quoted in units of concentration. Binding free energy has the same value and same dependence on C° regardless of  which of the two conventions is used to define K_d . 

(Update 17-Feb-2018) 

I'd now like to talk a bit about the 'p' operator that we use to transform IC₅₀ and K_d values into logarithms. This makes it much easier to perceive structure-activity relationships and provides a better representation of measurement precision than when the IC₅₀ and K_d values themselves are used. To calculate pK_d,, first express K_d in molar concentration units, dump the units and calculate minus the logarithm of the number. I realize that this may come across as arm waving but the process of converting  K_d, to  pK_d, can actually be expressed exactly in mathematical terms as follows:

 pK_d = –log₁₀(K_d/M)

The 'p' operator has a 1 M concentration built into it. Although this choice of unit is arbitrary, it doesn't cause any problems if you're doing sensible things (e.g. subtracting them from each other) with the pK_d values. If, however, you're doing silly things (e.g. dividing them by numbers of non-hydrogen atoms) with the pK_d values then the plot starts to unravel faster than you can say 'Brexit means Brexit'. 

I'd like you take a look at another article which also has a Box 1 although I won't bother you with another tiresome 'spot the errors' quiz. The equation that I'll focus on is:

pK_d = pK_{H +} pK_S 

This equation describes the decomposition of affinity into enthalpic and entropic contributions and you might think this means that you can write:

K_d = K_H × K_S 

As Prof. Pauli would have observed, this is an error in the 'not even wrong' category and it is clear that a difference in opinion as to the importance of units was as much responsible for the unraveling of the Austro-Hungarian empire as that unfortunate wrong turn in pre-SatNav Sarajevo. The 'p' operator implies that each of K_d, K_H and K_S has units of concentration. However, multiplying two such quantities will give a quantity that has units of concentration squared. 

It is actually possible to decompose K_d into enthalpic and entropic contributions a valid manner but you need to be thinking carefully about the meaning of the standard state. As noted previously DG° depends on the concentration used to define the standard state. This is a consequence of the dependence of DS° on the standard concentration and DH is independent of the standard concentration (the standard state is assumed to be a dilute solution). This suggests defining K_S as quantity with units of concentration and K_H as a quantity without units.

This is probably a good point to wrap things up. My advice to all the authors of the featured NRDD and FMC articles is that they read (and make sure that they understand) the section of this article that is entitled '8. Ligand Efficiency and Additivity Analysis of Binding Free Energy'. This advice is especially relevant for those of the authors who consider themselves to be experts in thermodyamics.

May I wish all readers a happy, successful and metric-free 2017.