A PAINful itch

I've been meaning to take a look at the Seven Year Itch (SYI) article on PAINS for some time. SYI looks back over the the preceding 7 years of PAINS while presenting a view of future directions. One general comment that I would make of SYI is that it appears to try to counter criticisms of PAINS filters without explicitly acknowledging these criticisms.

This will a long post and strong coffee may be required. Before starting, it must be stressed that I neither deny that assay interference is a significant problem nor do I assert that compounds identified by PAINS filters are benign. The essence of my criticism of much of the PAINS analysis is that the rhetoric is simply not supported by the data.  It has always been easy to opine that chemical structures look unwholesome but it has always been rather more difficult to demonstrate that compounds are behaving pathologically in assays. One observation that I would make about modern drug discovery is that fact and opinion often become entangled to the extent that those who express (and seek to influence) opinions are no longer capable of distinguishing what they know from what they believe.

I've included some photos to break up the text a bit and these are from a 2016 visit to the north of Vietnam.  I'll start with this one taken from the western shore of Hoan Kiem Lake the night after the supermoon.

Hanoi moon

I found SYI to be something of a propaganda piece with all the coherence of a six-hour Fidel Castro harangue. As is typical for articles in the PAINS literature, SYI is heavy in speculation and opinion but is considerably lighter in facts and measured data. It wastes little time in letting readers know how many times the original PAINS article was cited. One criticism that I have made about the original PAINS article (that also applies to SYI and the articles in between) is that the article neither defines the term PAINS (other than to expand the acronym) nor does it provide objective criteria by which a compound can be shown experimentally to be (or not to be) a PAINS (or is that a PAIN). An 'unofficial' definition for the term PAINS has actually been published and I think that it's pretty good:

"PAINS, or pan-assay interference compounds, are compounds that have been observed to show activity in multiple types of assays by interfering with the assay readout rather than through specific compound/target interactions."

While PAINS purists might denounce the creators of the  unofficial PAINS definition for heresy and unspecified doctrinal errors, I would argue that the unofficial definition is more useful than the official definition (PAINS are pan-assay interference compounds). I would also point out that some of those who introduced the unofficial definition actually use experiments to study assay interference when much of the official PAINSology (or should that be PAINSomics) consists of speculation about the causes of  frequent-hitter behavior. One question that I shall put to you, the reader, is how often, when reading an article on PAINS, do you see real examples of experimental studies that have clearly demonstrated that specific compounds exhibit pan-assay interference?

Restored bunker and barbed wire at Strongpoint Béatrice which was the first to fall to the Viet Minh.

Although the reception of PAINS filters has generally been positive, JCIM has published two articles (the first by an Associate Editor of that journal and the second by me) that examine the PAINS filters critically from a cheminformatic perspective. The basis of the criticism is that the PAINS filters are predictors of frequent hitter behavior for assays using an AlphaScreen readout and they have been developed using proprietary data. It's a quite a leap from frequent-hitter behavior when tested at single concentrations in a panel of six AlphaScreen assays to pan-assay interference. In the language of cheminfomatics, we can state that the PAINS filters have been extrapolated out of a narrow applicability domain  and they have been reported (ref and ref) to be less predictive of frequent-hitter behavior in these situations. One point that I specifically made was that a panel of six assays all using the same readout is a suboptimal design of an experiment to detect and quantify pan-assay interference.

In my article, bad behavior in assays was classified as Type 1 ( assay result gives an incorrect indication of the extent to which the compound affects the function of the target) or Type 2 (compounds affect target function by an undesirable mechanism of action). I used these rather bland labels because I didn't want to become ensnared in a Dien Bien Phu of nomenclature and it must be stressed that there is absolutely no suggestion that other people use these labels. My own preference would actually be to only use the term interference for Type 1 bad behavior and it's worth remembering that Type 1 bad behavior can also lead to false negatives.

The distinction between Type 1 and and Type 2 behaviors is an important and useful one to make from the perspective of drug discovery scientists who are making decisions as to which screening hits to take forward. Type 1 behavior is undesirable because it means that you can't believe the screening result for hits but, provided that you can find an assay (e.g. label-free measurement of affinity) that is not interfered with, Type 1 behavior is a manageable, although irksome, problem. Running a second assay that uses an orthogonal readout may shed light on whether Type 1 behavior is an issue although, in some cases, it may be possible to assess, and even correct for, interference without running the orthogonal second assay. Type 2 behavior is a much more serious problem and a compound that exhibits Type 2 behavior needs to be put out of its misery as swiftly and mercifully as possible. The challenge presented by Type 2 behavior is that you need to establish the mechanism of action simply to determine whether or not it is desirable. Running a second assay with an orthogonal readout is unlikely to provide useful information since the effect on target function is real.

Barbed wire at Strongpoint Béatrice. I'm guessing that it was not far from here that, on the night of 13th/14th March, 1954, Captain Riès would have made the final transmission: "It's all over - the Viets are here. Fire on my position. Out."

Most (all?) of the PAINSology before SYI failed to make any distinction between Type 1 and Type 2 bad behavior. SYI states "There does not seem to be an industry-accepted nomenclature or ontology of anomalous binding behavior" and makes some suggestions as to how this state of affairs might be rectified. SYI recommends that "Actives" be first classified as "target modulators" or "readout modulators". The "target modulators" are all considered to be "true positives" and these are further classified as "true hits" or "false hits". All the "readout modulators" are labelled as "false positives". Unsurprisingly, the authors recommend that all the "false hits" and "false positives" be labelled as pan-assay interference compounds regardless of whether the compounds in question actually exhibit pan-assay interference. In general, I would advise against drawing a distinction between the terms "hit" and "positive" in the context of screening but, if you chose to do so, then you do really do need to define the terms much more precisely than the authors have done.

I think the term "readout modulator" is reasonable and is equivalent to my definition of Type 1 behavior (assay result gives an incorrect indication of the extent to which the compound affects the function of the target). However, I strongly disagree with the classification of compounds showing "non-specific interaction with target leading to active readout" as readout modulators since I'd regard any interaction with the target that affects its function to be modulation. My understanding is that the effects of colloidal aggregators on protein function are real (although not exploitable) and that it is often possible to observe reproducible concentration responses. My advice to the authors is that, if you're going to appropriate colloidal aggregators as PAINS, then you might at least put them in the right category.

While the term "target modulator" is also reasonable, it might not be a such great idea to use it in connection with assay interference since it's also quite a good description of a drug. Consider the possibility of homeopaths and anti-vaxxers denouncing the pharmaceutical industry for poisoning people with target modulators. However, I disagree with the use of the term "false hit" since the modulation of the target is real even when the mechanism of action is not exploitable. There is also a danger of confusing the "false hits" with the "false positives" and SYI is not exactly clear about the distinction between a "hit" and a "positive". In screening both terms tend to be used to specify results for which the readout exceeds a threshold value.

The defensive positions on one of the hills of Strongpoint Béatrice have not been restored. Although the trenches have filled in with time, they are not always as shallow as they appear to be in this photo (as I discovered when I stepped off the path).

It's now time to examine what SYI has to to say and singlet oxygen is as good a place as any to start from. One criticism of PAINS filters that I have made, both in my article and the Molecular Design blog, is that some of the frequent-hitter behavior in the PAINS assay panel may be due to quenching or scavenging of singlet oxygen which is an essential component of the the AlphaScreen readout. SYI states:

"However, while many PAINS classes contain some member compounds that registered as hits in all the assays analyzed and that therefore could be AlphaScreen-specific signal interference compounds, most compounds in such classes signal in only a portion of assays. For these, chemical reactivity that is only induced in some assays is a plausible mechanism for platform-independent assay interference."

The authors seem to be interpreting the observation that a compound only hits in a portion of assays as evidence for platform-independent assay interference. This is actually a very naive argument for a number of reasons. First, compounds do not all appear to have been assayed at the same concentration in the original PAINS assay panel and there may be other sources of variation that were not disclosed. Second, different readout thresholds may have been used for the assays in the panel and noise in the readout introduces a probabilistic element to whether or not the signal for a compound exceeds the threshold. Last, but definitely not least, the molecular structure of a compound does influence the efficiency with which it quenches or scavenges singlet oxygen. A recent study observed that PAINS "alerts appear to encode primarily AlphaScreen promiscuous molecules". 

If you read enough PAINS literature, you'll invariably come across sweeping generalizations made about PAINS. For example, it has been claimed that "Most PAINS function as reactive chemicals rather than discriminating drugs." SYI follows this pattern and asserts:

"Another comment we frequently encounter and very relevant to this journal is that PAINS may not be appropriate for drug development but may still comprise useful tool compounds. This is not so, as tool compounds need to be much more pharmacologically precise in order that the biological responses they invoke can be unambiguously interpreted."

While it is encouraging that the authors have finally realized the significance of the distinction between readout modulators and target modulators, they don't seem to be fully aware of the implications of making this distinction. Specifically, one can no longer make the sweeping generalizations about PAINS that are common in PAINS literature. Consider a hypothetical compound that is an efficient quencher of singlet oxygen and that has shown up as a hit in all six AlphaScreen assays of the original PAINS assay panel. While many would consider this compound to be a PAINS (or PAIN), I would strongly challenge a claim that observation of frequent-hitter behavior in this assay panel would be sufficient to rule out the use of the compound as a tool.

SYI notes that PAINS are recognized by other independently developed promiscuity filters.

"The corroboration of PAINS classes by such independent efforts provides strong support for the structural filters and subsequent recognition and awareness of poorly performing compound classes in the literature. It is instructive therefore to introduce two more recent and fully statistically validated frequent-hitter analytical methods that are assay platform-independent. The first was reported in 2014 by AstraZeneca(16) and the second in 2016 by academic researchers and called Badapple.(27)"

I don't think it is particularly surprising (or significant) that some of the PAINS classes are recognized as frequent-hitters by other models for frequent-hitter behavior. What is not clear is how many of the PAINS classes are recognized by the other frequent-hitter models or how 'strong' the recognition is. I would challenge the description of the AstraZeneca frequent-hitter model as "fully statistically validated" since validation was performed using proprietary data. I made a similar criticism of the original PAINS study and would suggest that the authors take a look at what this JCIM editorial has to say about the use of proprietary data in modeling studies.       

The French named this place Eliane and it was quieter when I visited than it would have been on 6th May, 1954 when the Viet Minh detonated a large mine beneath the French positions. It has been said that the alphabetically-ordered (Anne-Marie to Isabelle) strongpoints at Dien Bien Phu were named for the mistresses of the commander, Colonel (later General) Christian de Castries although this is unlikely.

SYI summarizes as follows:

"In summary, we have previously discussed a variety of issues key to interpretation of PAINS filter outputs, ranging from HTS library design and screening concentration, relevance of PAINS-bearing FDA-approved drugs, issues in SMARTS to SLN conversion, the reality of nonfrequent hitter PAINS, as well as PAINS and non-PAINS that are respectively not recognized or recognized in the PAINS filters as originally published. However, nowhere has a discussion around these key principles been summarized in one article, and that is the point of the current article. Had this been the case, we believe some recent contributions to the literature would have been more thoughtfully directed. (21,32)"

I must confess that reference to the reality of nonfrequent hitter pan assay interference compounds would normally prompt me to advise authors to stay off the peyote until the manuscript has been safely submitted. However, the bigger problem embedded in the somewhat Rumsfeldesque first sentence is that you need objective and unambiguous criteria by which compounds can be determined to be PAINS or non-PAINS before you can talk about "key principles". You also need to acknowledge that interference with readout and and undesirable mechanisms of action are entirely different problems requiring entirely different solutions.

I noted that recent contributions to the literature from me and from a JCIM Associate Editor (who might know a bit more about cheminformatics than the authors) were criticized for being insufficiently thoughtful. To be criticized in this manner is, as the late, great Denis Healey might have observed, "like being savaged by a dead sheep". Despite what the authors believe, I can confirm that my contribution to the literature would have been very similar even if SYI had been published beforehand. Nevertheless, I would suggest to the authors that dismissing the feedback from a JCIM Associate Editor as if he were a disobedient schoolboy might not have been such a smart move. For example, it could get the JMC editors wondering a bit more about exactly what they'd got themselves into when they decided to endorse a frequent-hitter model as a predictor of pan-assay interference. The endorsement of a predictive model by a premier scientific journal represents a huge benefit to the creators of the model but the flip side is that it also represents a huge risk to the journal. 

So that's all that I want to say about PAINS and it's a  good point to wrap things up so that I can return to Vietnam for the remainder of the post.       

I'm pretty sure that neither General Giap nor General de Castries visited the summit of Fansipan which at 3143 meters is the highest point in Vietnam (I wouldn't have either had a cable car had not been installed a few months before I visited). It's a great place to enjoy the sunset.

Back in Hanoi, I attempted to pay my respects to Uncle Ho, as I've done on two previous visits to this city, but timing was not great (they were doing the annual formaldehyde change). Uncle Ho is in much better shape than Chairman Mao who is actually seven years 'younger' and this is a consequence of having been embalmed by the Russians (the acknowledged experts in this field). Chairman Mao had the misfortune to expire when Sino-Soviet relations were particularly frosty and his pickling was left to some of his less expert fellow citizens. It is also said that the Russian embalming team arrived in Hanoi before Uncle Ho had actually expired...

Catching up with Uncle Ho

   

PAINS and editorial policy

<< previous || next >>

I have blogged previously (1 | 2 | 3 | 4 | 5 ) on PAINS. In this post, I present the case against inclusion of PAINS criteria in the Journal of Medicinal Chemistry (JMC) Guidelines for Authors (viewed 22-Jan-2017) as given below:

"2.1.9. Interference Compounds. Active compounds from any source must be examined for known classes of assay interference compounds and this analysis must be provided in the General Experimental section. Compounds shown to display misleading assay readouts by a variety of mechanisms include, but are not limited to, aggregation, redox activity, fluorescence, protein reactivity, singlet-oxygen quenching, the presence of impurities, membrane disruption, and their decomposition in assay buffer to form reactive compounds. Many of these compounds have been classified as Pan Assay Interference Compounds (PAINS; see Baell & Holloway, J. Med. Chem. 2010, 53, 2719-2740 and webinar at bit.lyj/mcPAINS). Provide firm experimental evidence in at least two different assays that reported compounds with potential PAINS liability are specifically active and their apparent activity is not an artifact."

The term 'known classes of assay interference compounds' must be defined more precisely in order to be usable by both authors submitting manuscripts and reviewers of those manuscripts. Specifically, the term 'known classes of assay interference compounds' implies the existence of a body of experimental data in the public domain for which specific substructural features have been proven to cause the observed assay interference. The JMC Guidelines for Authors (viewed 22-Jan-2017) imply that any assay result for a compound with 'potential PAINS liability' should necessarily be treated as less informative than would be the case if the compound did not have 'potential PAINS liability'. I shall term this as 'devaluing' the assay result.              

The PAINS acronym stands for Pan Assay INterference compoundS and it was introduced in a 2010 JMC article (BH2010) that is cited in the Guidelines for Authors (viewed 22-Jan-2017). The PAINS filters introduced in the BH2010 study are based on analysis of frequent-hitter behavior in a panel of 6 AlphaScreen assays. Each PAINS filter consists of a substructural pattern and is associated with an enrichment factor that quantifies the frequent hitter behavior. Compounds that quench or scavenge singlet oxygen have the potential to interfere with AlphaScreen assays but individual PAINS substructural patterns were not evaluated for their likelihood of being associated with singlet oxygen quenching or scavenging. For example, the BH2010 study makes no mention of studies ( 1 | 2 | 3 | 4 ) linking singlet oxygen quenching/scavenging to the presence of a thiocarbonyl group which is a substructural element present in rhodanines.  

I argue that a high hit rate against a small panel of assays that all use a single detection technology is inadmissible as evidence for pan-assay interference.  I also argue that the results of screening against this assay panel can only be invoked to devalue the result from an AlphaScreen assay. In a cheminformatic context, the applicability domain of a model based on analysis of results from this assay panel is restricted to activity measured in AlphaScreen assays. Furthermore, it is questionable whether it is valid to invoke the results of screening against this assay panel to devalue a concentration response from an AlphaScreen assay because the results for each assay of the panel were obtained at a single concentration (i.e. no concentration response).

The BH2010 study does present some supporting evidence that compounds matching PAINS substructural patterns are likely to interfere with assays. In a cheminformatic context, this supporting evidence can be considered to extend the applicability domain of PAINS filters. However, supporting evidence is only presented for some of the substructural patterns and much of that supporting evidence is indirect and circumstantial. For example, the observation that rhodanines as a structural class have been reported as active against a large number of targets is, at best, indirect evidence for frequent hitter behavior which is characterized by specific compounds showing activity in large numbers of assays. There is not always a direct correspondence between PAINS substructural patterns and those used in analyses that are presented as supporting evidence. For example, the BH2010 study uses substructural patterns for rhodanines that specify the nature of C5 (either saturated or with exocyclic carbon-carbon bond).  However, the sole rhodanine definition given in the BMS2006 study specifies an exocyclic carbon-carbon double bond. This means that it is not valid to invoke the BMS2006 study to devalue the result of every assay performed on any rhodanine.

The data (results from 6 AlphaScreen assays and associated chemical structures) that form the basis of the analysis in the BH2010 study are not disclosed and must therefore be considered to be proprietary. Furthermore, some of the supporting evidence that compounds matching PAINS filters are likely to interfere with assays is itself based on analysis (e.g. BMS2006 and Abbott2007) of proprietary data. The JMC Guidelines for Authors (viewed 22-Jan-2017) make it clear that the use of proprietary data is unacceptable:

"2.3.5.2 Proprietary Data. Normally, the use of proprietary data for computational modeling or analysis is not acceptable because it is inconsistent with the ACS Ethical Guidelines. All experimental data and molecular structures used to generate and/or validate computational models must be reported in the paper, reported as supporting information, or readily available without infringements or restrictions. The Editors may choose to waive the data deposition requirement for proprietary data in a rare case where studies based on very large corporate data sets provide compelling insight unobtainable otherwise.

2.3.6 QSAR/QSPR and Proprietary Data. The following are general requirements for manuscripts reporting work done in this area:

(3) All data and molecular structures used to carry out a QSAR/QSPR study are to be reported in the paper and/or in its supporting information or should be readily available without infringements or restrictions. The use of proprietary data is generally not acceptable."

Given JMC's stated unacceptability of analysis based on proprietary data, to use such analysis to define editorial policy would appear to contradict that editorial policy.

To sum up:

• Analysis of the screening results for the BH2010 assay panel can only be invoked  invoked to devalue or otherwise invalidate the result from an AlphaScreen assay.

• Additional supporting evidence is only provided in BH2010 for some of the PAINS filters. In these cases, the evidence is not generally presented in a manner that would allow a manuscript reviewer to assess risk of assay interference in an objective manner.

• Most of the analysis presented in the BH2010 study has been performed on proprietary data. To base JMC editorial policy on analysis of proprietary data would appear to contradict the Journal's policy on the use of proprietary data.

I rest my case.

Is the literature polluted by singlet oxygen quenchers and scavengers?

<<   previous || next  >> 

So apparently I’m a critic of the PAINS concept so maybe it’s a good idea to state my position.  Firstly, I don’t know exactly what is meant by ‘PAINS concept’ so, to be quite honest, it is difficult to know whether or not I am a critic. Secondly, I am fully aware that many compounds are observed as assay hits for any of a number of wrong reasons and completely agree that it is important to understand the pathological behavior of compounds in assays so that resource does not get burned unnecessarily. At the same time we need to think more clearly about different types of behavior in assays.  One behavior is that the compound does something unwholesome to a protein and, when this is the case, it is absolutely correct to say, ‘bad compound’ regardless of what it does (or doesn't) do to other proteins.  Another behavior is that the compound interferes with the assay but leaves the target protein untouched and, in this case, we should probably say ‘bad assay’ because the assay failed to conclude that the protein has emerged unscathed from its encounter with the compound. It is usually a sign of trouble when structurally-related compounds show activity in a large number of assays but there are potentially lessons to be learned by those prepared to look beyond hit rates. If the assays that are hit are diverse in type then we should be especially worried about the compounds.   If, however, the assays that are hit are of a single type then perhaps the specific assay type is of greater concern. Even when hit rates are low, appropriate analysis of the screening output may still reveal that something untoward is taking place. For example, a high proportion of hits in common may reflect that a mechanistic feature (e.g catalytic cysteine) is shared between two enzymes (e.g. PTP and cysteine protease)  

While I am certainly not critical of attempts to gain a greater understanding of screening output, I have certainly criticized over-interpretation of data in print ( 1 | 2 ) and will continue to do so.  In this spirit, I would challenge the assertion, made in the recent Nature PAINS article that “Most PAINS function as reactive chemicals rather than discriminating drugs” on the grounds that no evidence is presented to support it.  As noted in a previous post, the term ‘PAINS’ was introduced to describe compounds that showed frequent-hitter behavior in a panel of six AlphaScreen assays and this number of assays would have been considered a small number even two decades ago when some of my Zeneca colleagues (and presumably our opposite numbers elsewhere in Pharma) started looking at frequent-hitters. After reading the original PAINS article, I was left wondering why only six of 40+ screens were used in the analysis and exactly how these six screens had been selected.  The other point worth reiterating is that only including a single type of assay in analysis like this makes it impossible to explore the link between frequent-hitter behavior and assay type. Put another way, restricting analysis to a single assay type means that the results of the analysis constitute much weaker evidence that compounds interfere with other assay types or are doing something unpleasant to target proteins.

I must stress that I’m definitely not saying that the results presented in the original PAINS article are worthless. Knowledge of AlphaScreen frequent-hitters is certainly useful if you’re running this type of assay.  I must also stress that I’m definitely not claiming that AlphaScreen frequent hitters are benign compounds.  Many of the chemotypes flagged up as PAINS in that article look thoroughly nasty (although some, like catechols, look more ‘ADMET-nasty’ than ‘assay-nasty’).  However, the issue when analyzing screening output is not simply to be of the opinion that something looks nasty but to establish its nastiness (or otherwise) definitively in an objective manner.   

It’s now a good time to say something about AlphaScreen and there’s a helpful graphic in Figure 3 of the original PAINS article. Think of two beads held in proximity by the protein-protein interaction that you’re trying to disrupt.  The donor bead functions as a singlet oxygen generator when you zap it with a laser. Some of this singlet oxygen makes its way to the acceptor bead where its arrival is announced with the emission of light.  If you disrupt the protein-protein interaction then the beads are no longer in close proximity and the (unstable) singlet oxygen doesn’t have sufficient time to find an acceptor bead before it is quenched by solvent.  I realize this is a rushed explanation but I hope that you’ll be able to see that disruption of the protein-protein interaction will lead to a loss of signal because most of the singlet oxygen gets quenched before it can find an acceptor bead.

I’ve used this term ‘quench’ and I should say a bit more about what it means.  My understanding of the term is that it describes the process by which a compound in an excited state is returned to the ground state and it can be thought of as a physical rather than chemical process, even though intermolecular contact is presumably necessary.  The possibility of assay interference by singlet oxygen quenchers is certainly discussed in the original PAINS article and it was noted that:

“In the latter capacity, we also included DABCO, a strong singlet oxygen quencher which is devoid of a chromophore, and diazobenzene itself”

An apparent IC50 of 85 micromolar was observed for DABCO in AlphaScreen and that got me wondering about what the pH of the assay buffer might have been.  The singlet oxygen quenching abilities of DABCO have been observed in a number of non-aqueous solvents which suggests that the neutral form of DABCO is capable of quenching singlet oxygen.  While I don’t happen to know if protonated DABCO is also an effective quencher of singlet oxygen, I would expect (based on a pKa of 8.8) the concentration of the neutral form in an 85 micromolar solution of DABCO buffered at neutral pH to be about 1 micromolar.   Could this be telling us that quenching of singlet oxygen in AlphaScreen assays is possibly a bigger deal than we think?

Compounds can also react with singlet oxygen and, when they do so, the process is sometimes termed ‘scavenging’. If you just observe the singlet oxygen lifetimes, you can’t tell whether the singlet oxygen is returned harmlessly to its ground state or if a chemical reaction occurs.  Now if you read enough PAINS articles or PAINS-shaming blog posts, you’ll know that there is a high likelihood that, at some point, The Great Unwashed will be castigated for failing to take adequate notice of certain articles deemed to be of great importance by The Establishment.  In this spirit, I’d like to mention that compounds with sulfur doubly bonded to carbon have been reported ( 1 | 2 | 3 | 4 | 5 ) to quench or scavenge singlet oxygen and this may be relevant to the ‘activity’ of rhodanines in AlphaScreen assays.

The original PAINS article is a valuable compilation of chemotypes associated with frequent-hitter behavior in AlphaScreen assays although I have questioned whether or not this behavior represents strong evidence that compounds are doing unwholesome things to the target proteins.  It might be prudent to check the singlet oxygen quencher/scavenger literature a bit more carefully before invoking a high hit rate in a small panel of AlphaScreen assays in support of assertions that literature has been polluted or that somebody’s work is crap.  I’ll finish the post by asking whether tethering donor and acceptor beads covalently to each other might help identify compounds that interfere with AlphaScreen by taking out singlet oxygen. Stay tuned for the next blog post in which I’ll show you, with some help from Denis Healey and the Luftwaffe, how to pollute the literature (and get away with it).         

It's a rhodanine... fetch the ducking stool

| next >>

So I promised do a blog post on rhodanines. 

This isn’t really a post on rhodanines or even PAINS.  It’s actually a post on how we make decisions in drug discovery.  More specifically, the post is about how we use data analysis to inform decisions in drug discovery. It was prompted by a Practical Fragments post which I found to be a rather vapid rant that left me with the impression that a bandwagon had been leapt upon with little idea of whence it came or whither it was going.   I commented and suggested that it might be an idea to present some evidence in support of the opinions presented there and my bigger criticism is of the reluctance to provide that evidence.  Opinions are like currencies and to declare one’s opinion to be above question is to risk sending it the way of the Papiermark.

However, the purpose of this post is not to chastise my friends at Practical Fragments although I do hope that they will take it as constructive feedback that I found their post to fall short of the high standards that the drug discovery community has come to expect of PracticalFragments.   I’ll start by saying a bit about PAINS which is an acronym for Pan Assay INterference compoundS and it is probably fair to say that rhodanines are regarded as the prototypical PAINS class.  The hydrogen molecule of PAINS even?  It’s also worth stating that the observation of assay interference does not imply that a compound in question is actually interacting with a protein and I’ll point you towards a useful article on how to quantify assay interference (and even correct for it when it is not too severe).  A corollary of this is that we can’t infer promiscuity (as defined by interacting with many proteins) or reactivity (e.g. with thiols) simply from the observation of a high hit rate.  Before I get into the discussion, I’d like you to think about one question.  What evidence do you think would be sufficient for you to declare the results of a study to be invalid simply on the basis of a substructure being present in the molecular structure of compound(s) that featured in that study?

The term PAINS was introduced in a 2010 JMC article about which I have already blogged.  The article presents a number of substructural filters which are intended to identify compounds that are likely to cause problems when screened and these filters are based on analysis of the results from six high throughput screening (HTS) campaigns.  I believe that the filters are useful and of general interest to the medicinal chemistry community but I would be wary of invoking them when describing somebody’s work as crap or asserting that the literature was being polluted by the offending structures.   One reason for this is that the PAINS study is that it is not reproducible and this limits the scope for using it as a stick with which to beat those who have the temerity to use PAINS in their research.  My basis for asserting that the study is not reproducible is that chemical structures and assay results are not disclosed for the PAINS and neither are the targets for three of the assays used in the analysis.  There are also the questions of why the output from only six HTS campaigns was used in the analysis and how these six were chosen from the 40+ HTS campaigns that had been run.  Given that all six campaigns were directed at protein-protein interactions employing AlphaScreen technology, I would also question the use of the term ‘Pan’ in this context.  It’s also worth remembering that sampling bias is an issue even with large data sets.  For example, one (highly cited) study asserts that pharmacological promiscuity decreases with molecular weight while another (even more highly cited) study asserts that the opposite trend applies.

This is probably a good point for me to state that I’m certainly not saying that PAINS compounds are ‘nice’ in the context of screening (or in any other context).  I’ve not worked up HTS output for a few years now and I can’t say that I miss it.  Generally, I would be wary of any compound whose chemical structure suggested that it would be electrophilic or nucleophilic under assay conditions or that it would absorb strongly in the uv/visible region or have ‘accessible’ redox chemistry. My own experience with problem compounds was that they didn’t usually reveal their nasty sides by hitting large numbers of assays.  For example, the SAR for a series might be ‘flat’ or certain compounds might be observed to hit mechanistically related assays (e.g. cysteine protease and a tyrosine phosphatase).  When analyzing HTS results the problem is not so much deciding that a compound looks ‘funky’ but more in getting hard evidence that allows you to apply the molecular captive bolt with a clear conscience (as opposed to “I didn’t like that compound” or “it was an ugly brute so I put it out of its misery” or “it went off while I was cleaning it”).

This is a good point to talk about rhodanines in a bit more detail and introduce the concept of substructural context which may be unfamiliar to some readers and I'll direct you to the figure above.  Substructural context becomes particularly important if you’re extrapolating bad behavior observed for one or two compounds to all compounds in which a substructure is present. Have a look at the four structures in the figure and think about what they might be saying to you (if they could talk).  Structure 1 is rhodanine itself but a lot of rhodanine derivatives have an exocyclic double bond as is the case for structures 2 to 4.  The rhodanine ring is usually electron-withdrawing which means that a rhodanine with an exocyclic double bond can function as a Michael acceptor and nucleophilic species like thiols can add across the exocyclic double bond.  I pulled structure 3 from the PAINS article and it is also known as WEHI-76490 and I’ve taken the double bond stereochemistry to be as indicated in the article.  Structure 3 has a styryl substituent on the exocyclic double bond which means that it is a diene and has sigmatropic options that are not available to the other structures.  Structure 4, like rhodanine itself, lacks a substituent on the ring nitrogen and this is why I qualified ‘electron-withdrawing’ with ‘usually’ three sentences previously.   I managed to find a pKa of 5.6 for 4 and this means that we’d expect the compound to predominantly deprotonated at neutral pH (bear in mind that some assays are run at low pH).   Any ideas about how deprotonation of a rhodanine like 4 would affect its ability to function as a Michael acceptor?  As an aside, I would still worry about a rhodanine that was likely to deprotonate under assay conditions but that would be going off on a bit of a tangent.

Now is a good time to take a look at how some of the substructural context of rhodanines was captured in the PAINS paper and we need to go into the supplemental information to do this.  Please take a look a the table above.  I’ve reconstituted a couple of rows from the relevant table in the supplemental material that is provided with the PAINS article.  You’ll notice is that there are two rhodanine substructural definitions, only one of which has the exocyclic double bond that would allow it to function as a Michael acceptor.  The first substructure matches the rhodanine definitions for the 2006 BMS screening deck filters although the 2007 Abbott rules for compound reactivity to protein thiols allow the exocylic double bond to be to any atom.  Do you think that the 60 compounds, matching the first substructure, that fail to hit a single assay should be regarded as PAINS?  What about the 39 compounds that hit a single assay?   You’ll also notice that the enrichment (defined as the ratio of the number of compounds hitting two to six assays to the number of compounds hitting no assays) is actually greater for the substructure lacking the exocyclic double bond.   Do you think that it would appropriate to invoke the BMS filters or Abbott rules as additional evidence for bad behavior by compounds in the second class?  As an aside it is worth remembering that forming a covalent bond with a target is a perfectly valid way to modulate its activity although there are some other things that you need to be thinking about. 

I should point out that the PAINS filters do provide a richer characterization of substructure than what I have summarized here.   If doing HTS, I would certainly (especially if using AlphaScreen) take note if any hits were flagged up as PAINS  but I would not summarily dismiss somebody's work as crap simply on the basis that they were doing assays on compounds that incorporated a rhodanine scaffold.  If I was serious about critiquing a study, I’d look at some of the more specific substructural definitions for rhodanines and try to link these to individual structures in the study.   However, there are limits to how far you can go with this and, depending on the circumstances, there are number of ways that authors of a critiqued study might counter-attack.  If they’d not used AlphaScreen and were not studying protein-protein interactions, they could argue irrelevance on the grounds that the applicability domain of the PAINS analysis is restricted to AlphaScreen being used to study protein-protein interactions.  They could also get the gloves off and state that six screens, the targets for three of which were not disclosed, are not sufficient for this sort of analysis and that the chemical structures of the offending compounds were not provided.  If electing to attack on the grounds that this is the best form of defense, they might also point out that source(s) for the compounds were not disclosed and it is not clear how compounds were stored, how long they spent in DMSO prior to assay and exactly what structure/purity checks were made.

However, I created this defense scenario for a reason and that reason is not that I like rhodanines (I most certainly don’t).   Had it been done differently, the PAINS analysis could have been a much more effective (and heavier) stick with which to beat those who dare to transgress against molecular good taste and decency.   Two things needed to be done to achieve this.  Firstly, using results from a larger number of screens with different screening technologies would have gone a long way to countering applicability domain and sampling bias arguments.  Secondly, disclosing the chemical structures and assay results for the PAINS would make it a lot easier to critique compounds in literature studies since these could be linked by molecular similarity (or even direct match) to the actual ‘assay fingerprints’ without having to worry about the subtleties (and uncertainties) of substructural context. This is what Open Science is about.

So this is probably a good place to leave things.  Even if you don't agree with what I've said,  I hope that this blog post will have at least got you thinking about some things that you might not usually think about. Also have another think about that question I posed earlier. What evidence do you think would be sufficient for you to declare the results of a study to be invalid simply on the basis of a substructure being present in the molecular structure of compound(s) that featured in that study?