In which I’m pretty sure I disagree with Lior Pachter and try to figure out why

[Note 6/14/15. Before reading this, see the comment thread here.]

I recently read a thoughtful blog post by Lior Pachter on the importance of admitting errors in one’s work. There’s much to agree with in the post, but on first reading something seemed slightly off to me, though I couldn’t quite figure it out.

I just got off a long flight where I could sit down and read the post over again, and I think I have a better sense of what was bugging me. In this post, I’m jotting down a few thoughts on the last part of Lior’s post, the subsection titled “The personal critique of professional conduct”. Specifically, I think he’s guilty of some serious dissembling in this section and I would prefer he be more direct.

The claim

Lior argues that we should not be afraid to criticize the professional conduct of our colleagues in a personal manner. That is, if people refuse to admit errors, or oversell their work to the popular press, we should call them out on it. And if people benefit (via grants, jobs, etc.) from questionable conduct, then this isn’t a purely scientific issue, it’s partially a personal issue, and discussing these personal issues shouldn’t be taboo.

It’s of course hard to argue with some aspect of this claim, almost to the point that it’s a bit of a straw man (is it really considered taboo to criticize people for overselling their work? This type of criticism is like half of my Twitter feed).

An observation

This is where I think Lior is dissembling; my reading of his blog is actually rather different than what is being claimed. Let’s take a recent example:

In the post The most embarrassing citation ever?, Lior takes aim at a paper with a bone-headed citation error. As the title of the post points out, this was indeed an embarrassing error. The text of the post also points out how embarrassing the error was. For good measure, he includes a cute internet poll where readers can vote on just exactly how embarrassing this error was. Get it? The citation was an error! And it was was embarrassing!

The (presumably appropriately-shamed) student and her advisor actually do acknowledge the error in the comments of the post, but no one seems to really care, because let’s admit it–this post obviously isn’t really about a mistake in a citation. Nor is it about criticizing anyone’s professional conduct, or any such high-minded pursuit. It seem clear (to me at least) that the real goal here is to mock people Lior sees as his adversaries, in this case members of the GTEx Consortium and their colleagues.

My claim

So I’m going to make a claim, which I think is consistent with the observations. Lior is saying something (via his actions) much stronger than “we should feel free to criticize the professional conduct of our colleagues”. Rather: if you think your colleagues are undeservedly sucking up resources like grant money, then you should Take. Them. Down. By any means necessary. Accuse them of fraud. Call them innumerate. Mock their minor errors [3]. Is your colleague a male that asks lots of questions at meetings? Perfect, imply he’s holding back female scientists. Very little is off limits–once a person “deserves it”, then the gloves come off.

Lior clearly goes back and forth on whether he should openly endorse the view that public mockery and derision of “deserving” people should be part of the standard toolkit in scientific discussion. I think he should be honest and embrace it.

Some problems

I see two potential problems, however, with this approach. First is how one decides who is “deserving”. Lior seems to mostly dislike 1) overhyped claims that don’t stand up to scrutiny and 2) people who don’t use Cufflinks [1]. Dan Graur, on the other hand, thinks scientists with lots of middle author publications deserve to be put in their place [2], as do people who use too many big words in their papers. But I suppose everyone could use their own standards as they see fit.

The second problem for me is that a “culture of denunciation” sounds rather unpleasant. As I’m building a lab, I’m trying to convince very intelligent people to turn down excellent jobs outside of academia in exchange for the intellectual community that we provide. Robust and even intellectually-aggressive criticism is obviously part of that community. But are we really saying that being publicly denounced as an ‘abuser of science‘ is standard rhetoric now? Will my future advice to students be “If someone you dislike gets a grant, accuse them of fraud and–who knows–maybe it’ll stick”? What person in their right mind would possibly want to join such a community?

[1] Clarification: the Cufflinks remark is meant entirely tongue-in-cheek. I admit I chuckled while writing it, but your mileage may vary.

[2] Correction: This sentence used to read “women with lots of middle author publications”. A commenter writes (correctly) that a more generous interpretation of this post is that Dan is singling out a single woman, rather than “women”, and that his argument is that scientists with middle author publications should know their place in the “scientific hierarchy”. I have edited the post to reflect this correction, and I regret the error.

[3] Clarification: in response to a comment from Lior Pachter, I have added a link to this sentence (I’d thought the reference to the previous section was clear).

Advertisements

62 thoughts on “In which I’m pretty sure I disagree with Lior Pachter and try to figure out why

  1. Thanks for this post, it’s very accurate IMO. Lior’s blog is sometimes a good read, there are some purely scientific posts which are worthwhile. But these are not his popular posts. His particular brand of mixing ad hominem argument and peer review is what gets him notoriety (and one presumes, his kicks). Most anonymous postpub review has more merit than his blog, but he’s got the hyperbolic language (similar to the use of “striking”) and some celebrity status (similar to being a Kellis/Lander), so he gets more “citations.”

    It seems his original celebrity derives from his contribution to Cufflinks, a program which most biologists use incorrectly and prefer because others have used it and because it is highly cited. I’d love to see Lior pick apart papers that use Cufflinks improperly on his blog.

  2. I went to a talk where Lior was pretty lukewarm about Cufflinks, so I think his gripe is more accurately stated as “Don’t use tools that are demonstrably worse than other tools.” which sounds like a good plan to me.

    • Yes, I violated my own guidelines about avoiding jokes on the internet with that one, sorry. The Cufflinks thing was tongue-in-cheek.

  3. “I’d love to see Lior pick apart papers that use Cufflinks improperly on his blog.”

    A blog that regularly invites authors of popular software packages to do post-mortems on papers abusing their software would be very engaging.

  4. I think you are being too generous to Graur (in the addendum). He frames his post as an attempt to add to his list of great female scientists. Then he proceeds to eviscerate an attractive young woman (photo provided) described as ‘hot’, which of course could mean physically attractive and/or experiencing a meteoric career rise. The implication is she is not a serious scientist and has garnered attention due to her attractiveness. I have a hard time believing that if he truly wished to add to the list of female scientists who meet his exacting standards, he couldn’t find one or two with first/last author publications of major significance. He seems more interested in constructing a straw woman he can knock down.

  5. I just noticed Lior Pachter posted this comment on his blog earlier today, I’m copying it over to readers can follow the thread.

    Dear Joe,

    You make many claims in your blog post that are completely unfounded and presumptuous. You say that “I mostly dislike people who don’t use Cufflinks”. Could you please point to any evidence for this assertion? Nothing could be farther from the truth and I believe that my actions and statements on RNA-Seq and RNA-Seq software speak very clearly for themselves. I mean, do you think I hate myself? You say that I wrote ” The most embarrassing citation ever?” to mock people. On what basis do you say that? I made it very clear in my post that my concern was specifically with the practice of using software that is unpublished, and then citing it via websites. To wit, when the authors wrote to me personally to let me know they would ask the journal to “fix” the website link to the Flux Capacitor website, I replied that this was not the right thing to do, and that instead they should use published software (I personally believe even BiorXiv or arXiv is a publication; for Flux Capacitor there is literally nothing). Was their action embarrassing? Very much so. I highlighted it because it is relevant in the context of a general disdain for computational biology methods in biology papers. You say that I see members of the GTEx consortium as my “adversaries”. This could not be farther from the truth. A number of GTEx consortium members are my colleagues/collaborators and people I very much respect. You just made that up. You write that my objective is to “Take Them Down” (I assume your “them” refers to Manolis Kellis). It is of course your prerogative to presume whatever you wish, but you could have just asked me what my objective is instead. You mock my accusation of fraud (I assume you are referring to my post on the Network Nonsense of Manolis Kellis) but in fact, it is a very very serious accusation that I made carefully and after much consideration. I don’t think it, and the other criticisms I’ve posted constitute “minor errors”. If you think so then all I can suggest is please read the posts. Otherwise all you have done is slander.

    Lior

    • Lior,
      Interesting that you don’t actually say what the objective of repeatedly targeting Manolis Kellis’ work *is*. If the point of your blog is to highlight examples of science that you find substandard, why the relentless focus on the work of one person? Surely there are many other examples around that you could direct your energies towards? Also – if your motives are as high-minded as you claim – how do you not understand that the single-minded pursuit (some might say obsession) with the work of one person seriously undermines any claim you have to legitimacy. For one thing, it helps perpetuate that “devil / saint dichotomy” that you’ve said on your blog you’re trying to get away from. How does vilification of one person’s work achieve this, exactly? For another, repeated targeting of the same person looks suspiciously like a grubby personal vendetta, not some noble quest for truth. I’m not saying it *is*, but I don’t think it does you any favours.

      • To be fair to Lior, if a single person writes a bunch of papers he thinks are terrible, it’s fair to go down the line and criticize them all.

        But that type of exercise should speak for itself–what I have a visceral reaction to is dressing up the criticism with hyperbole and nasty personal flourishes, which IMO distract from the content in obviously unproductive ways.

      • Dear Joe,

        I think it would be helpful if you pointed to specific examples on my blog that you feel are hyperbole or that constitute “nasty personal flourishes”.

        Thanks,
        Lior

      • Hi Lior,

        I’ve written a blog post about this, and we’ve of course had this discussion before.

        Obviously I have a visceral reaction to your style, and I doubt we’re going to come to an agreement here.

        I hope I haven’t upset you too much by writing this, since of course I do appreciate many aspects of your blog. Ideally this might even qualify as a version of the “personal criticism of professional conduct” that you’re advocating :)

        Best,

        Joe

      • Ha! You certainly know how to defuse any disagreement here :)

        Absolutely looking forward to seeing you in August.

        Joe

  6. Hey Joe,

    Interesting exchange between you and Lior. I had one question/pushback for you. Do you think that perhaps the theatricality of Lior’s posts (i.e., offering cash prizes, online polls, attention-grabbing headlines) is required to counter the prestige of CNS or consortium paper(s)?

    • Hi Anon,

      That’s an idea worth considering, but I don’t think so. One specific thought:

      One of the best things about Lior’s blog is that he is directly critical of papers that he thinks are wrong. Too often (in my opinion) we privately tell our colleagues “This paper is nonsense”, but publicly say nothing or–if we choose to publish something–dress our objections up in dry academic prose: “it appears a suboptimal choice of parameter settings in the algorithm for processing of the original data may have led to a bias in…” or something similarly dull :)

      I think it would be great if people cut loose a little bit and didn’t feel the need to be overly academic when discussing papers publicly. If people felt comfortable (in blogs, on PubPeer, or wherever) saying “I think the main claims of this paper are wrong and here are the three reasons why”, I think that alone would be enough to get people reading and discussing. (See for example, the discussion described here).

      Joe

      • “That’s an idea worth considering, but I don’t think so.”

        Wow, what a cogent and trenchant analysis.

      • Hi Anon,

        Can you post under “Anon from Berkeley”, to contrast you with the original Anon? Or can you choose another handle if you comment again please? It’s sometimes useful to keep track of which comments are the same person.

        Thanks,

        Joe

  7. There is a very worrying trend going on in life sciences and also in the disciplines which used as tools in life sciences, like for example bioinformatics, statistics, systems biology, computational biology, etc. This trend is irreducible research. This is very well documented, like for example her e http://www.nature.com/news/reproducibility-1.17552 . This is very shocking experience for some one who comes from math/computer-science/statistics/mechanic into life science field to discover that scientific work/methods/experiments are not reproducible (due to reason like butter-fly effect explanations). There are many reasons (and I might say excuses) why there are irreproducible science going on. The problem is that this is quite well acknowledged fact in life sciences and there are examples where one’s newest-greatest mumbo-jumbo hypothese/method/algorithm is “hiding” behind this “irreducible life science”.

    I think that this is what happened to Lior. It is true that there are things which he could improve but overall I think that the way forward is to be critique about scientific work already published (or made publicly available) in order to stop spreading wrongness and make the life sciences more reproducible.

    Regarding the “most embarrassing citation ever” it is known tactic used in academia to cite a negative finding as a positive finding thru citation! I didn’t say this and I did not made this up! Read it for yourself here: Citation Bias – Confirmation Bias for Scientists http://theness.com/neurologicablog/index.php/citation-bias-confirmation-bias-for-scientists/

  8. There is typoe in my previous post. “This trend is irreducible research” should be “This trend is ireproducible research/science”.

  9. Aren’t you just as concerned that when you recruit people into permanent academia they’ll look around and say ‘gee, charlatans are kind of running amok here and face zero impunity and get all the accolades’. I don’t think the situation is so bad as that to be sure, and I think that the recent gilad et al case is probably closer to the ideal model than the more acerbic bloggers. And still synder comes out and demands to be coddled.

    Success is going to breed contempt and i think thats a good thing, because we shouldn’t keep lauding successful people or ideas without any real merit. And the rightly successful should be able to deal with it. The ultimate utility and legacy of the encode data has little to do with press releases and blog posts.

    If your recruits are heading to google or facebook to serve ads instead, they’ll face similar scathing critiques, and thank goodness.

    • Hi anon,

      Thanks, I think that’s a great point. But who defines “charlatan”?

      Here’s an observation: there are a few scientists who I used to hold in low regard, but now have a grudging respect for. Should I have denounced them as “charlatans”? A more worrying thought: if I had publicly denounced them as charlatans, would I still have changed my mind? Or would the psychology of a public statement have kept me from re-evaluating? Obviously rhetorical questions for myself, but this is where I’m coming from :)

      Joe

      • How to develop grudging respect for charlatan: Just keep reading their pnas papers until you see a good one! Tenure is also the great equalizer.

        And i think you’re being a bit unfair here in terms of the quality of discourse that pachter can achieve. Lander and kellis both freely commented on his blog and others chimed in with something resembling civility.

        These are the high profile cases, but throughout academia and certainly in the less hallowed institutions where concepts like openness and transparency and PPPR have yet to circulate the 50+ tenured set have a great deal of power due to their grant and publication history (much of it rightly earned). Should their be more avenues for publicly identifying the ones who aren’t worth their resources but who have entrenched power positions? I think so. Thats not exactly the conflict you’re discussing, but look at pachters commendable investigation of the King saud citation payment scheme. If your’e a young scientist thats gotta make you pull your hair out.

      • You are right though, people become too dismissive of a whole body of work. The encode haters are a great example. But i doubt Pachter is going to start abandoning the principles of shotgun sequencing just because he has concerns about landers neofunctionalization theory.

      • Hi anon,

        Yes, point taken about the “P-value prize” thread, that discussion was awesome.

        I’m less sure about the idea of targeting people in “entrenched power positions” who don’t deserve their resources. (And note Lior claims in this thread that his blog has nothing to do with taking people down). Obviously I need to think about it. But you start getting into these weird mental gymnastics where you compare what someone “deserves” versus where they are, which seems really tough to calibrate (and is obviously prone to subjective biases).

        Thanks again,

        Joe

  10. Actually Lior, to be accurate, what Joe has done would be libel, not slander, if it were defamatory. Joe, I don’t see as much evidence of disagreement as disapproval. It appears you don’t approve of the tone he uses in his critiques. Personally I think his posts are usually well thought out and supported, although I haven’t conducted an exhaustive analysis. The most common problem seems to be his critiques make people feel uncomfortable. It is true that the contempt he has for the various targets can be palpable. The question I have is whether that is a problem. Your concern about recruiting talented scientists for fear they might be afraid of criticism worries me. There is a lot that is awful in modern biomedical sciences, from the intentional, major research misconduct to more insidious things like failing to cite relevant work, to the unintentional, not thinking through a hypothesis, publishing an out of date data set, etc.

    I’m supportive of efforts to make people think about these problems, as long as the critiques ate substantive, and I’m less concerned about mollifying the targets. But given the comments I’ve seen on Lior’s blog I am probably in the minority.

    • Hi Brian,

      Thanks a lot for the comment. I think Lior’s posts are often thoughtful. But it’s sometimes difficult to know when he’s on less-firm ground because in order to get to his substantive arguments, sometimes you have to get through several layers of contempt. In this situation I think the overloaded rhetoric actually distracts from the discussion. I understand if you disagree with this.

      Re: “Your concern about recruiting talented scientists for fear they might be afraid of criticism worries me”. I think there’s a misunderstanding. Can you quote the section of the blog post where you got the idea that I have this “concern”? I don’t have any such concern, and will clarify the post if it comes across that way.

      Best,

      Joe

      • I was responding to this

        “As I’m building a lab, I’m trying to convince very intelligent people to turn down excellent jobs outside of academia in exchange for the intellectual community that we provide.”

        I was typing that this morning on my phone while walking the dogs, so I don’t think I dealt well with the context of your comment. I am in agreement that intelligent and creative scientists may choose to leave academics to participate in careers where they may get less critical feedback. It would have been more proper to say, “worries me too.” It was not your behavior that worried me, sorry if that was unclear.

        Scientists are barraged by evaluation (grants, papers, students, peers, etc.). In some cases the critique is valid, and we need to be able to move beyond the damage to our ego to make changes. In others, perhaps there is a misunderstanding that can only be addressed by two-way communication. That is a major failure of pre-publication anonymous review, it offers a chance to be a bad reviewer with little consequence. And all too often it is a one-way street with onus entirely on the author(s) to overcome bad reviewing.

        What I like about Lior’s approach is that we can all see the comments. If the community supports the author or paper, or finds the tone of the critiques is off, they can say so. Whether we as a community decide on new norms for how to have that discussion remains to be seen. But, I would rather we do so with the intention of creating a better scientific process then making people feel better about themselves even when they are in error.

      • DrWorms,

        Thanks for the clarification, I appreciate it. Agreed with everything you say, including “I would rather we do so with the intention of creating a better scientific process then making people feel better about themselves even when they are in error.”

        I think it’s important to build a system where admitting error is accepted. This is hard–Lior is advocating that we question whether people deserve their jobs and funding, and I understand where he’s coming from. But it seems this is almost exactly the way to ensure that no one ever admits an error. So…hm. I don’t pretend to have all the answers here.

        Joe

      • Unrelated note: what’s with everyone getting out their law books every time there’s a disagreement on the internet? Every time Lior writes a critical post, someone accuses him of defamation or something. Now I write a critical post and I’m accused of libel. Is this just an super-extra-fancy way of saying “you’re wrong”, or what?

        I have lots of lawyer friends, I wonder if we should try to get them to incorporate scientific language into their disagreements. Like they should say “you’re violating the 2nd law of thermodynamics” when they think something is stupid :)

  11. Lior, you wrote to Joe: “You write that my objective is to “Take Them Down” (I assume your “them” refers to Manolis Kellis). It is of course your prerogative to presume whatever you wish, but you could have just asked me what my objective is instead.”

    When I met you for the first time 1.5 years ago in Berkeley, you told me that “Manolis Kellis should lose his job” (direct quote) and continued to describe how you’ll use your blog to convince the world about this. I’ve heard that you’ve told similar things to several other members of the community – who are not close friends of yours – and of course the word gets around. Thus, I don’t think that you’re really in a position to criticize Joe for “presuming” that you want to take Manolis Kellis down, when you’ve told many people that this is exactly what you want.

    best,
    Tuuli Lappalainen

    • Dear Tuuli,

      I don’t dispute your assertion that it is my opinion that Manolis Kellis should lose his job. I do believe that is what should happen to faculty who commit fraud. But I don’t believe that I should be the arbiter of what should or shouldn’t happen to him and I certainly am not trying to play that role; ideally MIT and his funding agencies should have looked carefully at my claims but alas that seems not to have happened. I do believe I am having an impact with my blog, not just in terms of the specific paper by Manolis Kellis that was fraudulent, but in general via the process of post-publication peer review. So you are also right that I blog not only for myself (though admittedly I’ve learned a lot in the process) but for others as well. For example, I was pleased that Pavel Pevzner’s Coursera class had the opportunity to participate in the recent discussion on evolution and duplication as part of the “p-value prize”.

      What Joe wrote was that “if you think your colleagues are undeservedly sucking up resources like grant money, then you should Take. Them. Down. By any means necessary. Accuse them of fraud. Call them innumerate. Mock their minor errors” That is a gross mischaracterization of the objectives of my blog and also of the claims I made in my post on which I based my assertion of fraud:

      https://liorpachter.wordpress.com/2014/02/11/the-network-nonsense-of-manolis-kellis/

      In other words, Joe’s remarks were based on an incorrect presumption about what I am trying to do. His attempt to “fix” his statement by adding the link [3] is insufficient and inappropriate but at this point with all the updates and jokes I must admit I don’t really understand the point the post is trying to make anyway.

      Thanks,
      Lior

  12. Dear Lior,

    You know what your objectives are, but the rest of the world needs to infer them from your blog posts and other communication with you. If people interpret your message as something that you feel is a gross misrepresentation of what you are truly trying to say and do, this can be because 1) people are stupid and sloppy readers, 2) your blog posts give a wrong impression of your actual thoughts.

    If people misunderstand my writing, I tend to think that it’s my fault for not expressing myself clearly enough, instead to blaming the reader. You have chosen a confrontational style, which IS difficult to interpret. Based on what I’ve heard, a large number of people have the impression that one of your objectives is to “Take. Them. Down. By any means necessary” – and the vast majority of these people (Joe in particular) are not stupid or intentionally trying to harm you by twisting your message. If/since you feel that this is not a fair general judgement of your message, maybe you could be more upfront in the future about what you really are after.

    all the best,
    Tuuli

    • Thanks Tuuli. You make a very good point that when readers fail to understand or appreciate a point a writer trying to make the failure is generally with the writer and not the reader. In my case there have certainly been instances where my explanations and/or writing could have been better.

      Thanks again,
      Lior

      • “In my case there have certainly been instances where my explanations and/or writing could have been better.”

        Same for me.

        I withdraw all claims of knowing Lior’s intentions. He claims that I and others have wildly misinterpreted him, and I take him at his word. Lior, please accept my apology.

        I’m leaving the post public because I’ve found the discussion useful.

        Joe

  13. Pingback: To Maintain Integrity of US Science, Kellis and Birney Should be Investigated « Homolog.us – Bioinformatics

  14. Wow !!! What a surprise. Joe Pickrell is talking about Lior Pachter.

    Joe, I called you out several months back (http://www.homolog.us/blogs/blog/2014/02/12/lior-pachter-exposes/) and need to do it again. This time, Lior Pachter brought very serious charges against Kellis, and if you think he is doing it frivolously, you should make a case to ask to get him fired/rebuked. However, to do that, you need to actually read his network nonsense posts and provide proper critique of why you think they were written frivolously. Instead you write a lot of meaningless garbage in your usual style. Based on your writing, I do not see why Pachter’s accusations are wrong, and it makes me think that you agree with his charges against Kellis.

    I would also argue that if Pachter’s charges stand, the organizations hiring/supporting Kellis need to open some kind of investigation to show that they are serious organizations.

    http://www.homolog.us/blogs/blog/2015/06/12/to-maintain-integrity-of-us-science-kellis-and-birney-should-be-investigated/

    Joe, to be ethical, you also need to clearly mention what fraction of your funding comes from NHGRI, where Kellis seemed to have served on various advisory committees (Ref. Pachter).

  15. Thanks for writing this Joe. I don’t want to weigh into the “he said, s/he said”, but I do want to say that I also had general misgivings with the p-value prize and the announcement of its winner. Mainly because I think the entire episode wasn’t very constructive, nor do I think it could have been, which does lead me to question motivations (of Lior, commenters & rubber-neckers like myself).

    Firstly, extensive arguments about the “best” null model is exactly the kind of sterile discourse that gives evolutionary genetics a poor reputation outside the field [1]. What purpose did this discussion actually serve to advance knowledge? What purpose could it have served maximally? Did we actually learn any new biology here? Any new theory? I think not. The way I see it, we got a tutorial in the DDC model and conditional probability, plus another reminder that the review process for Nature papers can be weak and hard to correct by letters to the editor. That an ultimately sterile discourse was personal just makes the whole episode to me more Heat than light.

    Secondly, the premise of the p-value prize was to evaluate whether Kellis et al’s claim of “Strikingly,…” was justified or not. Ironically, this challenge was motivated by reading a preprint by Lan & Pritchard which makes IMO an similarly worded claim of “Surprisingly, many gene pairs instead show an unexpected asymmetric pattern of gene expression”. I couldn’t find a clear description of a model supporting this statement in the Lan & Pritchard paper or its supplement [2], but I might have missed it. Why do Lan & Pritchard get a pass, when Kellis et al. do not? Wouldn’t it have been more constructive to put the spotlight on Lan & Pritchard’s claim in a pro-active way to allow the community to contribute to evaluating this statement collectively? I’d say a good opportunity was missed here to develop a model that appears to me to be missing from their paper and to prevent a possibly incorrect/overstated claim from getting into the literature in the first place. I realize this is question of PrePPR vs PostPPR, but IMO open PrePPR of the Lan & Pritchard preprint would have been much more constructive than of a 10+ yr old paper that was already widely appreciated to contain overstated claims.

    Thirdly, how many biologists does it screw in a p-value? The math(s) community used Gowers’ blog years ago to actually solve new, outstanding problems in the field [3]. We biologists struggle to collectively compute a p-value. If the p-value prize is the pinnacle of biologists using blogs to critique each other’s ideas, I think this shows clearly that our community has a very, very long way to go before we are able to effectively practice science in the open. I for one would love to Lior (or others with big audiences) use his (their) blog(s) to set grander, more constructive challenges that advance new knowledge in biology. If this case, I doubt anyone would have misgivings about intention…

    [1] Note: I am a recovering population geneticist.
    [2] http://biorxiv.org/content/early/2015/05/10/019166
    [3] https://en.wikipedia.org/wiki/Polymath_Project

    • So, to put it shortly, your suggestion is to ignore scientific fraud, like one of Anil Potti because it does not advance knowledge, does not bring any new biological, and it is not constructive. Right?

      • No, that’s not my argument. I am in favor of rooting out fraud by all means. However, we are not discussing matters of fraud here, but rather applying an incorrect null hypothesis. Comparing Kellis et al.’s incorrect application of a null hypothesis (which happens routinely in evolutionary biology, and ultimately has little impact outside academic circles) to Potti’s falsification/manipulation of data (which led to clinical trials that could have actually harmed people) is an example of the kind of hyperbole that doesn’t help this discussion.

      • @caseybergman

        Unfortunately applying an incorrect null hypothesis looks into my eyes that is not a random mistake. Just the ridiculous low value of p-value would signal to anybody who knows a little bit of statistics that the chosen null hypothesis is plainly bad or something else than statistics should be used. As far as I know, for the best known and understood physical phenoma where very accurate measurements are done, one gets p-values with maximum of 9 significant digits. And now one claims that it gets p=10^(-87) for a not-that-well understood and defined biological phenoma which is not-that-well measured!!!! Seriously????!!!!! It is really difficult to believe that this is a honest mistake. It is basically manipulation of the null hypothesis! It is clear that this is an perfect example of irreproducible science!

        Before Potti was proven of fraud, first there were discussions and and tentative accusations (like in this case) which were not taken seriously. Many times Baggerly who was accusing Potti has been told over and over again that he has a personal issue with Potti and that his personal mission is to take him down. Baggerly was told that Potti’s mistakes were just honest mistakes while Potti was a rising star in academia. It is basically a deja-vu!

        Already ~10%-30% of published life science articles have results which are reproducible! See: http://www.jove.com/blog/2012/05/03/studies-show-only-10-of-published-science-articles-are-reproducible-what-is-happening
        The situation is already pretty bad and having more articles with shameful low p-values obviously does not make the things better for life sciences and biology.

    • Hi Casey,

      Thanks for your comment. I don’t follow the math community much, but there must be personal animosity, occasional overheated tempers, etc. (we’re all human after all). When these things flare up, do people keep showing up to comment/discuss online? Or does the intellectual action move elsewhere?

      Best,

      Joe

      • Joe: I can try to answer your question since I follow the math community fairly closely. My personal opinion is that the atmosphere in math is very different compared to genomics. For example, when I read Lior Pachter’s post, I thought of the difference with Edward Nelson’s response when Terry Tao found a fundamental flaw in his proof: “You are quite right, and my original response was wrong. Thank you for spotting my error. I withdraw my claim.” https://golem.ph.utexas.edu/category/2011/09/the_inconsistency_of_arithmeti.html

      • Of course, in math, somebody is either right or wrong, end of story. There aren’t alternative interpretations of data or a “different sense” in which one is right, etc. That’s sort of a fundamental difference.

    • caseybergman: Thanks for your comments – I think you make some very good points. Perhaps I am not an expert on gene duplications but I for one learned some interesting biology from the discussion on the P-value post. I agree with your comparison with math blogs – in addition to Tim Gowers’ blog, Terry Tao’s blog has also been remarkable in encouraging community research. But in the case of the P-value post, what I recall was that everyone agreed immediately on the null model and P-value calculation except for Manolis Kellis who continued to argue for another interpretation, which is why it apparently took many biologists to compute a P-value. Finally I just want to point out that a fraud accusation was made in Lior Pachter’s Network Nonsense post so whatever you or I think about the validity of that accusation, Dr Pachter can reasonably argue that the P-value post is addressing a larger issue, and Pacala’s comparison to Anil Potti (who I’m not very familiar with, I admit) may not be too far-fetched.

    • Yes, some reactions to that post made me think my writing was unclear, and so I deleted it while I think about how to communicate more clearly. Sorry.

      • As someone who worries perhaps excessively about what I put in print, I sympathize with the hesitancy about posting, but (as a useful counterpoint someone once offered me) the back-and-forth of commenting and pushback and trying to clarify your words can be a useful record to leave visible to the public gaze.

      • Yes, but when intelligent people think I’m saying extremely stupid things, that sets off panic alarms in my head :) I’d like to think through some of these issues without thousands of people watching.

  16. Pingback: Links 6/14/15 | Mike the Mad Biologist

  17. Pingback: Scientist wants to sack everyone who is not good at Math | The Science Web

  18. Cross-posted from Lior’s blog:

    I would like to clarify my comment about compbio retractions on Lior’s “I was wrong” post, to make it clear that this comment was not directed at Kellis et al, but was a general remark in response to Lior’s remarks about computational biologists admitting when they are wrong. My observation is that compbio as a field seems to have relatively few retractions, and my curiosity is whether this impression is supported by data. I did not mean to imply that the KBL paper should be retracted.

    I followed this comment with a remark about releasing code. Again this is a belief that applies to all compbio work and not just KBL. In general I think compbio papers should post their code, for reasons of reproducibility. Verbal descriptions of code are almost always incomplete, and without a way to run the code itself (and ideally scrutinize it), I consider that a methods section is incomplete. I favor the Titus Brown approach of releasing the entire workflow.

    The KBL yeast paper in particular is one where I would like to see the code released, because that work described major advances in genefinding sensitivity & specificity by using the indel patterns as a signature (indels within ORFs are a multiple of 3 bases long). I think it is important to verify and reproduce this work, it is an area I care about (having been modeling indels and alignments for some time) and so I would like to see the KBL code.

    Speaking more generally to Lior Pachter’s criticisms of Manolis Kellis, in his “Network Nonsense” post Lior says “In academia the word fraudulent is usually reserved for outright forgery” and goes on to argue for a broader use of the word “given what appears to be deliberate hiding, twisting and torturing of the facts”. I believe this is unhelpful. The word “fraud” is reserved to mean forgery because forgery, properly, has very severe consequences. In other places Pachter has indicated that he wishes those consequences upon Kellis (saying that Kellis should lose his job, for example). But “hiding, twisting, and torturing the facts” is a subjective description of events, as (in fact) is “deception deliberately practiced in order to secure unfair or unlawful gain” (a quote Lior does not source, but which Google attributes to the American Heritage Dictionary). The fact is that compbio is a hype-rich field, and few (including myself) would escape the charge of using at least some exaggeration or hype to describe their work, a practice which I agree is deplorable and could easily be characterized by someone more rigorous as “deception deliberately practiced in order to secure unfair gain”. Because, after all, we all deceive ourselves first, to some extent; don’t we?

    The most serious accusation Lior makes is that Kellis et al replaced the text of a figure to subtly but significantly alter its meaning. I personally think that publicly inviting them to publicize this change as an erratum would be a more helpful approach than accusing them of fraud. Clearly Lior believed differently: he says that he thought long and hard before making this accusation, so one must assume he considered less draconian options than calling for Manolis to be fired as a fraud.

    I completely support Lior’s right to criticize specifics of Manolis’ work, using whatever theatrical stunts he chooses to draw attention to these criticisms, including prizes and hyperbole. I would indeed praise his work as a post-publication peer reviewer: I believe the criticisms are, to a greater or lesser extent, valid. Network deconvolution probably has more hidden data-dependent parameters than Feizi et al admitted at first (so do a lot of methods: compbio code is ridden with hidden parameters). The choice of models for homologous protein rates in the yeast genomes paper could have been broader. But these are not out of line with the sorts of distortions or vagaries that (unfortunately) occur very often in compbio.

    I would dearly like to see compbio become more self-critical. I think Lior’s blog is an important step in this direction, and a valuable experiment in post-publication peer review. I think that one take-home message of this experiment is that one should be careful of accusations like “fraud”, which (as Lior acknowledges) have more than one meaning: a dictionary meaning in common usage, and a far more precise meaning that is specific to scientific ethics. Conflating the two risks devaluing the latter, and muddling up valuable scientific discussion with ad-hominem criticism. Let us reserve the word “fraud” for outright forgery. There are other terms (bluster, hype, subjectivity, bias, lack of rigor) that better characterize what Lior is getting at.

    Lastly, I am trying to tread a fine line here. Unlike others I am not attacking Lior. He has broken new ground with this blog. Rhetoric and showmanship are an important part of what he has doing. Knowing him, I expect he will not back down from his accusations of fraud nor his demands for Manolis’ job. That’s up to him. I’m simply saying where I stand. Deliberate, outright, result-faking fraud is a very serious issue that rightly needs to be a line in the sand for all sciences. Hype, self-serving bias, and irreproducibility are major problems that confound bioinformatics; they need to be solved, but not by conflating them with fraud. Prizes, critiques, fierce hyperbole: all are fair game. I find Lior’s style entertaining, and his work is excellent, but I also need to mention that I am a great admirer of work that’s come from Manolis’ lab too. The phylogenomics methods with Matt Rasmussen spring to mind. Manolis has also participated in some amazing biological discoveries, such as those involving the Piwi-interacting RNAs. There are many, many more. So I would be very disappointed if his career were significantly negatively affected as the result of one figure change which could easily be published as an erratum, or because he defended a choice of null model.

  19. Pingback: Friday links: side projects > main projects, tuning your scientific bogosity detector, and more | Dynamic Ecology

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s