Project Talk Page

A pair of cartoon talk bubbles in conversation, on a blue background

Welcome!

By Hilda Bastian:

This is a personal page on the update of the Cochrane review on exercise and ME/CFS. Formal reports about the updating of the Cochrane review on ME/CFS and exercise are on the Cochrane website, including FAQs. That’s the place to go if you have a question about this project.

However, there’s no avenue for engagement there. Twitter used to be a way that people could engage with me publicly about this project, but I left Twitter in 2022. So this page offers a broadly accessible avenue for questions, suggestions, and addressing issues that come up between public reports at the Cochrane website.

This is a WordPress blog, and though there may be occasional posts – for example, addressing frequently asked questions – the purpose of the page is to enable comments. You can subscribe to comments by subscribing to this blog, and then ticking the option for subscribing to comments within the subscription manager. You can also follow comments by RSS here.

Disclaimer

Any views I express here are my own and are not necessarily shared by anyone else involved in this project, including the project’s Independent Advisory Group (IAG), the review team, and Cochrane.

Privacy

You can comment or subscribe with a pseudonym. I will be able to see the subscriber list, but I won’t provide a copy of the list to anyone else. I am paying for an ad-free option of this blog for the duration of this project only.

Comments submitted here will ordinarily be published. So if you want to use this page to send me a message that is not intended for publication, please say so clearly.

Moderation

I moderate all comments made here, and could block for various reasons, including:

  • Defamatory comments,
  • Disrespectful or inflammatory language,
  • Length,
  • Repeated or very similar comments and questions.

I won’t edit a comment without having contacted the author asking for permission to do so at the email address provided in this system.

There could be delays in comments appearing, as I may not check for comments every day. I may also consult with others about some comments.

About me:

Hilda Bastian

83 responses to “Welcome!”

  1. Katie Johnstone Avatar
    Katie Johnstone

    It would mean a lot if Cochrane could acknowledge the harm this flawed review has done to people with ME. Many patients have been pressured into harmful and dangerous exercise programs, or branded as malingerers because they refused such programs. Some of have understandably lost trust in doctors, with the result that they have no-one to turn to for medical help. Families have been split apart when one parent believes that a child who is seriously ill with ME should exercise, on a doctor’s advice, while the other parent rightly insists that the exercise program could make the child sicker. The Cochrane review on exercise and ME/CFS is not the only culprit, but it is one of the bigger ones. Patient-advocates have been pointing all of this out for years and have mostly been ignored. It is understandable that the process to make changes takes time. But it is also a reality that, no matter how no matter how understandable the reasons, a community of vulnerable patients has been subjected to harm.

    Liked by 7 people

    Reply
    1. Hilda Bastian Avatar
      Hilda Bastian

      Thank you for articulating important points, Katie. I don’t speak for Cochrane, but I will make sure your comment is properly considered by the group developing the next Cochrane editorial note, which is a short-term option we’re working on.

      I wasn’t one of the authors or editors, but I think the expectation among them was that the 2019 amendment would address problems such as people applying the results of trials in people who aren’t seriously ill, to people who are. I don’t think that had the desired impact.

      I agree that critical issues that would have ensured patients were better served by the review were repeatedly raised by advocates. That should have been properly addressed, but was not. Cochrane established our advisory group so that would not be the case for the new updated review. The group will be working through concerns expressed in the past as well as ongoing feedback as the review progresses.

      Liked by 3 people

      Reply
      1. trishrhymes Avatar
        trishrhymes

        You say: “I wasn’t one of the authors or editors, but I think the expectation among them was that the 2019 amendment would address problems such as people applying the results of trials in people who aren’t seriously ill, to people who are. I don’t think that had the desired impact.”
        I’m afraid that entirely misses the point. It is often applying the (subjective and invalid) results of trials to people mildly or moderately affected by ME/CFS that has CAUSED many people to become seriously ill. Countless patients tell of walking in to their first graded exercise therapy session, and being wheelchair or bed bound by the end of the course of therapy.

        Liked by 5 people

        Reply
      2. Bob ollerton Avatar
        Bob ollerton

        I was a 27 year old male who had my own business, social life , home , I caught a virus and became very unwell ,I spent 6 years with what I know now as post viral fatigue, I was dismissed by every dr and specialist and told my blood were clear so my symptoms mustn’t be real , it got to the stage where I could only work 3 half days a week and couldn’t socialise anymore , 6 years later I seen a new gp who said I think you have m.e but don’t worry there’s a cure , over the moon I attended the m.e/cfs clinic and was advised that cbt/get was the answer , I attended the clinic in sept 2016, I drove there , walked into clinic and listened to everything they told me to do , what a mistake, by December 2016 I became completely bedridden, I couldn’t walk , talk , feed or care for myself, I had to have careers come in 3 times a day to help with my basic tasks , my mum phoned the m.e/cfs clinic and they told her to tell me to come in , my mum said he’s completely bedridden, he can’t even talk , the woman said he probably needs more intense cbt to which my mum replied he can’t even listen to sounds or watch tv he’s extremely sick to which she replied o well he will have to be discharged then if he’s not will it accept the treatment that will make him better and that was that , last I ever heard from them , this review should been completely withdrawn not just a little note pinned on , I pray one day that m.e patients get justice and everyone involved in harming them is brought to justice

        Liked by 7 people

        Reply
        1. Hilda Bastian Avatar
          Hilda Bastian

          I am so very sorry for what you’ve suffered and lost. I wish I knew what else to say. I haven’t been through this, and though I know I can’t really imagine what it feels like, you’ve painted a very clear image here. Thank you for sharing it.

          Liked by 2 people

          Reply
          1. trishrhymes Avatar
            trishrhymes

            Thank you Bob Ollerton for spelling out so eloquently the horrendous treatment you were subjected to. I’m so sorry that you and many thousands like you have been so mistreated. I agree, patients who have been harmed deserve justice.

            I hope Hilda will share your story with the senior editors at Cochrane who are being so adamant about keeping the harmful review. Any note, if that’s all we get, MUST give a warning about serious harm to people with ME/CFS from graded exercise therapy and any therapy that involves time contingent rather than symptom contingent activity increases.

            Liked by 2 people

            Reply
            1. Hilda Bastian Avatar
              Hilda Bastian

              Thanks, Trish.

              Liked by 1 person

              Reply
  2. mareecandish Avatar
    mareecandish

    Thank you Hilda for the update. I imagine that many good people have had some difficult times behind closed doors for the last two years.

    Clearly a new review is some way off completion. And, given a view expressed by a decision-maker in Cochrane that ‘middle-ground’ is a desirable outcome, I have little hope that the review will be what it needs to be, soundly based in science.

    Katie mentions some of the harms caused by leaving the old review in place. Another is that it signals to psychosomatic researchers that it’s totally fine to undertake studies that combine unblinded treatments that encourage highly selected participants to ignore their symptoms with outcomes that are surveys asking the participants how they feel. It signals that it is fine to undertake studies that ignore dropouts and do not adequately record harm. It signals that the strongly held concerns of virtually all consumer organisations for a disease can be ignored. As a result, the stack of rubbish studies concluding that these treatments ‘show promise’ continues to grow higher every week.

    The harms go well beyond people with a diagnostic label of ME/CFS or CFS. There’s an international crisis with Long Covid, that’s not hyperbole, and still Cochrane refuses to upset the old review authors. Still, studies with inadequate research methodologies are funded and published and used to guide clinical practice, partly because organisations like Cochrane do not penalise the poor practice.

    Hilda, we can’t wait for an adequate review that may never come. The past two years surely show you that no guarantees about the timely delivery of an acceptable review can be made. Do you have suggestions as to what the ME/CFS community can do to get the Larun et al 2019 review removed from the Cochrane Library?

    Liked by 7 people

    Reply
    1. Hilda Bastian Avatar
      Hilda Bastian

      Thanks for that, Maree.

      I’m familiar with the reported statement about there being no middle ground, but understand this to mean the opposite of seeking a negotiated position or compromise. I think a scientifically sound review is what all the involved parties are seeking to ensure. However, since there isn’t a single view of what’s scientifically sound on each point, a critical part of the process will be to ensure strong, open discussion of the issues, and transparent justification for scientific decisions.

      I don’t believe lobbying could get the Larun et al 2019 removed from The Cochrane Library, as I understand the current arguments and the Cochrane policy.

      As reported in the recently updated Cochrane FAQs*, the decision by the editor in chief not to withdraw the publication from the journal was reviewed by a group convened by Cochrane’s Editorial Board, who came to the same conclusion as the editor in chief. That included consideration of the contents of the then-recent NICE review – and similar arguments to those I’ve seen made recently calling for retraction.

      I realize that’s a very bare bones explanation, and I’m happy to discuss this further.

      *
      https://community.cochrane.org/organizational-info/people/central-executive-team/evidence-production-methods-directorate/stakeholder-engagement-high-profile-reviews-pilot

      Edit shortly after posting: “or compromise” added.

      Like

      Reply
      1. mareecandish Avatar
        mareecandish

        My interpretation of the reported ‘middle ground’ comment was that it was along the lines of exasperation that people were not able to agree. I think there was an expectation from some at the beginning of the new review process was that what was needed was a bringing together of the two sides to listen to each other and come to some compromise. As I have said, that makes as much sense as bringing together dyed in the wool climate change deniers and climate scientists and expecting a compromise position about whether the climate is warming to reflect reality.

        I think the core research methodology issues are much more straightforward than you seem to be suggesting. Do you not agree that the case that I mentioned above – having only participant reported outcomes in an unblinded trial for a treatment that encourages participants to ignore symptoms – is problematic? Do you not agree that ignoring the impact of a substantial level of dropouts is problematic? (These aren’t rhetorical questions, I would like to know. If you don’t agree, if the Editorial Board don’t agree, I’d really like to know why not.) Surely these are basic experimental design issues?

        I think it’s rather unfortunate to use the term lobbying for what people are doing in trying to get the Larun et al review removed. Volunteers, largely people themselves disabled by ME/CFS, are pointing out the poor quality of the review and the impact of it on their lives and those of others.

        The fact that the Editorial Board has not formally recognised the problems with the review yet is not surprising. They have been signing off on many reviews that have the same flaws for years. Perhaps that is a significant reason why they are hesitant to retract the Larun et al review? If they acknowledge that basic experimental design principles apply to non-drug therapies as much as they do to drug treatments, a large number of Cochrane reviews would have to be retracted.

        Liked by 3 people

        Reply
        1. Hilda Bastian Avatar
          Hilda Bastian

          The use of the word lobbying wasn’t intended to diminish people’s efforts, or cause offence: I apologize, and will use other phrasing in future.

          I doubt anyone went into this process thinking that the most opposing viewpoints on this review could ever be reconciled, and the task was certainly never described to me in those terms. Cochrane has been clear that they understand that concerns need to be addressed, and that’s the reason for this process, even though the results can never be to everyone’s satisfaction.

          Consulting on issues of concern was high on the IAG’s priorities from the start, and as discussed in the reports, we’ll be organizing a consultation on a draft summary of concerns to make sure the IAG captures the issues raised in the past, and incorporates any additional points of concern.

          Like

          Reply
          1. mareecandish Avatar
            mareecandish

            Further on my concern about your contention that much of what is scientifically sound is just a matter of opinion; and on my question about subjective outcomes which was not addressed:

            Consider this experimental design recipe:
            a. Tell someone that there is a really great treatment for their illness and that people who put the work in will get results
            b. Tell them that it is important that they stop focussing on their symptoms so much, everyone gets a bit of tired and has pain sometimes
            c. Deliver an ineffective treatment using kind therapists who make the person feel understood and cared about, perhaps conduct the therapy in a group setting where the participants provide each other moral and practical support
            d. Soon after the treatment ends, ask them if they feel better.
            Many people will say ‘yes, I probably do feel a bit better’.

            Many of those who definitely don’t feel a bit better gave up part of the way through the treatment because they feel embarrassed about not improving, or it wasn’t working so what’s the point, or they are just too sick to continue. These people don’t make it to the final survey. People who knew from the beginning that the treatment would not work didn’t even make it to the starting line.

            An optional extra is to have a control group who are just on the waiting list for the ‘really great treatment’. They are more likely to say when asked at the end of the study how they feel ‘no, I’m still definitely not well, so please give me that really great treatment now’.

            This is the experimental design recipe used by the BPS proponents, for exercise therapy, for CBT, for a whole range of treatments claiming to be innovative that really are just trying to convince the person that they only think they are sick. It’s also used by people who want to make evidence for supplements and the like. Open label studies with subjective self-reported outcomes. The result is virtually always a benefit from the treatment over the control arm. And the benefit can be substantial. This is a truth. The problems with this experimental design are taught in university courses, are documented in experimental design manuals and are, I think, self-evident. It’s why drug trials are blinded and double-blinded.

            If people involved in the new review process can genuinely say that they understand the substantial bias created by that sort of trial design and have designed a protocol to adequately address it, that would go a long way to increasing trust in the process.

            If the Cochrane Editorial Board could say that they understand the bias implicit in that experimental design, that would also be helpful. However, I have to assume, incredible as it is for an organisation aiming to provide reliable analysis of treatment trials, that they do not. If they did, they would understand that the 2019 Review bakes in the bias and incorrectly reports the small reported mean improvement that is the product as attributable to the treatments. And they would understand that this is a fundamental error, not an opinion, and would remove that review from their library.

            I am concerned that any new note added to the 2019 review will not address the core issue of the unreliability of the finding of improvement created by the bias implicit in the studies. I am concerned that instead the focus will be on suggesting that the recommendation may not apply to the subset of people with PEM. While that may be a face-saving option for Cochrane and the authors, the fact is that the trial design used by the studies means that we can’t say that exercise therapies helped any subset of people included in the studies.

            If you think that the experimental design I have described does not create bias that is likely to be larger than the benefit of the exercise therapy, or you think that the studies included in the 2019 review did not follow that recipe, I am very interested to hear why you think that way.

            Liked by 5 people

            Reply
            1. Hilda Bastian Avatar
              Hilda Bastian

              From the comment: “[Y]our contention that much of what is scientifically sound is just a matter of opinion.” I don’t believe that, and I don’t believe I’ve ever contended that, either. I think you may be referring to what I wrote here in reply to Robert Saunders* – that “there isn’t a single view of what’s scientifically sound on each point.”

              To the second point about a specific methodological discussion I didn’t respond to. As I’ve said in response to another concern about me not pursuing a discussion on a methodological question: While I can see my personal views would be of interest, I don’t think they’re a gauge of how others might approach the issues. And as the time I can spend on writing replies here is limited, I have to prioritize. Consultations will provide a better opportunity for those discussions than this page.

              Finally, if I don’t respond to something, that doesn’t mean I haven’t read it, or taken further action about it.

              Like

              Reply
      2. trishrhymes Avatar
        trishrhymes

        Hilda, you say: “I don’t believe lobbying could get the Larun et al 2019 removed from The Cochrane Library, as I understand the current arguments and the Cochrane policy.”

        Setting aside the use of the term lobbying which you have addressed in your reply to Maree, can you please explain what the arguments are that you ‘understand’ and what specifically about the Cochrane policy that led to this decision you ‘agree with’? And perhaps why you think others should find those arguments acceptable?

        Liked by 4 people

        Reply
        1. Hilda Bastian Avatar
          Hilda Bastian

          The first question doesn’t have a short answer, so I’ll put a pin in this and give some thought to how best to respond to it. In the meantime, I’d like to clarify where you’re drawing the quote about me agreeing with a decision? I don’t recall saying that.

          I’m not sure what the final question here means, but it doesn’t sound like anything I’d think.

          Like

          Reply
          1. trishrhymes Avatar
            trishrhymes

            I apologise, I misinterpreted ‘understand’ as ‘agree’ and then mistakenly put it in quotes. My excuse is I’m finding the structure of this discussion space difficult to deal with. It has the same flaws I found so frustrating on Twitter of multibranching threads.

            I think what I was getting at was that one can understand why an organisation has made a decision on the grounds that it fits with their policies while also making it clear that those policies are problematic, and therefore the decision so flawed that it should not be acceptable.

            So my conclusion would be Cochrane’s policies are wrong if it can draw such flawed conclusions about a review. I think this goes to the heart of my worries about Cochane’s editorial stance. If they don’t expect reviewers to downgrade as essentially worthless trials that rely entirely on subjective outcomes when the study is open label, especially when the treatment involves persuasion to fill in questionnaires differently, then what can we do?

            Liked by 2 people

            Reply
            1. Hilda Bastian Avatar
              Hilda Bastian

              Ah, I see – no worries.

              To the general question on limitations of Cochrane methods: I think there’s often room for resolving issues within the constraints of the methods, and taking issues into consideration in interpreting and weighing the evidence.

              Some of the limitations, of methods and/or the available evidence, though, can’t be overcome – and those need to be acknowledged, including the impact that has on the ability to draw conclusions.

              Methods and practices also evolve over time, including Cochrane’s, though that is a longterm process.

              Like

              Reply
              1. a97b Avatar
                a97b

                “I think there’s often room for resolving issues within the constraints of the methods, and taking issues into consideration in interpreting and weighing the evidence”

                That is an interesting statement in that it says Cochrane methods for evidence assessment are weak and ill defined such that they are open to manipulation. Surely there needs to be a well defined set of biases that are examined along with definitions of when they occur and the likely impacts on results? Otherwise its just opinions of a group of people weighting things and subject to personal biases.

                The methods also don’t seem to examine the validity of the statistical approach. Can you really ask a random set of questions, add up scores and quote differences and mean differences. This is what the 2019 review is doing and it doesn’t make any sense – although seems far too common in healthcare research.

                My conclusion looking at Cochrane reviews is the methods are weak and results not to be trusted. I say that as a researcher working in a different field.

                Liked by 2 people

                Reply
                1. Hilda Bastian Avatar
                  Hilda Bastian

                  I disagree. Being able to resolve and address issues doesn’t mean methods are weak. Yes, there are well-defined sets of biases that are examined,* and defined methods for coming to judgments on the strengths and weaknesses of the evidence. That doesn’t mean, though, that there aren’t limitations that need to be discussed, or that other biases not specified shouldn’t be addressed in addition.

                  Minimizing the impact of personal biases in methods, analyses, and statements is a goal of the methods, and should be a focus of editorial processes and response to criticisms of protocols and reviews. The methods are pre-specified in a protocol, which for this review, will be open for public consultation before being finalized for publication.

                  There are many criticisms of the 2019 review, and criticisms of earlier versions that still apply to the 2019 version. We’ll be providing a summary of them, and people will be able to let us know if they see additional points that should be added.

                  Cochrane Handbook: https://training.cochrane.org/handbook/current

                  Like

                  Reply
      3. a97b Avatar
        a97b

        How can we trust Cochrane to produce a scientifically sound review when they keep the current review up. The current review is scientifically unsound in so many ways – as have been pointed out by many for so many years.

        In the long term the way this review has been published will do serious damage to the Cochrane brand.

        Liked by 5 people

        Reply
  3. Ynske Jansen Avatar
    Ynske Jansen

    Hilda, my condolences on the loss of your son. That must be very sad and difficult for you.
    I understand that the review process has been delayed due to personal, social and organizational circumstances. As a patient representative in the Netherlands, I am involved in the revision of the old CFS medical guideline into a new ME/CFS guideline. It took ten years of lobbying, consultation, etc. before a new guideline development is now underway. The old guideline recommends CBT (including a form of GET) and GET as the only treatments for ME/CFS. Much to the detriment of the patients. I fear that the revision of the Cochrane GET review comes too late to help with the Dutch guideline. But I hope that the Cochrane process will bring out information about choices and considerations regarding definition, diagnostic criteria, etc. that can help us.

    Liked by 1 person

    Reply
    1. Hilda Bastian Avatar
      Hilda Bastian

      Thank you, Ynske.

      Liked by 1 person

      Reply
  4. trishrhymes Avatar
    trishrhymes

    Thank you Hilda for providing this space for communication and comment. It is long overdue. Twitter was never an appropriate way to communicate with people with ME/CFS over this project, as it required signing up to a platform many prefer to have nothing to do with, thus excluding many. The provision of a non functioning email address for 3+ years, and the recent reply from the Editor in Chief’s office to S4ME that all concerns about the Exercise therapy for ME/CFS old and new reviews should be addressed to that email demonstrates to me just how low people with ME/CFS are on Cochrane’s priorities.

    I apologise that the following is rather long. I have been involved, as you are aware, in the Science for ME forum’s current attempt to get Cochrane to withdraw the harmful 2019 review. We have been dismayed by Cochrane senior management’s barely polite and unhelpful dismissal of our concerns. It is not appropriate for them to pass this on to you and the IAG for a response, as you have no control over whether the 2019 review is withdrawn. We have now submitted a complaint to Cochrane about their handling of our letters and their refusal to withdraw the 2019 review. Unlike the previous complaints that we now learn have been going on, we will make every step of the way on that complaint process public.

    I cannot accept that Cochrane will not withdraw the 2019 review, and will continue to do all I can from my bed to get them to change their minds. Another editorial note clearly won’t work if the last one hasn’t. This is not about different equally vaild interpretations of scientific data, this is about Cochrane’s refusal to wake up and realise they are doing serious harm.

    Cochrane has demonstrated itself unfit to call itself a trusted source of clinical evidence while it refuses to understand that data from subjective outcomes on unblinded trials is not scientifically valid data. This is especially true when the treatment involves persuading patients to interpret their serious symptoms as improved when they have not. And it is also true when they refuse to take into account thousands of patient reports of serious long term harm from a treatment, as is the case with graded exercise therapy for people with ME/CFS that includes post exertional malaise.

    How can Cochrane’s charitable purpose be fulfilled if it is more concerned with sticking to its inadequate self defined rules and procedures, rather than seeing the wood for the trees and acknowledging that its Exercise review is causing serious ongoing harm, not only to people like me who have had ME/CFS for decades, but to the millions now experiencing PEM and fitting ME/CFS criteria with Long COVID.

    The fact that Cochrane’s antedeluvian procedures of taking a complaint from the proponents of exercise therapy took a year or more to be discussed by a series of committees, and required the new review process and updates to be stopped while they did so seems to me to demonstrate where Cochrane’s interest lies, and it’s not with patients welfare.

    I do not lay any blame for the delays on any of the IAG or review writing group members, and thank especially the patient and carer representatives for their efforts on behalf of patients and of good science. I will endeavour to participate constructively in the upcoming consultations.

    But I will not give up on campaigning publicly for the removal of the travesty that is the Larun et al 2019 review. The petition remains open, our complaints to Cochrane are public and still stand.
    https://chng.it/zTZ7vX9Czd
    https://www.s4me.info/threads/s4me-2023-open-letter-to-cochrane-request-for-action-on-the-me-cfs-exercise-therapy-review.34973/#post-501221

    Liked by 6 people

    Reply
    1. Hilda Bastian Avatar
      Hilda Bastian

      Thank you, Trish. And I agree Twitter was never a good solution either, and neither was the email address – few people used it, with most of the questions and comments coming elsewhere. I am hopeful that this talk page will be more effective.

      The Independent Advisory Group (IAG) is an exercise in stakeholder engagement, and I think that’s a reasonable way to ensure that questions can be explored and discussed in some depth. I don’t think establishing a process for that, and referring people to that process, indicates disregard. Rather, it indicates understanding that this requires more time commitment than is generally available for other Cochrane reviews, not less. Of course, I agree it has taken too long for us to be in public mode again, and that wasn’t satisfactory.

      A point of clarification: The work by the authors for the review was not stopped while complaints were being addressed. As I said in my recent report,* meaningful public reporting was affected, “The authors’ work on the draft protocol continued, however.”

      https://community.cochrane.org/organizational-info/people/central-executive-team/evidence-production-methods-directorate/stakeholder-engagement-high-profile-reviews-pilot-3

      Like

      Reply
      1. a97b Avatar
        a97b

        Given the way the review has failed and the lack of importance Cochrane gives to getting the science right in this area I would say that this exercise is a complete failure in patient engagement.

        It would be better to talk about patient co-production of a review which could have been done in quite a short time – I suspect delayed by Cochrane politics.

        Liked by 3 people

        Reply
  5. Caroline Struthers Avatar
    Caroline Struthers

    I used the term investigate not because I prefer it, but because it’s the term used in the COPE flowchart about how to deal with allegations of research or publication misconduct. Mine was an allegation of publication misconduct and was backed up with evidence. The wording in the flowchart uses investigation throughout. It is not a synonym for consider.

    “Respond to the person who raised concerns saying that you
    are going to investigate and will let them know the outcome but will not necessarily be in contact regularly before then….Investigate according to the
    appropriate COPE Flowchart or guidance, and also follow own publisher’s guidance…if there is an outcome to your investigation such as a correction or retraction, inform the person who originally raised the concern.

    Liked by 2 people

    Reply
  6. jonathancwedwards Avatar
    jonathancwedwards

    Dear Hilda,

    I am disappointed. As others say, patient subgroups and different views on detailed points are irrelevant. There is only one valid scientific position on exercise therapy for ME, CFS etc.: no reliable evidence of efficacy. This view was clear to me reading the 2019 review when acting as independent expert witness for NICE. It was clear to the NICE committee. Moreover, it was unanimously agreed by the UCL Division of Medicine when I presented the case at Ground Rounds. I was encouraged to proactively support the NICE view – no study meets a minimum for reliable evidence.
    The only people with different views are clinicians with a vested interest in usage of the therapies, including therapists, psychiatrists, rehabilitationists, and physicians/GPs with an interest. The justification for Cochrane’s existence is the exclusion of these interests from appraisal. It has failed, and if there are people with these views involved in the project it will continue to fail.
    When I first looked at the evidence I was ready to give benefit of the doubt. However, in recent months credibility of those championing these therapies has reached zero, for two reasons. 1. Specious arguments about subgroups and difficulties in doing trials raised at the NICE Round Table and succinctly dealt with by the committee spokesperson are repeated in journals and the press. 2. Interested parties and Cochrane senior members have bizarrely written in support of therapies such as the Lightning Process, with no scientific or evidence base. If those supporting exercise therapies for ME/CFS have so little understanding of basic scientific method it is hardly surprising trials were so bad in the first place. If senior Cochrane members have so little understanding, the whole exercise seems empty.
    We need to see the wood for the trees. The 2019 review should be withdrawn.
    You indicate that despite a public consultation blackout a new review process has gone ahead. That seems even worse. Surely the public consultation was designed to inform the process?

    Jo Edwards

    Liked by 4 people

    Reply
    1. Hilda Bastian Avatar
      Hilda Bastian

      Dear Jo

      Regardless of my personal conclusions on the quality of the evidence on this question, I don’t agree that everyone who doesn’t have a vested interest in the use of exercise therapies has concluded that there is “no reliable evidence of efficacy” in strong terms like this.

      Take for example the recent CDC review. [1] In their methods, they had the option to rate the evidence as “insufficient,” to indicate that:

      “evidence is too limited to estimate an effect, there is no confidence in the effect estimate, no
      evidence is available, or the body of evidence has unacceptable deficiencies, precluding reaching
      a conclusion.” (page 16)

      Instead, they rated it as “low”. NICE didn’t give its lowest certainty rating (“very low”) to every outcome either (eg for graded exercise therapy on page 148). [2] While “low” still isn’t a rating of evidence as very reliable, there are going to be differences from review to review result in variations – variation isn’t the same as scientifically invalid.

      In relation to this part of your comment: “You indicate that despite a public consultation blackout a new review process has gone ahead. That seems even worse. Surely the public consultation was designed to inform the process?”

      As I reported, we’ll be having a public consultation on our paper summarizing past critiques of the Cochrane review, a draft of which was provided to the authors to inform them early in the process. That will be followed by a public consultation on a draft of the protocol for the new review. Drafting the protocol is the part of the process that has gone ahead.

      Edit: Soon after posting, I deleted a remark after an objection.

      [1] CDC review https://www.cdc.gov/me-cfs/pdfs/systematic-review/file1-final-report-MECFS-Systematic-Review-508.pdf

      [2] NICE 2021 https://www.nice.org.uk/guidance/ng206/evidence/g-nonpharmacological-management-of-mecfs-pdf-9265183028

      Like

      Reply
      1. mareecandish Avatar
        mareecandish

        Hilda, in the reply to Jonathan Edwards, you seem to be suggesting that people who continue to have a fundamental problem with the experimental design of the exercise therapy trials are closed-minded and are assuming the other side is ignorant. You say that you are not like that. Can you see that you are playing the man, not the ball here? You are calling us narrow-minded for believing what we do and asserting a position of moral superiority.

        Can we focus on the scientific questions please? We have come to your platform to explain why we think the way we do. I have invited you to explain why others might hold a different view. But, so far, you have ignored my invitation.

        Liked by 1 person

        Reply
        1. Hilda Bastian Avatar
          Hilda Bastian

          I’m surprised at that interpretation of what I wrote: I don’t think I wrote anything that implied that. However, as I left my words open to that interpretation, I’m not happy about them either, so I’ve deleted it. Thanks for letting me know.

          Liked by 1 person

          Reply
      2. jonathancwedwards Avatar
        jonathancwedwards

        Dear Hilda,
        Of course they did! NICE and probably CDC committees were stuffed with people with vested interests.

        Moreover, NICE and probably CDC committees were constrained by illogical and ungrounded instruments like GRADE. Fortunately, evidence was so poor that common sense prevailed, at least at NICE, despite emotive objection from those with vested interests.

        People do not seem to understand the psychology – inability to see blindness to evidence, the lengths people will go to get positive results, group loyalties… Unblinded trials with subjective outcomes are a waste of time. For treatments that actually work, there are legitimate ways to overcome the problem. The study authors’ real problem is that for treatments with little (if any) effect, positive results can only be had by invalid methods.

        The ME/CFS debate uncovered a much bigger problem with therapist-delivered treatments. Almost all were introduced without evidence – including procedure specificity, dose response and therapeutic ratio, just as for drugs. For exercise we have none of that. Physios, psychologists and rehabilitationists need to face up to this, because treatments can harm. There are no ‘societal’ factors (suggested by a lame recent BMJ editorial) that justify compromise. The societal interest is in reliable evidence – as the patients have made plain.

        Jo

        Liked by 4 people

        Reply
  7. mareecandish Avatar
    mareecandish

    Hilda, I’m copying the comment I made above, as, given that there was no reply, I think you may have missed it. (Sorry, I’m finding this blog facility rather awkward for having productive conversations. I added the comment on to an existing discussion, rather than creating a new discussion.) I look forward to seeing your reply.

    “Further on my concern about your contention that much of what is scientifically sound is just a matter of opinion; and on my question about subjective outcomes which was not addressed:

    Consider this experimental design recipe:
    a. Tell someone that there is a really great treatment for their illness and that people who put the work in will get results
    b. Tell them that it is important that they stop focussing on their symptoms so much, everyone gets a bit of tired and has pain sometimes
    c. Deliver an ineffective treatment using kind therapists who make the person feel understood and cared about, perhaps conduct the therapy in a group setting where the participants provide each other moral and practical support
    d. Soon after the treatment ends, ask them if they feel better.
    Many people will say ‘yes, I probably do feel a bit better’.

    Many of those who definitely don’t feel a bit better gave up part of the way through the treatment because they feel embarrassed about not improving, or it wasn’t working so what’s the point, or they are just too sick to continue. These people don’t make it to the final survey. People who knew from the beginning that the treatment would not work didn’t even make it to the starting line.

    An optional extra is to have a control group who are just on the waiting list for the ‘really great treatment’. They are more likely to say when asked at the end of the study how they feel ‘no, I’m still definitely not well, so please give me that really great treatment now’.

    This is the experimental design recipe used by the BPS proponents, for exercise therapy, for CBT, for a whole range of treatments claiming to be innovative that really are just trying to convince the person that they only think they are sick. It’s also used by people who want to make evidence for supplements and the like. Open label studies with subjective self-reported outcomes. The result is virtually always a benefit from the treatment over the control arm. And the benefit can be substantial. This is a truth. The problems with this experimental design are taught in university courses, are documented in experimental design manuals and are, I think, self-evident. It’s why drug trials are blinded and double-blinded.

    If people involved in the new review process can genuinely say that they understand the substantial bias created by that sort of trial design and have designed a protocol to adequately address it, that would go a long way to increasing trust in the process.

    If the Cochrane Editorial Board could say that they understand the bias implicit in that experimental design, that would also be helpful. However, I have to assume, incredible as it is for an organisation aiming to provide reliable analysis of treatment trials, that they do not. If they did, they would understand that the 2019 Review bakes in the bias and incorrectly reports the small reported mean improvement that is the product as attributable to the treatments. And they would understand that this is a fundamental error, not an opinion, and would remove that review from their library.

    I am concerned that any new note added to the 2019 review will not address the core issue of the unreliability of the finding of improvement created by the bias implicit in the studies. I am concerned that instead the focus will be on suggesting that the recommendation may not apply to the subset of people with PEM. While that may be a face-saving option for Cochrane and the authors, the fact is that the trial design used by the studies means that we can’t say that exercise therapies helped any subset of people included in the studies.

    If you think that the experimental design I have described does not create bias that is likely to be larger than the benefit of the exercise therapy, or you think that the studies included in the 2019 review did not follow that recipe, I am very interested to hear why you think that way.”

    Like

    Reply
    1. Hilda Bastian Avatar
      Hilda Bastian

      Not having replied quickly doesn’t mean that I have missed a post. As I wrote in my introductory post for this talk page, I may not be attending to this page every day. The time I spend replying will be limited. I hope that delays will usually only be a matter of days.

      To avoid this, since you submitted this comment, I’ve held onto comments I need time to respond to. Unfortunately, I can’t see a way in this particular WordPress design to write a note for email responses when someone submits a comment – if there was, everyone could get a “thank you” and a warning that comments and/or replies may not appear online quickly. I’ll try to find one that does, though I can’t be sure there is.

      My reply to your original comment is here: https://hbprojecttalk.wordpress.com/2023/11/10/welcome/comment-page-1/#comment-55

      Like

      Reply
  8. Robert Saunders Avatar
    Robert Saunders

    Hilda, Thank you for the long awaited update.

    On 12 April 2021, I Tweeted you to ask why we hadn’t received any updates for so long and when we would be receiving the next update. You were unable to give me a date and replied: “I’m sorry to have let you down.“ (Ref 1). Since then, many of us have sought updates but we were told nothing for two and a half years.

    As you will recall, when this process was launched it was billed as a “pilot project for engaging stakeholders (such as consumers…)”. From that perspective it’s hard to imagine how it could have been worse.

    I appreciate that you may have been dealing with objections from scientists whose behaviour you have previously described as “unconscionable” (ref 2) but I still don’t understand why nobody was able to update us about what was happening for so long – particularly as it appears that BACME was being kept informed. Can you explain why that was?

    I’m also disappointed that pandemic has been cited as a reason for delays and silence. Long Covid should have increased the priority given to this project, not delayed it.

    Above, you write: “I’m familiar with the reported statement about there being no middle ground, but understand this to mean the opposite of seeking a negotiated position or compromise. I think a scientifically sound review is what all the involved parties are seeking to ensure. However, since there isn’t a single view of what’s scientifically sound on each point, a critical part of the process will be to ensure strong, open discussion of the issues, and transparent justification for scientific decisions.”

    I’m not convinced that is what all parties want. On one side there are patients – supported by NICE, the CDC and disinterested scientists – many of whom have showed a willingness to accept that their perceptions and beliefs may be wrong by agreeing to try GET, despite their concerns. In many cases those patients appear to have paid a terrible price for the trust they have placed in medical authorities.

    On the other side, there is a small group of medical professionals who, for unknown reasons, appear to be so wedded to their views that they are unwilling to accept the evidence from their own trials.

    I would like to believe that these medical professionals want a sound scientific review, but the historic evidence suggests otherwise. It is hard to know whether that may be due to conscious or unconscious biases. Whatever the cause, my perception is that authority is being prioritised over evidence, and that professional reputations are being prioritised over patient welfare.

    Finally, the “strong, open discussion of the issues” you call for has already happened at the NICE Guideline Roundtable discussion (Ref 3). Health professional stakeholders raised concerns. These were all addressed with succinct responses from representatives of the guideline committee, and no further objections were raised.

    Given that NICE and Cochrane signed a collaborative agreement in 2021 (ref 4),
    I don’t understand why this process needs to be repeated.

    Meanwhile, patients continue to be harmed by the Cochrane review.

    References:

    1) https://x.com/hildabast/status/1381727858846965765?s=20

    2) https://www.s4me.info/threads/independent-advisory-group-for-the-full-update-of-the-cochrane-review-on-exercise-therapy-and-me-cfs-2020-led-by-hilda-bastian.13645/page-29#post-267260

    3) https://www.nice.org.uk/guidance/ng206/documents/minutes-31

    4) https://www.nice.org.uk/news/article/nice-and-cochrane-sign-collaborative-agreement-to-deliver-living-guideline-recommendations

    Liked by 2 people

    Reply
    1. Hilda Bastian Avatar
      Hilda Bastian

      I agree that a huge unexplained delay is awful, especially for a stakeholder engagement process, and I am very sorry.

      I haven’t been trying to excuse or downplay it, but explain it, and update people on what was happening that hadn’t been reported during that period. Some of the issues are separate – eg why the original review timetable itself went out the window (a combination of pandemic and funder announcing withdrawal of funding), and why the IAG stopped and reports stopped. And some of the issues that were reported aren’t responsible for particular parts of the delay, but it’s catch-up reporting.

      I’ve had several questions about why I didn’t provide reports etc. So I’m collecting up questions and remarks on this, and will re-visit the issue in my next report and try to clarify some things then. You asked me to clarify why BACME was kept informed: I’m not sure what you mean by that, but I hadn’t been in touch with them or any other organization.

      On your point about the pandemic. It slowed a lot of things down for many people, especially in some countries. For this project, it affected progress more than usual, as the authors who work for Cochrane worked on Covid reviews when the Cochrane team turned to that. In terms of reporting, though, the pandemic was only a factor early on.

      Your point about not everyone wanting a scientifically sound review is another important one, and I agree. When I said “the involved parties” did, I meant the authors, editors, and advisory group (IAG).

      I’m sure that most if not all of the issues that could arise will already have been discussed in consultation processes for reviews by NICE and those in several other countries. I don’t think that means we can forego discussion on Cochrane’s, or just refer people to those other discussions. Trying to save people having to spend time making the same arguments again is something we’re keen to do. Which is why we’ll be putting out a paper on the criticisms of the Cochrane review we’ve found from the past, and trying to ensure that people can see what ground has already been considered.

      If that’s not what you were referring to, but rather, why another review needs to be done at all, I agree that’s another important point. It’s an issue that’s been under intense discussion within this process, and I’ll report on this later, too.

      As Cochrane doesn’t plan on retracting the review it already has, that’s the complex context for us considering this now within the process. We’re convinced if Cochrane just stopped now, and left things as they were it would be problematic for many people. Given that there hasn’t been a large body of randomized trials since that review, the review would still be seen as current by many.

      Liked by 1 person

      Reply
      1. rhsaundersb51afeb15d Avatar
        rhsaundersb51afeb15d

        Thanks for your reply, Hilda.

        You write: “You asked me to clarify why BACME was kept informed: I’m not sure what you mean by that, but I hadn’t been in touch with them or any other organization.”

        In September BACME wrote to S4ME. The email stated that the review had “temporarily been paused” and that BACME understood “that it is now being reactivated”. That was before you or Cochrane had made any public statements about the review having been paused or reactivated. At that time, S4ME and others stakeholders had been told nothing despite having repeatedly asked for updates. (See: https://www.s4me.info/threads/petition-s4me-2023-cochrane-withdraw-the-harmful-2019-exercise-therapy-for-cfs-review.35109/page-4#post-494507)

        That is why I wrote that it appears that BACME was being kept informed when other stakeholders were not.

        Do you know why that was? Do you infer that someone may have broken a confidentiality agreement?

        Like

        Reply
        1. Hilda Bastian Avatar
          Hilda Bastian

          Ah, I see. That’s unfortunate: One of the reasons I wasn’t replying to emails was so as not to be selective.

          By September, it’d been active for a while, and we thought an update report would be quicker than it turned out to be. So there were a few messages out there I think – including one to S4ME from Cochrane late that month, if I recall correctly.

          Like

          Reply
  9. mareecandish Avatar
    mareecandish

    Hilda,
    You seem to be suggesting that the fact that NICE didn’t rate all the evidence relating to GET as ‘very low’ somehow means that they didn’t conclude in general terms that there was ‘no reliable evidence of efficacy’.

    I have gone through the evidence evaluation for GET (versus standard care) in NICE. While it is true that NICE did not give every outcome a rating of ‘very low evidence quality’, it did rate most of the outcomes in that way.

    That included the evidence for harms – all three harm outcomes were rated as being of very low quality. Given the precautionary principle and the strong views of virtually every ME/CFS patient organisation in the world that there is harm, we should not dismiss the possibility of harm. Therefore, in order to recommend the treatment, we would want to see clinically important benefits for which there is good quality evidence.

    But, there just aren’t clinically important benefits for which there is good quality evidence.

    Here are the six outcomes NICE reported as having low quality evidence (‘low’ is still really bad, but better than ‘very low’.)

    Quality of Life, a subjective outcome from one study. The reported benefit is small. NICE noted that there was no clinically important difference.
    General Symptom scales. One of the three outcomes for this category was rated low quality evidence, but the other two were very low quality evidence. The one that was reported as low quality was only from 1 study for 12 weeks. The two general symptoms scale outcomes that were reported as very low quality consisted of more studies and/or had a longer assessment period. NICE noted that there was no clinically important difference in General Symptom scales at 134 weeks.
    Psychological status HADS – depression.
    Psychological status HADS – anxiety
    The evidence consists of 2 studies, including one with an assessment at 12 weeks which is too short in ME/CFS as impacts of increased exertion are cumulative. The differences between controls and intervention were small. I expect that people who were able to get to sessions did feel less depressed and anxious as a result of participating in a group; there would have been a feeling of mutual support, less of a feeling of being alone. A wait list control is not an adequate control for GET. Despite that, NICE noted that there was no clinically important difference in either outcome.
    Activity levels (subjective). This was only from a 12 week trial length which is too short. The claimed increase in activity levels was not corroborated by an increase in fitness in the objective measures. (NICE reported no clinically important differences in physical functioning SF36physical, the 6 minute walk, VO2 peak, peak power, elapsed exercise time test.)
    Return to school/work (subjective). One study, from a 12 week trial length. The reported benefit over controls was small. The evidence at 134 weeks was assessed as of very low quality and the reported benefit became minuscule. NICE noted that there was no clinically important difference in Return to school/work.

    Therefore, of the six outcomes with low quality evidence, all either had a benefit that was not clinically important, was assessed after too short a period of intervention and/or was not corroborated by other measures that we would expect to track in a similar way.

    Liked by 1 person

    Reply
    1. Hilda Bastian Avatar
      Hilda Bastian

      That discussion was one of the ones flowing from my having written that everyone didn’t agree on every point, and that it was complicated. So my discussion of examples of rating the strength of evidence was to point to how much nuance there is, and that things vary, including rating definitions and decisions. I’ve just re-read it, and think it’s clear that I don’t think evidence rated “low” could be considered very reliable.

      Like

      Reply
  10. mareecandish Avatar
    mareecandish

    Hilda, on a different topic, can you please explain what communication constraints have been placed on the three organisations with representatives on the IAG?

    Is it correct that the organisations may not comment on anything to do with Cochrane, both regarding the 2019 review and the new review process? Are the representatives allowed to talk with people within their organisation about the process, with members of their organisations and their affiliated organisations?

    How far do the restrictions on who may speak go? For example, with ME Action, is it just the IAG representative who may not criticise Cochrane and the process? Or is it that no one in the head office of ME Action may comment publicly, even about the need to retract the 2019 review? Are the national groups such as ME Action USA, ME Action UK covered by the requirement not to speak to the ME/CFS community? Does it include the regional groups under the national ones?

    For ME/CFS Australia, is it just the two representatives who may not comment about Cochrane? Or is it everyone in ME/CFS Australia? And does that include all of the regional organisations affiliated with ME/CFS Australia such as ME/CFS South Australia and ME/CFS and Lyme Association of Western Australia?

    And for EMEA, again, is it just the representative on IAG, or is it EMEA, or all of the organisations affiliated to EMEA, and all of the regional organisations affiliated to those organisations?

    Thanks.

    Like

    Reply
    1. Hilda Bastian Avatar
      Hilda Bastian

      Dear Maree

      There were no communication constraints imposed on the 3 organizations with representative positions on the IAG: European ME Alliance, #MEAction, and ME/CFS Australia – or on any organization with which any IAG member is associated.

      The representative members of the IAG are in the same position as other members with regard to confidentiality. There will be documents and matters under discussion throughout the process that are confidential to members and designated proxies. And I am the spokesperson for the Group. Beyond that, the IAG can determine its own approach to confidentiality.

      While I am the spokesperson for the IAG, this blog is personal, and so are the views I express here. If I’m communicating a position of the IAG, I will say so.

      Like

      Reply
Further Comments

Leave a comment