Thank you @llynn ! It is indeed a fascinating topic. The framework that you mentioned seems nice, but it seems that the paper that was linked is a different one. Concerning the DCGs, I used differential equations and large samples for this paper, but I intend to do something for small samples next. I am considering CRQA and symbolic regression for this (CRQA.jl + SymbolicRegression.jl)

R_cubed: > Perhaps @llynn will correct me if I’m wrong, but his complaint seems to be: how do you discover what variables to condition on when the measurement process is so noisy? Professor, I agree with your emphasis on replication and scientific integrity, but I would frame my concern somewhat differently. My primary criticism is not that measurements are too noisy or that randomization, even when combined with covariate adjustment, is fundamentally inadequate for achieving balance (although I will show when that IS true). Rather, the problem I raised is that the causal target is often ill-defined before randomization ever occurs. In the 1980s, critical care syndrome RCTs deviated from Hills methodology and acquired enrollment gates that are disease- and mechanism-agnostic. Participants are selected by prognostic thresholds (e.g., Sepsis-3, Berlin ARDS, AKI) rather than by a disease or causal mechanism. The trial question became essentially centrally controlled. The consensus gate produces a synthetic data generating process (SDGP) which is unique to the gate and population sampled. Randomization then estimates the average treatment effect within that unique constructed disease mixture (the SDGP). The key point is that the covariate probability distribution (E2) exists within each underlying disease, whereas the disease-mixture distribution (E3) sits one layer above it. E3 absorbs the covariate structure from all of the constituent E2 distributions. Consequently, covariate adjustment cannot recover a coherent biological estimand once heterogeneous diseases have been combined at the enrollment gate (although Professor Harrell makes an excellent point about near universally applicable adjustments, like age). Still, regardless of adjustment, the resulting estimate may be internally valid for that particular mixture, yet fail to transport or reproduce the treatment effect polarity when the disease mixture changes with the next trial despite the same setting and gate.That is the cause-mixture paradox. Here we see the math is not independent of the social structure of the discipline, it is integrated with it. (Which is why it is quite courageous to engage my critique in the way you have.) When this social to trial structure integration occurs this is pathological science on its face. This is true because a social structure is not self corrective so that when the social structure develops an inevitable pathology, then the trials become pathological. Without the pathology of social integration with trial structure, independent groups would challenge whether the enrollment gate represents a coherent biological system and in the alternative whether the mixture can be reproduced in the population during transport. These would be investigated in the interest of assuring the integrity of scientific self-correction rather than seeking to confirm a numerical treatment effect responsive to social expediency. That is structured failue mode detection which is a fundamentally different form of error detection then replicating the trial. For example, in corticosteroid trials of community-acquired pneumonia and sepsis, the 39 year effort (first massive trial in 1987 n 382. 19 centers) has largely been an unsuccessful attempt to consistently reproduce a positive numerical treatment effect when those few that were generated and reversed may simply have reflected the chance occurrence of a favorable steroid-responsive disease mixture as predicted by the math of the cause mixture paradox. My proposal is therefore to make the causal structure, especially the enrollment gate, an explicit object of scientific scrutiny free of social control. My critique of the Society presentation was that it was incomplete, especially in light of the massive failure of RCT based ventilator guideline transport early in the COVID epidemic. We have to address the real world failures of RCT based EBM as it is presently applied rather than continue to debate the adjustment, minimization, etc. So my comment was focused on the “gate to covariate adjustment dependency” and the correction of ongoing related RCT pathologies which @Stephen did not address. Until we address that, randomization, replication, and increasingly sophisticated statistical methods all remain vulnerable to the same structural design error.

R_cubed: > Perhaps @llynn will correct me if I’m wrong, but his complaint seems to be: how do you discover what variables to condition on when the measurement process is so noisy? Professor, I agree with your emphasis on replication and scientific integrity, but I would frame my concern somewhat differently. My primary criticism is not that measurements are too noisy or that randomization, even when combined with covariate adjustment, is fundamentally inadequate for achieving balance (although I will show when that IS true). Rather, the problem I raised is that the causal target is often ill-defined before randomization ever occurs. In the 1980s, critical care syndrome RCTs deviated from Hills methodology and acquired enrollment gates that are disease- and mechanism-agnostic. Participants are selected by prognostic thresholds (e.g., Sepsis-3, Berlin ARDS, AKI) rather than by a disease or causal mechanism. The trial question became essentially centrally controlled. The consensus gate produces a synthetic data generating process (SDGP) which is unique to the gate and population sampled. Randomization then estimates the average treatment effect within that unique constructed disease mixture (the SDGP). The key point is that the covariate probability distribution (E2) exists within each underlying disease, whereas the disease-mixture distribution (E3) sits one layer above it. E3 absorbs the covariate structure from all of the constituent E2 distributions. Consequently, covariate adjustment cannot recover a coherent biological estimand once heterogeneous diseases have been combined at the enrollment gate (although Professor Harrell makes an excellent point about near universally applicable adjustments, like age). Still, regardless of adjustment, the resulting estimate may be internally valid for that particular mixture, yet fail to transport or reproduce the treatment effect polarity when the disease mixture changes with the next trial despite the same setting and gate.That is the cause-mixture paradox. Here we see the math is not independent of the social structure of the discipline, it is integrated with it. (Which is why it is quite courageous to engage my critique in the way you have.) When this social to trial structure integration occurs this is pathological science on its face. This is true because a social structure is not self corrective so that when the social structure develops an inevitable pathology, then the trials become pathological. Without the pathology of social integration with trial structure, independent groups would challenge whether the enrollment gate represents a coherent biological system and in the alternative whether the mixture can be reproduced in the population during transport. These would be investigated in the interest of assuring the integrity of scientific self-correction rather than seeking to confirm a numerical treatment effect responsive to social expediency. That is structured failue mode detection which is a fundamentally different form of error detection then replicating the trial. For example, in corticosteroid trials of community-acquired pneumonia and sepsis, the 39 year effort (first massive trial in 1987 n 382. 19 centers) has largely been an unsuccessful attempt to consistently reproduce a positive numerical treatment effect when those few that were generated and reversed may simply have reflected the chance occurrence of a favorable steroid-responsive disease mixture as predicted by the math of the cause mixture paradox. My proposal is therefore to make the causal structure, especially the enrollment gate, an explicit object of scientific scrutiny free of social control. My critique of the Society presentation was that it was incomplete, especially in light of the massive failure of RCT based ventilator guideline transport early in the COVID epidemic. We have to address the real world failures of RCT based EBM as it is presently applied rather than continue to debate the adjustment, minimization, etc. So my comment was focused on the “gate to covariate adjustment dependency” and the correction of ongoing related RCT pathologies which @Stephen did not address. Until we address that, randomization, replication, and increasingly sophisticated statistical methods all remain vulnerable to the same structural design error.

llynn: > But my point, which was not addressed in Professor Senn’s paper is that the type of gate to matters when considering adjustment. For the adjustment to be valid the gate must select a biologic data generating process not a synthetic data generating process. There’s a lot of value in that way of thinking; I just wanted to point out that even silly factors that are still true baselines (can’t be caused by treatment, etc.) can explain variance in Y and increase power and precision.

f2harrell: > A big mistake is to fail to explain easily explainable outcome variation by not conditioning on covariates, which boosts power and makes the model fit better for free. Perhaps @llynn will correct me if I’m wrong, but his complaint seems to be: how do you discover what variables to condition on when the measurement process is so noisy? My complaint is that randomization can work **if other parts of the model have very low probabilities of being in error.** This isn’t the case in sepsis research, as Lawrence has overwhelmingly demonstrated In the context of sepsis research: the RCT methodology didn’t aid his community of medical scientists from discovering a crucial flaw in their assumptions. This lead to 30+ years of wasted research resources. This wasn’t clear in Fisher’s time, but any protocol that is deemed “scientific” needs to be: 1. decentralized ie. no central authority that determines “truth”. Consensus determines truth. 2. resistant to misleading reports, whether through honest error, or through intentional manipulation. All of these properties are implied in Feynman’s classic Cal Tech speech, where he introduced the notion of Cargo Cult Science. The essence of science, according to Feynman is: 1. Don’t fool others, have scientific integrity. 2. Don’t fool yourself, and never forget you are very easy to fool. There are a number of methodological guidelines he provides, that statistical recommendations (ie. randomization) are in conflict with – especially the need for repetition in order to validate a claim. calteches.library.caltech.edu ### Cargo Cult Science > Other kinds of errors are more characteristic of poor science. When I was at Cornell. I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this—I don’t remember it in detail, but it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do, A. So her proposal was to do the experiment under circumstances Y and see if they still did A. > I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person—to do it under condition X to see if she could also get result A—and then change to Y and see if A changed. Then she would know that the real difference was the thing she thought she had under control. > She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1935 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happens. These intuitive notions of Feynman regarding honest scientific process were formalized in the computer science literature, starting with the discussion by: **Lamport, L.; Shostak, R.; Pease, M. (1982).** “The Byzantine Generals Problem” (PDF). ACM Transactions on Programming Languages and Systems. 4 (3): 382–401. No communications engineer would today implement a protocol that does not at least have some form of error detection or correction if possible. In critical systems that are decentralized, resistance to parts of the system to arbitrary errors in other parts is known as being tolerant to Byzantine Fault This problem of Byzantine fault tolerance is a family of problems related to how much computational power you grant to the agents submitting misleading signals/reports. A system is Byzantine Fault Tolerant if > 2/3 of the signals/reports are reliable. (Agents in this context need not be human beings; they could be sensors or computer processors). The question I’m asking myself currently: how can this finding be adapted to the clinical trial context, without requiring huge sample sizes? **I believe this total cost of experimentation (initial claim + validation) can done with sample sizes of < 2n (with n being the maximum size of an initial randomized trial)** with some hard thinking, as E.T. Jaynes recommended. But that will require rethinking the role of randomization among a community of scientists. Who could object to a process that provides Byzantine fault resistance for less than the cost of 2 RCTs?

llynn: > Senn advocates adjustment for prognostic covariates, my question is, prognostic for what? It seems that the concept of a prognostic covariate cannot be separated from the enrollment gate. The answer depends on the population constructed by the gate. If the gate selects a coherent causal system, the prognostic model has a clear biological referent. If the gate constructs a cause-agnostic disease mixture, then the prognostic model pertains to that potentially non-reproducible mixture rather than to any individual disease represented within it. It’s a lot simpler than this. Adjust for baseline factors strongly suspected or known to predict the outcome. And if you adjust for a couple of factors that turn out to be non-prognostic, no harm done. A big mistake is to fail to explain easily explainable outcome variation by not conditioning on covariates, which boosts power and makes the model fit better for free.

isagonas: > The discussion was a bit of a let down to be honest, since no one disagreed with him A fair rebuttal to Senn’s paper deserves much more detail and citations than I can provide here. But a quick sketch of my thoughts, that builds upon the observations of @llynn are as follows: 1. Just a few days before Senn’s talk, the New England Journal of Medicine retracted a Phase 3 study which the author notes, is extraordinary for NEJM. NEJM Retracts Avacopan - by Mike Putman > This is not just a story about avacopan. It is a story about how the normal systems of sponsor oversight, contract research organization (CRO) data handling, peer review, regulatory approval, and post-publication scrutiny failed to detect a shocking case of data manipulation. Throughout a number of papers, Senn makes the argument that randomization protects the trial from “unscrupulous actors.” More detail can be provided in his paper Fisher’s Game with the Devil. The arguments in the paper are correct **if you are either the agent, or trust the agent in control of the allocation process.** The growing number of case studies on scientific fraud should provide evidence that this faith in randomization is misplaced. Would randomization be credible if the Devil, in Senn’s paper, had control of the allocation process? PubMed Central (PMC) ### Preventing fraud in biomedical research Scientific fraud represents, to varying degrees, an increasingly important part of medical literature and is estimated to make up nearly 20% of this literature. The increase in the number of articles accessible in preprint without peer review during... > The problem [of deceptive research reports] is larger than one might imagine and could be as high as 20% of publications. He makes useful points regarding randomization (from the perspective of the agent in control of the allocation mechanism running an honest experiment), but as in all of his writings on this topic, it makes a critical assumption that I no longer believe holds: the agent in control of the allocation process that using randomization is either trustworthy (will not attempt to deceive), or strategies for cheating are too costly to implement, and are easily detected. This makes it problematic for modern science that must rely upon the reports of others. 2. If we wish to discount the lack of protection from actors with the intent to mislead, that brings us to Lawrence’s complaint – randomization in the context of clinical trials in sepsis, have not produced much in useful results. The fundamental problem in this context is that the measurement process, the definition of the condition under study depends on other hidden assumptions (that may be wrong) in order for the assumption of groups created via a randomization process to be exchangeable for statistical purposes. Principled Bayesians used to complain about randomization and randomized decision rules. As was pointed out in another thread, Jaynes wrote:: > Of course, Fisher’s randomized planting methods – **which we think to be not actually wrong, but hopelessly inefficient at information handling** – were not reproduced by Jeffreys, nor would he wish to. Randomized non-comparative trials: an oxymoron? > Found it here, thanks! [image] It is not disputed that you can control more prognostic covariates in a balanced, controlled trial at a smaller sample size than you can with randomization. If you budget for a large trial, you can perform some error detection by running 3 smaller controlled, but balanced trials, and then combine them via meta-analysis at the end. You can actually test if your 3 samples are reasonably homogeneous using one of the tests of distributional equality. The most general is the Kolmogorov-Smirnov. Low p values indicate heterogeneity and information loss, and the effective sample size can be adjusted downward from the relevant experiment. Of course, there are going to be logistical challenges and various economic and clinical considerations to balance. I’m not sure how to handle safety monitoring, for example. I’d be inclined to let each experiment assess that independently, but that might not be economical nor ethical, and centralized safety monitoring might be better. I don’t think you would need to present much evidence to me that centralized safety monitoring along with decentralized efficacy monitoring, is preferable. This would require complex simulations, though. But it is now time to do the hard thinking and try to do better than randomization. Related thread: Generalizability vs. Transportability in Trials > I was thinking specifically of balanced allocation experimental designs such as minimization. The entire literature on this issue is rather frustrating to read, especially since the development of resampling methods. AFAICT, the entire dispute since Fisher and Gossett, was the validity of using a model such as Student’s t distribution, to examine the data from a controlled, but non-random experiment. The biggest complaint (which isn’t obvious) is that the data from such experiments has thinn…

@Alessandro_Rovetta I have to agree with @llynn that while you may have a point here that is important, you have to let go of the hot water and stick to the methodology otherwise it will get drowned out

@Alessandro_Rovetta I have to agree with @llynn that while you may have a point here that is important, you have to let go of the hot water and stick to the methodology otherwise it will get drowned out

@Alessandro_Rovetta I have to agree with @llynn that while you may have a point here that is important, you have to let go of the hot water and stick to the methodology otherwise it will get drowned out

@Alessandro_Rovetta I have to agree with @llynn that while you may have a point here that is important, you have to let go of the hot water and stick to the methodology otherwise it will get drowned out