The covid vaccines were a whole different beast, though interesting case studies they were done under emergency authorization and didn't follow standard protocols.

Vaccine studies today almost always use a previously approved vaccine as the "control" group. That isn't a true control and if you walk back the chain of approvals you'd be hard pressed to find a starting point that did use proper control groups.

Anyway, my point here wasn't to directly debate vaccines themselves, only to point out that its interest to me as someone without a career in health to see the same effective argument used in two different scenarios with drastically different common responses.

Right, but the people making the argument about vaccines don't understand the principles, because they're actually the same!

1) a double blind RCT with a placebo control is a very good way to understand the effectiveness of a treatment.

2) it's not always ethical to do that, because if you have an effective treatment, you must use it.

Even without a placebo control you can still estimate both FN and FPs through careful study design, it's just harder and has more potential sources of error. A retrospective study is the usual approach. Here, the problem is they only included true positives in the retrospective study, so they missed the opportunity to measure false positives.

And the problem with -that- is that it's very easy to have zero false negatives if you always say " it's positive". Almost every diagnostic instrument has something we call a receiver operating curve that trades off false positives for false negatives by changing the sensitivity for where you decide something is a positive. By omitting the false negatives, they present a very incomplete picture of the diagnostic capabilities.

(In medicine you will often see the terms "sensitivity" and "selectivity" for how many TPs you detect and how many TNs you call negative. It's all part of the same type of characterization.)

The two points you raise with regards to why vaccine or similar studies may be treat special, it doesn't replace the loss of data when a double blind study with a control or make estimates based on modelling indicate anything more than correlation.

We may broadly agree that submitting a control group to a placebo treatment for a particular disease is immoral, but that doesn't mean such a study isn't necessary to prove out the efficacy or safety of the treatment. As for modelling, for example trying to estimate FN and FP, it can only ever indicate correlation at best and will never indicate likely causation.

But it's not. You can do an RCT of the new treatment vs the old treatment. You won't get a direct measure of its absolute efficacy but you will know if it's superior/ non-inferior to the best known thing. And then you can use observational techniques to estimate the absolute values. That's exactly what you would do if you wanted to develop, say, a new flu vaccine that you thought would outperform current vaccines. You get the most important information: whether or not we should switch to the new one.

If you have a new vaccine for a disease for which there is no existing vaccine you do a standard placebo controlled RCT which gives you a direct, high quality measurement of efficacy and side effects.