Katalin Karikó appears twice in your thirty-two experiments, once as the CV that "looked mediocre by American standards," once as the good idea from an unexpected place. It's worth noticing what kind of failure she names. Peer review didn't misread her science; a whole system misread what merit looks like. Your own examples keep pointing outside the fence.
I want to say plainly that I'm for this agenda. I work inside a university research system — grant offices, impact assessment, valorisation — and "we have yet to develop enough solid empirical evidence" is a sentence I would happily staple to half the policy documents I read. The garden is the right image, too, because gardens are honest about cultivation and pruning in a way that funding calls never are.
But all thirty-two experiments happen inside the garden fence: funder to reviewer to PI to trainee. The soil — the society that staffs, funds, absorbs, and increasingly distrusts science — gets one experiment (#17), and that one equates translation with patents. From where I sit, a patent is a receipt, not a meal. So, two candidates for #33 and #34. Randomise the bridge: attach a resourced human layer between lab and public to some comparable grants and not others, and measure use rather than output. And vary whether funded projects engage the public as co-creators rather than audiences, then track trust and uptake. Consumption-only relationships to knowledge fail, as do consumption-only relationships to content. Europe, incidentally, has already run some of your list: Sweden kept professors' privilege, Norway abolished it in 2003. The data is sitting there.
A garden feeds people who never garden — but only where people still know food comes from soil. Which of the thirty-two teaches that?
The idea to engage the public as co-creators reminds me of eBird (https://ebird.org/home), maybe one of the best crowdsourced datasets ever. Birders have been contributing their field logs for decades (and many individuals and ornithological societies have backlogged historical data), compiling billions of data points that are used to understand migration, conservation, and general ecological trends.
The best part about eBird is that there is an immediate incentive for birders to use it. It acts as their personal list and they have access to the data, which is a useful resource for planning excursions (by looking at species density maps, briding hotspots, and yearly bar charts) and learning more about birds (through their Maculay Library, an annotated catalog of bird calls and images). Additionally, eBird gives users access to amazing tools that aren't crowdsourced, like BirdCast, a real-time migration tracker derived from weather radar signals, and Merlin, the bird identification app.
I don't have any suggestions for where this format could really improve a scientific field, but I agree it's worth testing.
Love this list, which highlights just how many areas of potential work there are.
On Distributed Peer Review, while UKRI and the Metascience Unit deserve all credit for establishing it as a legitimate mechanism within the UK system, it was actually first developed within the internal astronomy community (following a 2009 publication by Merrifield & Saari) and tried within various US agencies, including NSF in 2014. The UKRI trial took inspiration from trials by our European colleagues (including Dutch/NWO and German/Volkswagen Foundation). So it's an example of an idea that has required a bit of time to develop, with trials inspiring successors, but the mechanism not achieving recognition as legitimate in some places (e.g. NSF in 2014) simply because it was the wrong time/place, rather than any intrinsic differences in the implementation.
Thank you for writing this essay. I think there is something to be said about trying to inculcate the rising generations of scientists to a more open and diverse format of science funding and operation, and provide them the support through these newer mechanisms since the legacy system generally disenfranchises a vast majority of trainees (points 10-14). It seems that this is the goal of Analogue!
I also like the idea of tournaments and adversarial collaboration - competition can go a long way. There may be things that private institutes can do to compete against public funders like the NIH and provide them with a pressure that encourages them to operate quicker, more efficient, effective, or at least shed some bureaucracy.
Thirty-two experiments, almost all measuring output: papers, novelty, replication rate, volume, surprisal. The variable that decides whether any of that output is worth having is the predictive validity of the models the funded work rests on, and it appears nowhere as an outcome measure. Validity is measurable, and in preclinical work it separates a replication failure from a study that was never predictive to begin with. It belongs in the designs, not the background.
Katalin Karikó appears twice in your thirty-two experiments, once as the CV that "looked mediocre by American standards," once as the good idea from an unexpected place. It's worth noticing what kind of failure she names. Peer review didn't misread her science; a whole system misread what merit looks like. Your own examples keep pointing outside the fence.
I want to say plainly that I'm for this agenda. I work inside a university research system — grant offices, impact assessment, valorisation — and "we have yet to develop enough solid empirical evidence" is a sentence I would happily staple to half the policy documents I read. The garden is the right image, too, because gardens are honest about cultivation and pruning in a way that funding calls never are.
But all thirty-two experiments happen inside the garden fence: funder to reviewer to PI to trainee. The soil — the society that staffs, funds, absorbs, and increasingly distrusts science — gets one experiment (#17), and that one equates translation with patents. From where I sit, a patent is a receipt, not a meal. So, two candidates for #33 and #34. Randomise the bridge: attach a resourced human layer between lab and public to some comparable grants and not others, and measure use rather than output. And vary whether funded projects engage the public as co-creators rather than audiences, then track trust and uptake. Consumption-only relationships to knowledge fail, as do consumption-only relationships to content. Europe, incidentally, has already run some of your list: Sweden kept professors' privilege, Norway abolished it in 2003. The data is sitting there.
A garden feeds people who never garden — but only where people still know food comes from soil. Which of the thirty-two teaches that?
The idea to engage the public as co-creators reminds me of eBird (https://ebird.org/home), maybe one of the best crowdsourced datasets ever. Birders have been contributing their field logs for decades (and many individuals and ornithological societies have backlogged historical data), compiling billions of data points that are used to understand migration, conservation, and general ecological trends.
The best part about eBird is that there is an immediate incentive for birders to use it. It acts as their personal list and they have access to the data, which is a useful resource for planning excursions (by looking at species density maps, briding hotspots, and yearly bar charts) and learning more about birds (through their Maculay Library, an annotated catalog of bird calls and images). Additionally, eBird gives users access to amazing tools that aren't crowdsourced, like BirdCast, a real-time migration tracker derived from weather radar signals, and Merlin, the bird identification app.
I don't have any suggestions for where this format could really improve a scientific field, but I agree it's worth testing.
Love this list, which highlights just how many areas of potential work there are.
On Distributed Peer Review, while UKRI and the Metascience Unit deserve all credit for establishing it as a legitimate mechanism within the UK system, it was actually first developed within the internal astronomy community (following a 2009 publication by Merrifield & Saari) and tried within various US agencies, including NSF in 2014. The UKRI trial took inspiration from trials by our European colleagues (including Dutch/NWO and German/Volkswagen Foundation). So it's an example of an idea that has required a bit of time to develop, with trials inspiring successors, but the mechanism not achieving recognition as legitimate in some places (e.g. NSF in 2014) simply because it was the wrong time/place, rather than any intrinsic differences in the implementation.
All this and more here https://researchonresearch.org/distributed-peer-review-guidance/
Thank you for writing this essay. I think there is something to be said about trying to inculcate the rising generations of scientists to a more open and diverse format of science funding and operation, and provide them the support through these newer mechanisms since the legacy system generally disenfranchises a vast majority of trainees (points 10-14). It seems that this is the goal of Analogue!
I also like the idea of tournaments and adversarial collaboration - competition can go a long way. There may be things that private institutes can do to compete against public funders like the NIH and provide them with a pressure that encourages them to operate quicker, more efficient, effective, or at least shed some bureaucracy.
This is amazing!! I have been thinking about ways of commercializing research. I think I have some ideas that could work :)
Thirty-two experiments, almost all measuring output: papers, novelty, replication rate, volume, surprisal. The variable that decides whether any of that output is worth having is the predictive validity of the models the funded work rests on, and it appears nowhere as an outcome measure. Validity is measurable, and in preclinical work it separates a replication failure from a study that was never predictive to begin with. It belongs in the designs, not the background.