Studying Inquiry

Ideas for Metascience Experiments

If science is one of the most important engines of human progress, then it makes little sense to treat the organization and funding of science as areas where we should just wing it. Some ways are surely better than others.

More than a decade ago, Pierre Azoulay made exactly this point in a short but influential Nature piece, arguing that “it is time to turn the scientific method on ourselves.” His argument was simple: if scientists insist on evidence and experimentation in their own work, they should apply the same standards to the meta-issues of grantmaking, peer review, career structures, and research institutions themselves.

Patrick Collison and Tyler Cowen make a similar argument in We Need a New Science of Progress.

Even in 2026, this idea is still far too salient. We argue about whether to fund people or projects, whether peer review is too conservative, whether younger scientists are underfunded, whether longer grants would help, or whether AI will make science more productive, but we have yet to develop enough solid empirical evidence on any of these points.

To be sure, not everything can be optimized by a tidy randomized trial. Indeed, we are fans of Lant Pritchett’s critique of RCTs in development economics as focusing too much on trivial-but-measurable issues rather than broader macroeconomic questions that were much more important. At the same time, many important choices in science funding and organization are testable, and we should test them.

We want a scientific ecosystem that is more productive, creative, and better at finding real discoveries even in areas that are unpopular. To be clear, there is almost certainly no such thing as “one right way” to fund science. Instead, we need to learn which approaches work best, in which settings, and for whom. In its ideal form, our scientific system would look more like an enormous garden — one that includes everything from tomatoes to basil to zucchini to okra — than like a field of identical corn stalks. And like a good garden, it would require deliberate cultivation and pruning (i.e., the willingness to pull things that aren’t working).

If even three or four of these experiments are actually run in the next few years via new “metascience offices” at federal agencies or through a potential lab or fund studying Inquiry itself, we would be pleased.

We are actively looking for bright minds to develop these lines of inquiry with us: if this is you, reach out to us at aish@analoguegroup.org and stuartbuck@goodscienceproject.org.

Funding Mechanisms and Selection

1. Person vs. Project Funding

Traditional project grants ask scientists to predict, years in advance, what they will discover and exactly how they will do it. That is odd, since at the frontier of scientific research, the whole point is that you can’t predict what will happen in 5 years.

A routine suggestion is that agencies should fund the person rather than the project – i.e., pick smart, talented people, give them significant grants, and then get out of their way. Such an approach would give more room for creativity and switching directions in response to new evidence, while reducing the scientist’s need to spend endless time chasing new grants.

There’s a lot to like about that idea, and programs like the HHMI Investigator Program and the Wellcome Trust Senior Investigator Awards already take this approach. A famous paper by Pierre Azoulay and his colleagues found that HHMI’s grantees “produced high-impact papers at a much higher rate” than similar NIH-funded scientists. That said, Azoulay once told me [Stuart] that he thought the real secret of HHMI grants might be that they last 7 years with near-automatic renewal, which would give scientists more leeway to tackle huge problems that aren’t reducible to the timeline of a typical NIH grant.

A randomized trial at NIH could compare person-based and project-based funding directly. It could track the degree of novelty in future publications, the rate of breakthrough discoveries, the number of genuine failures due to tackling a huge problem, and the later careers of junior lab members.

2. Blinded Peer Review

Does anonymizing a researcher’s identity and institutional affiliation reduce bias and/or increase funding for unconventional or early-career applicants?

Consider which approach would have been more likely to fund Albert Einstein in 1904, when he was a patent clerk with no academic track record, or Katalin Karikó in the 1990s, when her CV looked mediocre by American standards. Blinded review might have given these ideas a fair hearing on their merits. Unblinded review, on the other hand, might have screened out both of them before anyone read the proposal.

Peer review could be influenced by prestige, reputation, career stage, gender, race, and institutional affiliation. An experiment could randomly assign proposals to blinded or unblinded review, then compare who gets funded, what kinds of projects are supported, and which system better predicts later research impact.

Worth reiterating: this idea is exactly the opposite of “fund the person, not the project.” Blinded peer review amounts to “fund the project, not the person.” Perhaps we need both approaches as part of a broader portfolio.

3. Golden Ticket System and Peer Review Variance

Consensus-based peer review often filters out exactly the kinds of proposals that might lead to breakthroughs. After all, highly original ideas can produce polarized reactions. A few reviewers may see major promise, while others see fatal flaws. Averaging the scores tends to make those proposals look exactly like the most mediocre proposals.

A “golden ticket” system would let each reviewer champion one proposal per cycle, even if others disagree. A looser version would track the variance in reviewer scores and preferentially fund proposals that receive both very high and very low ratings.

An experiment could test whether golden-ticket or high-variance proposals are actually riskier and more innovative than consensus-approved proposals, whether they fail more often, and whether their successes are more important when they land.

Indeed, this sort of policy might actually grow more effective the more widely it is adopted. If the theoretical justification is correct, we should see:

• The lowest effect when analyzing existing data (everyone was applying under the preexisting criteria, so the proposals are as homogeneous as ever);

• The next lowest effect when used in a small pilot experiment (the program may not be widely known yet);

• A stronger effect when used in a large RCT (although researchers might still pull their punches because they fear being in the control group);

• The strongest effect when a new policy is rolled out on a wide basis to great fanfare, in a way that attracts new applicants and encourages everyone to submit different proposals than before.

In other words, even a large RCT might still underestimate the possible effect due to system-wide behavioral changes.

NSF is currently running an experiment along these lines, although the results will be kept private. That is itself worth commenting on: it is ironic that a metascience experiment by a science funding agency would be withheld from the scientific community. Other agencies, such as NIH, could engage in a larger experiment on a more public basis.

4. Partial Randomization of Grants

Research on peer review reliability suggests that panels often disagree sharply about which borderline proposals should be accepted. That implies that many grant decisions near the cutoff are already partly arbitrary.

Fang and Casadevall made the case for lottery-based funding in a 2016 mBio paper: if the review process cannot reliably distinguish similar applications, a lottery for the “gray zone” may be more honest than pretending that tiny score differences reflect any actual difference. And it might be more efficient too.

An experiment could assign borderline proposals to lottery-based or standard allocation, and then compare their later research impact, as well as whether scientists feel like the process was fair. At a minimum, we think the process might be more efficient in terms of wasted time by both proposers and peer reviewers.

5. Distributed Peer Review

Should grant applicants be required to review one another’s proposals? The UK Metascience Unit has piloted this approach, and has suggested that distributed review may have many advantages over traditional peer review (including eliminating the need to spend time recruiting outside reviewers who are often late or non-respondent, eliminates the need for burdensome panels, solicits reviews from people who are highly motivated to be familiar with the funding call, and more).

This approach could speed up decisions, streamline the process, and give applicants more insight into how proposals are judged. As for conflicts of interest, the UK team divided proposals into two large groups at random, and required applicants in one group to rate the other group (i.e., only those applications not in direct competition with their own). Still, this new idea could result in lower quality reviews and/or more homogeneity.

An experiment could compare distributed review with the standard panel review on speed, quality, fairness, and reliability.

6. Program Officer Discretion

Would giving program officers more authority to overrule peer review lead to more breakthrough science?

Program officers often have deep knowledge of their fields and long-term views of where they are headed. They may spot opportunities that panels miss, especially when novel work looks too risky or strange, which is precisely where breakthrough ideas might be lurking.

DARPA is frequently cited as a model where program managers have unusual discretion, and DARPA’s track record — from the internet to GPS to foundational work on mRNA vaccines — is often invoked to argue for more discretion elsewhere. But how much of DARPA’s success is actually attributable to manager discretion, as opposed to its unique institutional position and its ability to sell research results to a large “customer,” is itself an open question.

An organization the size of NIH could experiment with giving certain program officers the discretion to bypass peer review, and then see whether their funded projects show more promise over time. To be sure, as with other interventions discussed above, the true effect of such a policy might be seen in the long run, as new program officers are recruited and as scholars are incentivized to submit ideas that they previously would have left in the file drawer.

7. Size of Grants

What is the optimal size of a scientific grant? This question gets surprisingly little attention given how much money is at stake. NIH R01 grants typically provide around $400,000 per year in direct costs. Is that the right number, or would science be better served by larger and fewer grants?

There are potential tradeoffs at any particular size of the average grant. Smaller grants would arguably spread resources more widely, fund a wider range of investigators, and possibly even increase the diversity of ideas. But smaller grants might force scientists to operate on a shoestring or to cobble together multiple small awards just to keep a lab running, which would increase the potential administrative burden plus the time spent grant-writing. Larger grants might give scientists more freedom to pursue longer-term questions without constantly begging for the next dollar.

An experiment could randomly vary the size of awards within the same program, holding total dollars constant, and then track the number and quality of publications, the novelty of the work, the number of researchers trained, and the cost per meaningful discovery.

8. Concentration of Grants: Should There Be Limits?

Should there be a cap on how many grants one scientist can hold at the same time?

Right now, there is no hard limit at NIH or most other agencies. Some scientists hold four, five, or even more concurrent R01-equivalent awards. That raises an obvious question: is the 5th grant to a single PI producing more value than a 1st grant to someone who currently has no funding at all?

The argument for concentration is that some scientists are simply more productive than others, and giving them more resources is just rewarding excellence. That could well be the case, but the case against concentration is also seemingly persuasive. After all, there are diminishing marginal returns in most areas of human life. Moreover, after a lab grows to a certain point, the PI is mostly not doing the science, but instead is acting as a manager of other scientists. That might be highly productive in some instances, but it might be the case that the post-docs or graduate students who are doing most of the real lab work would do equally well (or even better!) if more empowered to follow their own ideas.

There is also a portfolio theory argument here. If the goal is to maximize the probability of at least one breakthrough across the entire system, you may want more independent bets rather than fewer large ones. Five PIs with one grant may offer more “shots on goal” than one PI with five grants (emphasis on “may”: given the degree of groupthink and conformity in science, it is also quite possible that five PIs may just end up doing variations of the same thing).

The NIH has occasionally discussed grant caps, and the idea always generates fierce opposition from the scientists who would be affected (which is to say, the most powerful scientists in any given field).

In any event, the question is empirical, not political. An experiment could compare outcomes under different regimes — say, a maximum of three concurrent R01s in one arm, versus no cap in another — and track total scientific output, the diversity of funded ideas, and whether the cap pushes talented investigators to mentor more independent scientists rather than running everything themselves.

Side note: an NIH-wide policy makes little sense, in that a computational biologist working with public datasets needs a very different budget than someone running a wet lab with expensive reagents and a dozen postdocs.

9. Length of Grants

Most scientific grants last three to five years. Given review times and the need for resubmission, scientists often start planning renewals long before the current award ends. That can push them toward safer work with quicker payoffs.

As already discussed above, the well-known study by Pierre Azoulay and colleagues comparing HHMI investigators to NIH-funded researchers found that HHMI’s “person not project” model seemed to produce more impact. But one plausible hypothesis is that the real driver of HHMI’s success was simply the length of the grants: HHMI made seven-year awards with near-automatic renewal. That timeline gave researchers breathing room to tackle difficult problems where a breakthrough might take five years, in contrast to the NIH timeline where scientists need to stick to problems that will generate plenty of publishable results in the first three years. Pierre Azoulay even told me that he suspected that the time frame — seven years plus near-automatic renewal — was actually the key factor, rather than “person not project” per se.

If longer grants are the real driver, that would be one of the most practical policy levers available. Funders would not need to redesign their review systems, change their evaluation criteria, or adopt new metrics. They would just need to write bigger checks less often. A randomized experiment could compare shorter and longer grants, and then track the kinds of problems scientists choose, publication patterns, failure rates, and eventual impact.

10. Team Design: Size, Stability, and Turnover

What kinds of teams produce the best science? Research by Wu, Wang, and Evans (Nature, 2019) suggests that small teams are more likely to disrupt existing paradigms, while large teams are better at consolidating and extending established knowledge. A closely related question is whether it is better to fund a new assistant professor launching an independent lab or to add another person to a very large and already successful group. Funding the new investigator might increase the diversity of ideas and approaches, whereas funding the established lab may produce faster short-term outputs like papers and patents.

There’s another version of the question here: what is the impact of team/lab size not just on the science but on the people themselves? For example, if you’re a graduate student joining a lab with 30 people, you’re almost certainly signing up to work with someone who is well-known, and who will be able to further your career. At the same time, you could end up competing with many other trainees for the PI’s attention, and you may end up working on a small piece of a large project that was created long before you arrived. On the other hand, if you join a much smaller lab, you will likely get more face time with your advisor, more latitude to define your own project, and less internal competition.

We don’t actually know which environment (large lab vs. small lab) produces better scientists in the long run. An experiment (likely at NIH) could vary the likelihood of grant acceptance based on lab size, and then could track outcomes like scientific productivity of the labs as a whole, as well as individual-level outcomes as to the graduate students and post-docs that emerge from the labs. Whatever the outcome, this information would be useful both to funders and to thousands of students making a consequential decision with essentially no evidence to guide them.

11. Interdisciplinary Team Formation

Many important discoveries happen at the boundaries between fields. But interdisciplinary collaboration often amounts to the equivalent of stitching together an elephant’s trunk, a goat’s face, a tiger’s body, and a horse’s tail, and then expecting it to be a functional animal.

Simply putting people from different backgrounds together is not enough. The Human Genome Project succeeded as an interdisciplinary effort because it required biologists, computer scientists, and ethicists to work toward a shared concrete goal with clear milestones. By contrast, many university “interdisciplinary institutes” have become collections of disciplinary silos that happen to share a budget line. Goodhart’s Law takes effect: people aim to optimize for the measurement of interdisciplinarity as opposed to optimizing actual scientific work. Successful interdisciplinary work may require shared physical space, longer timelines, structured time for building common understanding, or individuals who can bridge multiple fields.

One design could compare different models for assembling and supporting interdisciplinary teams, including architectural design as to where offices are located and institutional choices as to which conferences people are funded to attend. There may be other ideas as to how best to create true interdisciplinary collaboration, and we should be exploring them all.

12. Training and Mentorship Methods

What kinds of scientific training actually work best? The usual model is apprenticeship: junior scientists learn by working closely with senior mentors. In the best cases, that model can result in high-power labs that produce a string of future Nobel Prize winners. But as far too many people know, direct apprenticeship can involve unhealthy power dynamics, PIs that take too much credit for other people’s work and ideas, and worse.

There is also a larger question about whether our best scientists are spending their time well. Consider an analogy from basketball. Steph Curry is the greatest shooter in NBA history, but he still works constantly with his shooting coach Bruce Fraser to find new ways to improve. There is no equivalent in science, i.e., where a Lasker winner works with a daily coach as to how to become even more creative and productive.

Why isn’t it possible to have the same sort of expert trainer in science — someone whose entire career is dedicated to finding new ways to help expert scientists get even better at management, creative thinking, and the like?

Instead, we do the opposite. We constantly “promote” the best scientists to be lead investigators in their own labs, where they then end up spending most of their time on fundraising, hiring, training and teaching, dealing with internal bureaucracy, and more, rather than on science!

We would never “promote” world-class athletes at the height of their careers to spend 90% of their time coaching other people and dealing with sports agents rather than actually playing. Instead, we let the top athletes focus on their sport while giving them external help from strength coaches, shooting coaches, and more.

An experiment here could compare different training structures and mentorship models over the long term, tracking productivity, career advancement, mental health, and eventual impact.

13. Early vs. Late Career Funding

Many of the greatest scientific breakthroughs in history have come from people in the early to mid-20s, whether Newton’s calculus, Einstein’s relativity, or Watson co-authoring the paper on DNA structure.

But in today’s academic system, there is almost nowhere that a genius at age 23 would be allowed to thrive. NIH data show that the average age at first R01 grant has risen from about 36 in 1980 to about 43 today. That is seven additional years of career spent in subordinate positions, often doing work that advances someone else’s agenda.

There are small-scale programs that attempt to provide funding to early-career investigators, but these seem insufficient. We should experiment with larger-scale efforts to steer more funding to younger investigators across all areas of science.

14. Graduate Student and Post-Doc Independence

Outside of some fellowship programs, graduate students and post-docs [which we will call “trainees” for short] are usually funded through their PI’s grants. That means that the trainee’s financial survival depends on keeping the PI happy.

This power dynamic can be unhealthy. Anecdotally, we have heard of many cases where a post-doc from another country (e.g., China or India) is forced to p-hack or even worse, because failing to produce a “good result” for a demanding PI might jeopardize their immigration status! More broadly than that, any trainee who disagrees with their advisor’s scientific direction, or who wants to pursue a side project, or who has a legitimate complaint about how they’re being treated, has to weigh those concerns against the fact that the PI controls their paycheck and ultimately their future career.

What if all trainees were funded independently through fellowships or training grants that they hold personally, rather than through their advisor’s research grants? The advisor would still provide mentorship and lab space, but the financial relationship would be different. A student with independent funding might be more empowered to try new ideas without worrying about whether the PI will cut them loose, since they can vote with their feet. And PIs would have to earn their trainees’ continued respect.

An experiment could randomly assign trainees to either PI-funded or independently-funded tracks within the same departments, and then compare their research productivity, the novelty of their work, their willingness to switch topics or labs, their completion rates, their mental health, and their long-term career outcomes. If independently funded trainees produce more original work and are more likely to stay in science, that would be a good case for restructuring how we fund graduate training and post-docs.

Incentives and Scientific Culture

15. Negative Results and Failure

Failure and null results are an inherent part of science. After all, if the results of an experiment could be perfectly predicted ahead of time, we would have already made that scientific discovery and the experiment wouldn’t be new science. But null and negative results are often hidden in file drawers because they are less exciting and less likely to be accepted for publication. As a result, other scientists can waste time on dead ends, and ideas can look much more certain than they really are.

The NIH Director is interested in requiring the publication of null results, as are people at the White House.

But therein lies a potentially interesting experiment. What if NIH created different disclosure requirements: some grants might require negative-results reporting, while other grants would just be business-as-usual.

The experiment could then track whether the requirement improves reproducibility, and if so, by how much. Over the long term, it would be interesting to see the effect on scientific culture and on the rate of discovery.

16. Red-Teaming Scientific Consensus

When a field is dominated by one theoretical framework or approach, should science agencies deliberately fund one or more “red teams,” i.e., serious critics of the current paradigm?

Scientific consensus can often turn into groupthink, whereas we know that many scientific breakthroughs were initially dismissed, ignored, or even ridiculed.

Red-teaming means deliberately supporting researchers who challenge the currently dominant ideas. This may be especially useful in fields where progress has stalled despite large investments over decades, or where the dominant theory has not been tested as rigorously as people assume. The Templeton Foundation has funded some work along these lines in physics, supporting challenges to certain interpretations of quantum mechanics.

An experiment could identify fields with an especially strong consensus around a view that has yet to be unambiguously successful in the real world, fund red teams in some but not others, and compare whether that type of funding leads to new insights and potentially breakthroughs.

Commercialization and Translation

17. Optimal Commercialization Pathways

What is the best path from discovery to market?

The standard university route runs through tech transfer offices, a system created largely by the Bayh-Dole Act of 1980, which gave universities the right to patent federally-funded inventions. But that system is often criticized as slow, bureaucratic, and ineffective, and it might be a far better idea to give professors/inventors the presumptive right to file for patents and negotiate with venture capital directly, with the university perhaps getting a small automatic fee.

The best pathway probably depends on the technology, the market, and the institution. We could imagine several experiments that would assign similar discoveries to different commercialization routes, and compare the time to market, financial returns, researcher satisfaction, and eventual social impact.

Methodological Rigor

18. Replication Requirements

Replication is an inherent part of the scientific process, yet too many scientists refuse to engage in direct replication because it is less likely to be funded or published. We have a collective action problem here, and large funders like NIH or NSF could solve it, because everyone cares about funding.

The Reproducibility Project: Psychology (2015) attempted to replicate 100 published studies and found that only about 36 percent produced statistically significant results in the replication, compared with 97 percent in the originals. The Reproducibility Project: Cancer Biology found similar results; in the cases where they even attempted a replication experiment (often impossible due to the lack of details and a lack of cooperation from the original labs), the effects in the replication experiments were only 1/6th the size of the original publications!

That said, we can’t replicate everything–that would be incredibly inefficient. How does this turn into an experiment? An experiment could vary replication requirements across similar research areas and compare both the reliability of published findings and the pace of genuine discovery.

19. Preregistration Impact

Does preregistration improve credibility, or does it get in the way of exploration? The theory behind preregistration is that it will reduce p-hacking, HARKing, and other questionable practices. But it may also fit some fields much better than others, and in some settings it may add bureaucracy without much gain.

In some fields (clinical trials), preregistration is already standard practice and is even legally mandated for trials submitted to the FDA. That said, we could do experiments in other fields, such as preclinical research. An experiment could require preregistration for some grants but not others, and then compare reproducibility, false positives, and the rate of useful exploratory findings. We should also study the model of registered reports, although a funding agency would need to collaborate with journals so as to set up a system in which papers were accepted based on the research design rather than the results.

Artificial Intelligence

Caveat: AI is a fast-moving field, and all of the experiments below would make sense if, and only if, we could deploy them quickly and get feedback on results within a year (or hopefully 6 months).

20. AI for Detecting Scientific Fraud and Errors at Scale

How successfully could AI screen the scientific literature for fraud, errors, and questionable practices at a scale humans cannot match? Today, most error detection relies on whistleblowers along with a few intrepid and dedicated investigators like Elisabeth Bik. But new AI tools could help scan thousands of papers for image manipulation, impossible numbers/data, or suspicious statistical patterns.

As an experiment, we could randomly assign different fields or sub-fields to be examined by different tools that might detect fraud or questionable research practices. We would then check for the downstream impacts on practices in those fields (e.g., what some would call data hygiene and similar issues). At the same time, we would want to look for possible downsides, such as a drag on creativity or exploration. As well, we would want to check for possible gaming (i.e., maybe people who are more worried about being caught p-hacking or cheating will just up their game by finding more creative ways to cheat).

21. AI Impact on Scientific Diversity

Will widespread AI use make science more diverse or more uniform? If most scientists use similar AI tools, research may start to converge more and more on the same methods, questions, and frameworks. A recent research paper by James Evans and his colleagues showed that scientists using AI published more papers but with a narrower range of topics and engagement with colleagues. On the other hand, AI tools could (at least in theory) help scientists borrow methods and ideas from distant fields.

This is more naturally studied as an observational or quasi-experimental design than as a true randomized experiment, since researchers cannot easily be prevented from using whatever tools they choose. We could compare fields with different patterns of AI adoption and track whether their agendas and topics become narrower over time.

22. AI-Driven Experimental Design

Can AI systems design better experiments than human scientists? This possibility might be especially promising in areas like materials science, drug discovery, and synthetic biology.

But AI-designed experimentation might miss out on tacit knowledge, real-world constraints, or unusual observations that a human would notice. A metascience experiment could partner with AI firms to look at human and AI workflows; we would want to look at results like novelty and surprisal (i.e., does a proposed experiment draw on a wider literature and propose more novel ideas to test)?

23. AI as Assistant for Peer Review

Peer reviewers for federal agencies are usually overworked and behind schedule (as has been the case for me on many occasions!). AI could help, if allowed. That is, AI tools could help peer reviewers by checking basic statistics, determining the novelty of a proposal, and evaluating a scientist’s prior work and track record. That sort of assistance could free human peer reviewers to focus on larger issues, such as the significance of the ideas at hand and the theoretical advances being proposed.

That said, AI-assisted review could lead to hidden bias against certain people or ideas, as well as reducing human attention in ways that undermine the process.

We could launch an experiment here by varying the level of AI assistance across peer reviewers, while looking at any number of outcomes (speed, quality, bias, novelty, and user acceptance).

24. AI-Assisted Grant Writing and Proposal Inflation

One of the most immediate uses of AI in science is likely to be grant-writing. That could save time, but it could also make proposals more formulaic and harder to distinguish. Moreover, federal agencies like NIH have already moved to ban the use of AI (“Applications that are either substantially developed by AI or containing sections substantially developed by AI are not considered the original ideas of applicants and will not be considered by NIH.”).

An experiment could randomly provide some applicants with access to approved AI tools and training, while other applicants would be forbidden from using AI. Reviewers could also be randomized as to whether they know that AI assistance was permitted. The study could then measure proposal quality and novelty, funding rates, and ultimately, the downstream research that gets funded.

25. Autonomous Lab Platforms vs. Conventional Lab Workflows

Do partially or fully autonomous labs produce better science than conventional labs?

There is a great deal of interest (and hype!) around self-driving labs and automated experimentation. But we don’t actually know whether these systems and companies will improve the quality of science and the follow-on innovation, rather than just increasing the volume of science being produced. Science isn’t measured by the pound, after all.

A metascience experiment could compare projects conducted through conventional lab workflows versus more automated or partially autonomous systems. We would look at the cost, the time spent, reproducibility, and the rate of genuinely surprising findings.

Scientific Communication and Publication

26. Adversarial Collaboration

Some scientific disputes go on for years because each side runs studies with its own favorite methods and theories, but dismisses the other side’s work. Adversarial collaboration is a concept developed by Daniel Kahneman, who described it in a 2003 American Psychologist article and later in Thinking, Fast and Slow. The idea is to force both sides to agree in advance on the design and analysis of an experiment, while committing to accept the results no matter how they turn out.

One of the best-known examples is the adversarial collaboration between Kahneman himself and Gary Klein on expert intuition, which produced a joint 2009 paper. The two researchers originally held different views about when expert judgment could be trusted. By working together, they were able to identify the specific conditions under which each view was correct, a result that neither would have reached alone.

A funding agency like NIH could create an adversarial collaboration program, fund such efforts in some fields but not others, and then compare whether scientific disputes are resolved faster and whether the process generates insights that neither side saw in advance.

27. Open Science Mandates: How Much Openness Is Optimal?

How much openness is actually best? As much as I have argued for open science for the past 15 years, mandatory openness could also create perverse incentives. That is, scientists might hold results back until they have extracted every paper they can, or they might avoid collecting expensive data if their competitors can free-ride immediately.

The NIH’s 2023 Data Management and Sharing Policy is the largest-scale openness mandate in US science. It is still too early to assess its effects, which is precisely why experimental variation in open science requirements would be valuable.

A metascience experiment could randomly assign different openness rules within the same grant program, such as immediate sharing, delayed sharing, data-only sharing, or sharing on request. It could then compare the levels of reuse and reproducibility over time, as well as the cost. (Not all data sharing is worth paying for it forever, and we should be measuring the cost of preserving all data versus the benefits as in how often it is reused).

Research Environment and Institutions

28. Geographic Concentration vs. Distribution of Funding

Is it better to concentrate funding in elite institutions and places, or spread it more widely around the country? Right now, research funding tends to be heavily concentrated in a small number of geographic clusters on the coastal US, with exceptions like Houston, Austin, Chicago, etc. Some think that this concentration of funding makes some sense. Perhaps there are agglomeration effects arising from dense talent networks and serendipitous contact from people who run into each other more often.conven

That said, some of the best ideas in research come from unexpected places, like Katalin Kariko in the 1990s. Perhaps it is the case that federal funding should be spread around the nation more broadly in an attempt to elicit more innovative ideas from places and people that wouldn’t have seemed obvious in advance.

A metascience experiment could direct supplementary funding to researchers outside major hubs and then compare their productivity and impact with similar researchers at elite institutions.

29. The Optimal Scientific Conference

Scientists spend a lot of time and money on conferences. The assumed benefits seem to be obvious: scientists get to hear the latest work in their field, and to have a more serendipitous exchange of new ideas. But we have no idea how much any of that actually matters.

A metascience experiment could fund a bunch of small workshops, virtual meetings, and unconferences, across many fields, and then look for outcomes such as the extent of new collaborations and the rate of idea generation.

30. The Sabbatical

The idea of a sabbatical for researchers rests on the idea that any scientist needs time for deep work, rather than being distracted by the constant demands of teaching classes and committee involvement. That intuition is widespread, but we don’t know actually know how much difference it makes (if any) to let scientists have sabbaticals.

A metascience experiment could randomly offer some mid-career researchers a fully funded semester or year with no teaching, administrative work, or grant-writing obligations. We would then compare their creativity, well-being, and career satisfaction with a control group. For the first time, we would have rigorous evidence about one of the most widespread academic practices.

Alternative Funding Models

31. Prizes versus Grants

Grants fund inputs, while prizes fund outputs. Quite the opposite approach! Prizes seem to work best when the problem is well-defined and success is measurable. A 2007 Harvard Business School study of InnoCentive (now part of Wazoku), an online prize platform for scientific and technical problems, found that problems were more likely to be solved by people from outside the problem’s home discipline.

But as yet there is little systematic evidence that prizes work better than grants. A metascience experiment could take a set of well-defined scientific problems and randomly assign some to be funded through grants and others through prize competitions. We would then compare the quality and speed of solutions, the diversity of applicants and solutions, and the total cost of the project.

32. Forecasting Tournaments for Scientific Proposals

Grant review at major funding agencies usually asks peer reviewers to come up with overall scores and narrative judgments, but rarely are they asked for clear predictions of whether a grant will meet its milestones. (Caveat: I have served as a peer reviewer for NSF, and their particular program did ask us all to estimate probabilities as to grant outcomes.)

Philip Tetlock’s work on superforecasting — including the IARPA-funded Good Judgment Project and his 2015 book Superforecasting — showed that some people are better calibrated than others when making predictions about uncertain events. Not only that, this skill is measurable and trainable.

The same principle could be applied to science funding. Reviewers and program officers could be asked to estimate the probability that a proposal will lead to a major paper, a useful tool, a replicable result, or a successful translation.

An experiment could then compare standard peer review versus explicit forecasting. Do the reviewers asked to make forecasts end up making better judgments? And how do we calibrate reviewer judgments over time? These are all unanswered questions in metascience.

---

Again, if even three or four of these experiments are actually run in the next few years via new “metascience offices” at federal agencies or through a potential lab or fund studying Inquiry itself, we would be pleased.

We are actively looking for bright minds to develop these lines of inquiry with us: if this is you, reach out to us at aish@analoguegroup.org and stuartbuck@goodscienceproject.org.

---

Thanks to Tom Kalil, Kris Gulati, and Misha Teplitsky for feedback on this essay.