External Value - CUREs and Reproducibility

Concept

When I first heard about the tenets of a CURE, from what might be the most often-cited publication about this topic (Auchincloss et al. 2014 CBE LSE), this essential concept was the most surprising to me: somebody other than the students and the instructor should care about the outcome of the research project. While I still think that this might be the least important core tenet, I've come to realize that it is still a valuable characteristic of a CURE, and that perspective is what this post is about.

Example

Presumably, like with Discovery and Autonomy, it could be that External Value will provide further motivation for students to be deeply engaged in their research project. Generic examples of External Value include those projects designed to address a local need (e.g. to research and develop sustainability plans for a college campus), or a global need (e.g. the discovery of new antibiotics produced naturally by bacteria collected by students from nature).

The real point is that the project have no obvious extrinsic utility. Again, this tenet closely aligns with Discovery, in that projects that do not explore new ideas or test new hypotheses will likely also not have External Value.

What's in it for me?

I'm really excited to write this post, to share what I think could be a very important focus that many instructors could adopt that could provide a great service to our students and to the scientific community (and to us instructors) at the same time.

Replication of experiments is critical to the scientific enterprise, in part because we should strive to ensure that others don't build on potentially faulty experimental work of our own, and certainly because knowing which results are able to be replicated and which are not is crucial for understanding the extents and limits of interpretation of those data.

However, our community has also developed an environment that does not actively support replication.

For example, the "first to publish" system discourages replication by removing considerable acknowledgment for spending effort (time and materials) on replicating other studies. Counterpoint: as a famous but rare example of when replication was undertaken in a rapid and concerted effort (but in a clear effort to debunk what many initially interpreted to be published data that were very likely not able to be replicated), recall the publication in Science of the extremophile bacterium GFAJ-1 that was claimed to be able to use the toxic atom arsenic in place of phosphorus in its DNA (Wikipedia Summary) and note how many high-profile researchers immediately undertook steps to publish concerns and then follow up with experiments that refuted the initial publication. Why do we see this sort of response so rarely? In part, it is likely because it takes considerable resources (particularly including time) to coordinate a meaningful (i.e. rapid and appropriately rigorous) replication of any published study.

Second, and perhaps more insidiously, funding agencies (rightly so), as well as most scientific journals, expect grant recipients to rigorously conduct experiments and to ensure reproducibility. As an outcome, then, most funding agencies don't provide overt opportunities to apply for funding to replicate prior findings.

Third, like funding agencies, most journals are about as excited at the prospect of publishing a replication study about as they are publishing negative results (which, if you don't know, is not usually a solid publication strategy).

I propose that CUREs are perfectly suited to help students learn techniques, to read scientific literature, and to produce publishable results that benefit them, their instructor, and the community. I'll argue here that a CURE is a great way for an instructor to conduct publication-quality research.

CUREs and Reproducibility

CUREs, in an instant, can address these major obstacles to scientific replication.

First, the instructor is already committed to spending time with their students, and the students can comprise a significant workforce for conducting research. I'll discuss potential concerns about this in a future post (e.g. how to support the quality of research that novice undergraduates perform). Thus, the question of workload needed to conduct a replication study is at least mostly obviated.

Second, because many lab courses already have lab equipment and are funded by student lab fees for reagents and lab supplies, the need for funding, for the carefully chosen replication study, becomes moot.

Finally, the advent of journals like Experimental Results (Cambridge Univ. Press) that aim to publish probably low-impact studies that are conducted rigorously and that expect only short-format (e.g. a couple of pages of text with one figure) are well suited to publishing reproduction studies and to allow not only the instructor but also their undergraduate students to publish!

Example

In Fall 2019, my Genetics and Cell Biology Lab CURE had a few groups that decided to almost-replicate prior studies. The "almost-replicate" is important because of the tenet of Discovery. I realized that a CURE might appropriately balance Discovery with replication when students discover and read a published manuscript, modify the published materials and/or methods in a systematic way, and then conduct a rigorous experiment. For example, one group took an experiment conducted in one species (Caenorhabditis elegans) and have replicated it in C. briggsae. Although this is not direct replication, it provided the students with a clear and literature-based hypothesis to test. Although it did not present a great opportunity for Autonomy, the students did take ownership of their original project, and their term-end results did support that the same effect true in C. elegans also occurs in C. briggsae.

Now, while this was not earth-shattering news, it is a novel discovery that is well-suited for publication in a short-format journal. So, we are currently working on completing edits on such a manuscript to send for consideration for peer review. Meanwhile, to be really sure that our replication can be replicated, I'm repeating the same experiment that the students performed during the academic term to ensure I collect similar data.

I hasten to add that this was the outcome from one group out of 30 last semester; that most student groups might not collect potentially publishable research. But, when I announced at the start of the term that there was a possibility that their term-end written project report (which was due in the format of a short-format journal publication…) could be submitted for publication, this was a clear motivator!

Why would I spend the time to do all of this extra work, mentoring a student group after the term was over and using my own time to replicate their study, even when it was not directly aligned with my own research lab's trajectory? My decision was circumstance-dependent, and I'll let you in on my next insight as a recently-tenured but still grant-hungry PI: if you are the instructor of a CURE where you can demonstrate that your students can generate publication-quality research, might you be able to leverage that toward an NSF broader impacts outcome of your next proposal?

Summary

CUREs, given the proper attention, and in the correct environment of instructor interest and institutional support, stand to make an impact on simultaneously helping students read and interpret published research studies, edit and then follow methods meant to replicate those studies, and then publish the results. At its most simple, and perhaps most effective, CURE students might identify one experiment in a published study that conforms to the budget, equipment, and other constraints. They could then simply reproduce the study (which might not quite be CURE-worthy, since that would not involve any Autonomy), or they might decide to make one simple but potentially important change (e.g. a temperature, or a species, or an incubation time), such that the experimental outcome is truly not known (Discovery). Either way, with appropriate support, you might not just be teaching a class when you instruct a CURE - you might be on your way to more publications. More importantly, you might be on your way to having undergraduate co-authors who have experienced so much more in your CURE course than just how to follow a protocol, get the correct result (or not), and have made no difference for their efforts.