External Value - CUREs and Reproducibility

Concept

When I first heard about the tenets of a CURE, from what might be the most often-cited publication about this topic (Auchincloss et al. 2014 CBE LSE), this essential concept was the most surprising to me: somebody other than the students and the instructor should care about the outcome of the research project. While I still think that this might be the least important core tenet, I've come to realize that it is still a valuable characteristic of a CURE, and that perspective is what this post is about.

Example

Presumably, like with Discovery and Autonomy, it could be that External Value will provide further motivation for students to be deeply engaged in their research project. Generic examples of External Value include those projects designed to address a local need (e.g. to research and develop sustainability plans for a college campus), or a global need (e.g. the discovery of new antibiotics produced naturally by bacteria collected by students from nature).

The real point is that the project have no obvious extrinsic utility. Again, this tenet closely aligns with Discovery, in that projects that do not explore new ideas or test new hypotheses will likely also not have External Value.

What's in it for me?

I'm really excited to write this post, to share what I think could be a very important focus that many instructors could adopt that could provide a great service to our students and to the scientific community (and to us instructors) at the same time.

Replication of experiments is critical to the scientific enterprise, in part because we should strive to ensure that others don't build on potentially faulty experimental work of our own, and certainly because knowing which results are able to be replicated and which are not is crucial for understanding the extents and limits of interpretation of those data.

However, our community has also developed an environment that does not actively support replication.

For example, the "first to publish" system discourages replication by removing considerable acknowledgment for spending effort (time and materials) on replicating other studies. Counterpoint: as a famous but rare example of when replication was undertaken in a rapid and concerted effort (but in a clear effort to debunk what many initially interpreted to be published data that were very likely not able to be replicated), recall the publication in Science of the extremophile bacterium GFAJ-1 that was claimed to be able to use the toxic atom arsenic in place of phosphorus in its DNA (Wikipedia Summary) and note how many high-profile researchers immediately undertook steps to publish concerns and then follow up with experiments that refuted the initial publication. Why do we see this sort of response so rarely? In part, it is likely because it takes considerable resources (particularly including time) to coordinate a meaningful (i.e. rapid and appropriately rigorous) replication of any published study.

Second, and perhaps more insidiously, funding agencies (rightly so), as well as most scientific journals, expect grant recipients to rigorously conduct experiments and to ensure reproducibility. As an outcome, then, most funding agencies don't provide overt opportunities to apply for funding to replicate prior findings.

Third, like funding agencies, most journals are about as excited at the prospect of publishing a replication study about as they are publishing negative results (which, if you don't know, is not usually a solid publication strategy).

I propose that CUREs are perfectly suited to help students learn techniques, to read scientific literature, and to produce publishable results that benefit them, their instructor, and the community. I'll argue here that a CURE is a great way for an instructor to conduct publication-quality research.

CUREs and Reproducibility

CUREs, in an instant, can address these major obstacles to scientific replication.

First, the instructor is already committed to spending time with their students, and the students can comprise a significant workforce for conducting research. I'll discuss potential concerns about this in a future post (e.g. how to support the quality of research that novice undergraduates perform). Thus, the question of workload needed to conduct a replication study is at least mostly obviated.

Second, because many lab courses already have lab equipment and are funded by student lab fees for reagents and lab supplies, the need for funding, for the carefully chosen replication study, becomes moot.

Finally, the advent of journals like Experimental Results (Cambridge Univ. Press) that aim to publish probably low-impact studies that are conducted rigorously and that expect only short-format (e.g. a couple of pages of text with one figure) are well suited to publishing reproduction studies and to allow not only the instructor but also their undergraduate students to publish!

Example

In Fall 2019, my Genetics and Cell Biology Lab CURE had a few groups that decided to almost-replicate prior studies. The "almost-replicate" is important because of the tenet of Discovery. I realized that a CURE might appropriately balance Discovery with replication when students discover and read a published manuscript, modify the published materials and/or methods in a systematic way, and then conduct a rigorous experiment. For example, one group took an experiment conducted in one species (Caenorhabditis elegans) and have replicated it in C. briggsae. Although this is not direct replication, it provided the students with a clear and literature-based hypothesis to test. Although it did not present a great opportunity for Autonomy, the students did take ownership of their original project, and their term-end results did support that the same effect true in C. elegans also occurs in C. briggsae.

Now, while this was not earth-shattering news, it is a novel discovery that is well-suited for publication in a short-format journal. So, we are currently working on completing edits on such a manuscript to send for consideration for peer review. Meanwhile, to be really sure that our replication can be replicated, I'm repeating the same experiment that the students performed during the academic term to ensure I collect similar data.

I hasten to add that this was the outcome from one group out of 30 last semester; that most student groups might not collect potentially publishable research. But, when I announced at the start of the term that there was a possibility that their term-end written project report (which was due in the format of a short-format journal publication…) could be submitted for publication, this was a clear motivator!

Why would I spend the time to do all of this extra work, mentoring a student group after the term was over and using my own time to replicate their study, even when it was not directly aligned with my own research lab's trajectory? My decision was circumstance-dependent, and I'll let you in on my next insight as a recently-tenured but still grant-hungry PI: if you are the instructor of a CURE where you can demonstrate that your students can generate publication-quality research, might you be able to leverage that toward an NSF broader impacts outcome of your next proposal?

Summary

CUREs, given the proper attention, and in the correct environment of instructor interest and institutional support, stand to make an impact on simultaneously helping students read and interpret published research studies, edit and then follow methods meant to replicate those studies, and then publish the results. At its most simple, and perhaps most effective, CURE students might identify one experiment in a published study that conforms to the budget, equipment, and other constraints. They could then simply reproduce the study (which might not quite be CURE-worthy, since that would not involve any Autonomy), or they might decide to make one simple but potentially important change (e.g. a temperature, or a species, or an incubation time), such that the experimental outcome is truly not known (Discovery). Either way, with appropriate support, you might not just be teaching a class when you instruct a CURE - you might be on your way to more publications. More importantly, you might be on your way to having undergraduate co-authors who have experienced so much more in your CURE course than just how to follow a protocol, get the correct result (or not), and have made no difference for their efforts.

Discovery & Autonomy

The Concept

Thinking back to my childhood (and, really, my adulthood), I know that the lessons I learned from making a mistake and then having to correct have been longer-lasting, at least in my conscious memory! As states one of my favorite quotes, defining "Experience" as "what you get after you needed it."

So, the teaching philosophy that I developed includes providing a safe and supportive space for students to try new things, which is a great way to stimulate intellectual growth. Providing time for Iteration partly supports this pedagogical thrust.

A CURE is a great place to promote Autonomy and Discovery. In short, support students in developing their own experiments (Autonomy) that do not already have known outcomes (Discovery).

Why would Autonomy and Discovery be critical tenets for any undergraduate course, let alone a CURE? In many situations, allowing students to develop and test their own hypothesis will be novel - something they haven't experienced in prior classes. This could benefit them in at least a couple of ways. Students who have designed their own experiment might feel more invested in the process of conducting the experiment. They might take more ownership and thus be more motivated to undertake the efforts to carefully conduct their project.

Likewise, it seems likely that students could have better focus and thus develop a greater understanding of the scientific method, including techniques and data analysis approaches, when they test a hypothesis that has no prior support. In other words, won't it be more interesting for students to work on a project where they are in charge of definitively testing a new hypothesis and are at the forefront of Discovery? And, after all, this is how "real" science works - so why wouldn't we provide this sort of experience to our students?

What's in it for me?

Why would the instructor want to take the time to develop a CURE that supports Discovery and Autonomy? Stay tuned for future posts addressing this question, but for now: I can personally attest that supporting novice scientists with the latitude to ask their own questions can be stimulating and engaging for the instructor, and it can also lead to the development of new research ideas and data that the instructor might use for research publications and grant proposals (with some caveats about intellectual property that I'll write about in future).

Details

I hasten to add that there are always real limitations on the types of projects that students can devise. Several relevant factors involve time, resources/budget, and safety (we might collectively call these "Feasibility.") Separately, the instructor will probably have an internal or external edict to constrain research projects to particular topics and techniques that align with the Student Learning Objectives for the course.

For example, in my Genetics and Cell Biology lab CURE, some groups hypothesized that the worms we use as an experimental system would experience reduced fertility at increasing temperatures. This is a testable hypothesis, and it was a technically feasible experiment. But, I provided some feedback to the group to suggest that they think of other types of experiments. While this intervention did slightly and at that moment reduce their Autonomy, it was justified for two reasons. First, as a subject matter expert, I knew that such experiments had already been published - thus the proposed experimental design lacked Discovery. Second, testing the effects of environmental variables on the biology of an organism is more of an ecology research project than a genetics or cell biology experiment. So, there are certainly times when the instructor will need to moderate the balance of course goals and CURE tenets.

Example

In my CURE, student groups present short oral presentations on specific manuscripts and topics at the start of the term. This helps them begin to learn about what types of questions and experiments have already been conducted and published. Then, I provide feedback on the feasibility of their hypotheses and methods, and whether they appropriate align with the course Student Learning Objectives, by assigning draft and final experimental design papers that include peer and instructor evaluation in between.

Summary

Best practices for designing a CURE that facilities Discovery and Autonomy include:

• integrate reading primary literature that is contemporaneous and relevant, so that students can develop a sense for past research related to their topic of interest
• during the project development phase, give regular feedback to steer students toward hypotheses and designs that are feasible
• build time into the course schedule for iteration: experiment, revise the design and/or hypothesis, and repeat

Thus, an effective CURE can balance the absence of a "right answer" to an experiment (Discovery) and allowing students to explore the discipline relevant to the class (Autonomy). In the next post, I'll provide a specific structure for a great way to design a CURE that meets all of the CURE tenets thus far: Iteration, Discovery, and Autonomy. Most importantly, I'll share how incredibly easy it can be to do all of this for very little work on the instructor's part!

Iteration

Now that I've introduced the basics of a CURE, I'll detail each of the tenets in its own post. However, many are inter-related, so it was difficult to choose which order to introduce the following posts. Here goes!

One of the tenets of a CURE is Iteration.

The Concept

Iteration sets a CURE apart from other types of course-based laboratories in the same way as another tenet: Discovery. While Discovery awaits its turn in the blog queue, I'll introduce it here briefly. Discovery means that students engage in research to which the answer is not known in advance. Some call this "authentic" research, because it forces students to grapple with an undoubtedly critical reality in science: practicing scientists have to be thorough and precise to design and execute an experiment that will (hopefully) definitively address a hypothesis.

For the instructor, and for the student, Discovery can be uncomfortable. Students grapple with the uncertainty that there is no "correct answer" (although they'll still ask you repeatedly to review their lab notebook to make sure their "data look right.") Instructors have the advance task of designing a class schedule that is flexible enough to accommodate the reality that experiments won't always work properly the first time.

In my intro chemistry labs, each experiment had a correct answer, and if we didn't arrive at the correct answer, we didn't earn full credit on that assignment. But if we didn't grasp the intricacies of that experiment, that was often OK, because the next week we were doing something totally different.

However, in many CUREs, as in all scientific research, if something doesn't work properly the first time, one either has to troubleshoot (which usually involves making a change to the protocol and repeating it) or identify a different method to perform the same sort of analysis. Because each project will encounter different hurdles, it is an important challenge for the instructor to design the schedule of a CURE to accommodate the fact that the experiments students will design will need some time for revision and repetition. Iteration.

Example Course

I teach in a semester-system school, so while I have much more time than those on the quarter system, I still struggled to fit what I thought would be enough "flexible time" into the class schedule. That struggle is why I'm sharing with you now the second iteration of my class schedule. The first iteration seemed to work pretty well last semester, and now I've done what any good scientist would do: collected data (feedback and observations) and revised for the coming term.

Before getting into the details, my own philosophy about lab course design for a CURE (and for most courses, for that matter) is to dispense with "breadth" and "content coverage" in exchange for "depth" in fewer topics. My perspective is that this leads to longer-lasting retention of the important scientific concepts I emphasize.

With that in mind, let's look at my course in detail. I supervise non-tenure-track faculty (lecturers) who teach an undergraduate, upper-division lab course in Genetics and Cell Biology that is required for all biology majors. The course meets once weekly in one three-hour block for a semester.

Enrollment tends to be about 18 students per section, and because of equipment (and space) limitations, we tend to have students work in six groups of three in this lab course.

Later (in another post), I'll delve into more detail about guidelines for how to pick a research area or topic for a CURE. Suffice it now to note that it makes a lot of sense (for so many reasons…) for the instructor to pick a research topic that aligns with their own research agenda/expertise. In my case, I wanted to develop a CURE related to worm genetics (specifically, in the species I study, Caenorhabditis briggsae).

Historically, this lab course introduced students to what our molecular/cell biology/genetics faculty felt were critical skills, such as microscopy (the main cell biology component) and Mendelian genetics and molecular genetics (e.g. micropipetting, PCR, and agarose gel electrophoresis). So, my plan for this course, redesigned as a CURE, was to instruct students on these techniques, as well as to the biology of C. briggsae, and then to ask them to design and execute research projects using at least some (but not necessarily all!) of these skills to study C. briggsae biology.

This presented the first question as I thought about the course schedule: how would I structure the introduction of students to the fundamentals of working with worms (e.g. life history and development, anatomy, husbandry, and setting crosses and so - necessarily - manipulating 1 mm-sized organisms) and of the techniques they might need to use in experiments they would design (e.g. measuring worm lifespan and fecundity, PCR and gel electrophoresis, microscopy).

CURE Schedule and Major Assignments

This coming semester, we have fifteen class meetings. The first half of the semester is dedicated to learning about the experimental species and practicing those core cell and genetics techniques (microscopy and PCR).

Importantly, in my experience, we've moved away from instructor "lecturing" about worm anatomy, how to set crosses, the genetic diversity among different populations, and so on. I found in past semesters that, when students presented their research at the end of the term, they couldn't explain WHY they had performed the experiment that we had led them to conduct. I reasoned (which is perhaps true) that it was our fault: we hadn't given students the opportunity and incentive to fully understand the intellectual basis and motivation for "going through the motions" in their lab exercises every week.

So, how did the lab experience evolve? Now, each week during the first four weeks, we introduce and have students practice a new technique: 1) microscopy and viewing worms, 2) pipetting and worm manipulation (picking up individual worms and moving them from one Petri dish to another, 3) pipetting (again, because students really need the iterative practice!) and then setting PCR, and 4) agarose gel electrophoresis. And, because PCR does not always work well the first time, we built in some time for Iteration: weeks 5 and 6 also have time set aside for each group to repeat the PCR and then gel.

Now, none of these exercises really takes a full three hours, and there are already some gaps of time (because once you set up the week 3 PCR reaction, it takes four hours for that reaction to complete…). So, perhaps oddly for a lab course, the rest of the in-lab time in weeks 3–4 involve small group presentations by students on worm biology and then on worm genetics research papers. Each group is assigned a different worm biology topic (and the next week a different research paper) and presents it to the class. The point of having students do this instead of the instructors is at least four-fold (please comment if you have additional thoughts about benefits of this approach!)

• This provides opportunities for students to practice oral presentations with their group members and to become more comfortable talking in front of the class (particularly important because they give an oral presentation of their research project at the end of the semester)
• Students who are giving the presentations learn their material better than they do when listening to the instructor tell them the same information
• Likewise, we've observed that students listening to peer presentations tend to be more engaged and alert (and willing to ask questions) than when an instructor gives the presentation
• Students practice reading both lay and review literature as well as primary research literature. Not only is this a good skill for undergraduate STEM students to develop, but we've also observed that this really helps students understand the intellectual background behind the published research projects they present (and later helps them understand and rationalize the research projects they will develop that are, at least in part, based on this literature review)

We also provided rubrics for the presentations and pointed groups directly to specific topics and literature to read, and specific points to consider making in their presentations. For example, in this worm species, there are two sexes: male and hermaphrodite. They can be visually distinguished, and it is important to do so if setting a cross. So, for the group that presents on worm anatomy, we point them toward published literature that contains useful diagrams and we suggest that the students describe how to distinguish the sexes under the microscope.

Back to the overall course schedule. After week 4, the introductory student presentations are complete, and (as mentioned above) weeks 5 and 6 are used by groups to perform any repetition of the fundamental lab techniques that might not have worked well the first time around (Iteration). In this CURE, even if PCR and gel electrophoresis worked well the first time, we still have students continue to practice isolating and moving worms, for example. During these same two weeks, students are also developing a draft hypothesis and experimental design, which they present (very briefly) to the class, and discuss with other classmates to get any feedback. Over the week, the instructor then reads and provides suggested changes to the hypothesis and to the experimental design. For the hypothesis, we stress that it must be founded on published research (i.e. they need to cite research that supports the intellectual merit of the hypothesis) and also related to the course topic (Genetics and Cell Biology). For the experimental design, we provide advice on what is feasible to accomplish given equipment and supplies (cost) as well as the time available for experimentation.

Then, the next week (6), the students make any revisions to the written design during class and turn in a "final" version (with the understanding that experiments are almost always in a constant state of change…). This version, most importantly to us, is accompanied by a list of anticipated materials each group will need. More on that aspect of CURE coordination to come in a future post!

The groups then have the next six weeks to conduct their experiments. There is no lab agenda, other than every student must show up and help their group! This gives most groups enough time for Iteration! If an experiment doesn't work the first (or second) time, they have the opportunity to demonstrate their perseverance as scientists, making changes to protocols and trying again.

Week 13 is dedicated to data analysis and preparation of their written report and their oral presentation, although the instructor consistently helps students analyze and interpret their experimental results during the entire research process (e.g. conducting and interpreting statistical tests, creating visual representations of their data, like graphs).

Finally, each group presents an oral report of their project to the class in Week 14. And that's the end (other than a lab practical final exam in Week 15). Of course, there are many more details to share about the end of the term, and I'll address them in a later post on another CURE tenet: students should be Assessed like scientists.

Summary

Ideally, the course schedule for a CURE will balance keeping students engaged and active (e.g. having meaningful activities each class meeting) and the flexibility for different students or groups to move forward at different paces. In a perfect world, the schedule will help students feel that there is no pressure for an experiment to work as expected the first time. It is important for scientists to realize that Iteration is a normal part of the process! Devising this type of schedule is no easy task. The approach I've described above, in which the first third of the course is dedicated to learning techniques and background information, with the rest of the course being relatively adaptable depending on progress, has worked well for me, and I hope this approach helps you consider how to schedule your CURE course!

If you find yourself struggling to eject "content" from your course to allow "unscheduled" time for Iteration, then I leave you with this thought. I have had numerous students approach me at the end of this CURE course design and offer various positive reflections on their experiences. In general, they tend to follow the topics of "I wish I had been able to take a class like this earlier," and "I finally feel like I understand the scientific method and how real research projects work." In other words, with your support in a CURE, students can develop important scientific skills (but maybe not quite all of the discipline-specific content you would prefer…) like information literacy, perseverance, troubleshooting and critical thinking, among others.

Content exposure can be important, especially if your course is a prerequisite for another, but there are still ways to facilitate student exploration of that content in a CURE.