Use this Table of Contents to navigate this article!
- The Essential Questions to Ask Your Mentees Early On During a Research Project
- My Most Valuable Question to Ask Research Mentees (or Just About Anyone Doing Research)
- Value #1: It Puts One’s Questions, Hypotheses, and Predictions to the Test (Pun Intended).
- Value #2: It Puts One’s Data (and the Analysis Thereof) at the Heart of the Design Process.
- Value #3: It Frames the Entire Project in the Lens of Communication from the Outset.
- Value #4: It Provides Clarity to What Can Often be a Winding Journey.
While my tone and presentation in this article may be a bit more lighthearted than usual, this week’s topic is about something I actually take very seriously: Providing mentorship during the scientific process. In fact, it comes straight off of my list of “Things I’m Pretty Sure I’m Sure About,” a concept I cribbed from one of my all-time heroes, chef Alton Brown.
Even before taking on my role as MAISRC’s Staff Quantitative Ecologist, in which I will be mentoring MAISRC fellows on all things “quant,” I had already spent more than a decade mentoring graduate and undergraduate researchers (formally and informally), much of that time while a graduate student myself.
What I learned throughout that time is that one fruitful way to guide research mentees through the scientific process is by asking them a series of key questions at key times during the research project and then working with them to find satisfying answers to those questions so their projects could move forward.
However, it wasn’t until very recently, while I was serving as an Assistant Professor at Drew University, that I stumbled upon what has seemed to me, at least, to be the most valuable such question to pose to your research mentees. It is but one question, out of so many you could challenge your mentees with, and yet it yields so many benefits! It’s also a question I have never myself been asked during any of my own research experiences as a mentee. Plus, when I ask my peers about their experiences, they say the same thing–they weren’t asked it either. And yet, it’s a question I really wish I had been asked early and often! And when I have posed this question to those I have mentored, I’ve found it to be uniquely clarifying, for both me and my mentees.
So, what’s this magical, all-important question?? Patience–we’ll get there.
The Essential Questions to Ask Your Mentees Early On During a Research Project
First, a preface. I chose my wording here–calling my question the most valuable question–intentionally! I want to distinguish here between valuable questions (ones that aren’t strictly essential but sure are useful!) and essential ones (those that are so fundamental to doing good science that I don’t really see how a successful research project will come about without them).
My MVQ (Most Valuable Question) is one designed to be asked very early on in a research project, all the way back during the study design phase. That’s a phase in which a lot of essential questions are often asked too, and my MVQ should follow those questions to be its most effective.
So, let’s take this opportunity to briefly recap the (I would argue, essential) questions that often get asked during the study design phase of a research project and that would tee up my MVQ to yield max benefits. These are (more or less) the questions I myself now ask my mentees to answer as they are envisioning their project in (more or less) the order I ask them. However, I would argue they are probably the questions any researcher should be asking themselves (and their colleagues!) at this stage in a research project! I’ve abstracted them in wording here, but you can of course “specifize” them as you see fit.
- What makes you curious? (Side-note: Here is one of my scientific mantras–If there is a single essential resource that fuels good research [besides coffee], it’s curiosity! Science can be downright tedious and frustrating and bewildering sometimes–being so curious about something that you still want the answer even after having to fight through such a grueling process to get it is a powerful thing! That’s why, bar none, I start research conversations with mentees in a place of curiosity rather than anything else. Call me a romantic if you must!)
- It is often valuable (though not essential) to have your mentees expand at this stage on why this phenomenon or subject makes them curious. It helps you get to know them and their motivations better so you can better guide them and advocate on their behalf later! It can also be useful to ask them to list many things that make them curious and then help them narrow down to the one thing that makes them most curious.
- What is your (scientific) question about this curious thing?
- At this stage, it is often valuable to work with your mentee, once they have drafted their question, to determine whether their question is likely to have a defensible (if not strictly objective) answer, is sufficiently grounded in what we do already know, hasn’t already been satisfactorily answered, and is scientifically “interesting” enough to warrant further investigation (**cough** funding agencies will take enough interest **cough cough**). However, the essential questions that follow this one will often suss out whether the answers to these questions are “yes” either way…
- How do you suspect this curious thing tends to play out, generally speaking? (AKA one’s “general prediction,” or, as I prefer to think about it, the “pattern” that results from the “process.” What would one tend to observe in the world with respect to this curious thing, if they went out and grabbed some typical, relevant data? Don’t fret–we get at so-called “specific” predictions later.)
- Why do you suspect this curious thing works that way? (AKA one’s hypothesis—an explanation for why the world works as it does. Often, it proves needlessly difficult, for me anyway, to disentangle the “process” conceptually from the “pattern” it yields, so I don’t! Instead, I usually pair this question and the last one together.)
- It is often valuable here to have your mentees consider what other hypotheses (could) exist and why they nonetheless believe their “preferred” hypothesis is the most promising one. Plus, I like to ask whether they really even need to pick a “favorite” or if they can instead pit the various hypotheses against each other more neutrally. At this stage, I usually direct my students to hit the literature (again) to immerse themselves in what is already known and thought.
- What scenario could you concoct to see if your explanation is correct? (AKA one’s test—a [possibly contrived] way of varying (or using variance in) the factor(s) of the world that one believes control the outcomes we observe, in the hopes of observing whether the outcomes we get are the outcomes we expected).
- At this stage, it’s actually perfectly ok for this scenario to be a rough sketch or a broad outline. Most tests are to start with!
- A public service announcement here, while we’re on the subject: While all experiments are tests, not all tests are experiments. If a change in X really does cause a predictable, direct change in Y, we might design an experiment to find this out–a style of test that forces X to take on various values, even potentially ones that are unrealistic or impossible otherwise.
- However, we might also simply observe many “natural” values of X and see if the corresponding Ys exhibit the pattern we’d expect–this would be called an observational study, and while it can’t (easily and firmly) establish causality, it’d still be a perfectly reasonable test. In fact, I would argue that observational studies have some key advantages over experiments and have become much maligned for bad reasons, but that’s a subject for a different post…
- No matter what, though, a scientific test needs to try to explain variation in outcomes (our Ys above) via variation in conditions (our Xs above). So, at this stage, one should at a minimum be able to articulate what condition(s) they intend to vary, roughly how, and roughly by how much to get their expected range of outcomes.
- If you’re right about how the world works, what do you expect to observe during your test? (AKA your test-specific prediction–the result(s) you expect that would support your hypothesis).
Now, I’ll be the very first person to concede I’ve invented zero wheels here–this general set of questions in this general order is what I suspect most research mentors ask their mentees as they guide them through the design phase of a research project (although I will toot my own horn to say I think I’ve laid them out quite nicely here!).
Where I have something potentially powerful–and dare I say even novel–to add to the conversation is with the next question I ask…
My Most Valuable Question to Ask Research Mentees (or Just About Anyone Doing Research)
Without further ado, it’s at this point that I ask all my research mentees my Most Valuable Question. Drum roll please…
- If you’re right about how the world works, and thus your results ultimately will support your hypothesis, what is the **graph** you will show me that will demonstrate this? (Ok, no prejudices here–maybe it could sometimes be a table or a picture or a diagram, but the idea is the same.)
First off: Did anyone, in your mentorship sphere, ever ask you this question? I wasn’t asked it. That’s no value judgment on any of my mentors, to whom I owe an immense debt! They gave me a lot of things; they just didn’t give me this question. I found this question only as a mentor myself, struggling to “demystify” the otherwise mysterious process of conducting good research. “What did I wish I had been asked?” I reflected on this question a lot, and one of the “prizes” I received for doing so was this question!
In my experience, unless they were mentored by me(!), no one I’ve mentored yet had already been asked this question either. From what I can tell, this question just isn’t kicking around in the current research ecosystem very much. However, I believe it absolutely should be! And I’m not basing this claim on just my intuition or values–it has been my experience in hard practice that this question proves immensely valuable. How so? Valuable question! I’m glad you asked…
Value #1: It Puts One’s Questions, Hypotheses, and Predictions to the Test (Pun Intended).
The principal goal of the research design phase is to ensure we craft the most robust project we can–one with a vital and intriguing question, thought-provoking hypotheses, perhaps a novel test, objectively measurable predictions, etc. No single question I’ve found so far does as much as my MVQ does to probe whether a budding research project is ready to bloom!
Let me recap some of the various issues my MVQ has showed the power to uncover, even in just my own narrow experience with it.
First off–can they (your mentee) even visualize a rough version of their graph at all? If not, this could signal that their prediction needs work. After all, a prediction (as I tend to think of one, anyhow) is little more than an idealized graph in words–“I’m expecting this group’s bar to be low and that group’s bar to be high and…” If they can’t picture what the graph would look like, even vaguely, that suggests to me that they don’t know what they’re expecting to see. Alternatively, if the prediction isn’t the issue, maybe the test is–it’s hard to know what you expect to see if you’re still fuzzy on what you’re planning to actually do!
Second: Can they kinda sorta articulate what their graph might look like, but when they go to actually draw it, it doesn’t go so well? This could mean a few things, actually. First, it could just mean they aren’t all that familiar with typical graph types yet. I’ve found that I can usually do a decent primer on bar charts (or, my preference, boxplots!), scatterplots, mosaic plots, line graphs, and histograms (the “Big 5”) in about 15 minutes [maybe a topic for another post?!]. Often, that clears things up, and we can soon proceed on our merry way!
Second, they might just not be fluent enough in data/statistics to be comfortable “thinking graphically” yet. For example, they may have a sense of how their test will work and the general outcomes it may yield but not how the data will look once they are in a computer or what a computer would do with the data once one hits the “graph” button in Excel.
If that’s the case, as it often is with undergraduate mentees, I usually have them talk their predictions out and I draw a mockup of their graph instead (or I talk them through drawing it). We then walk through the graph together to make sure it’s conveying what they had in mind. While not quite as ideal as if the mentee had to do the work of drawing out the graph themselves, this situation is still super valuable–we’ll get all the other benefits of my MVQ below, we’ll be able to proceed with our project, and, maybe most importantly, I know what kind/level of training and development my mentee needs in this area.
Because, after all, the hard truth is that some data/stats literacy is a prerequisite to asking good scientific questions and getting good answers at this point. Consider how hard it would be to convey a key result entirely and only in words, and only qualitatively, and have it be widely received and appreciated! A lack of data/stats literacy doesn’t threaten a project–such a lack that goes unaddressed might!
Third: A more nuanced problem I see a lot at this stage is that the mentee can basically draw the graph but, when it comes to labeling the axes (and especially putting concrete units on them!), that’s when the struggles begin. An example here is illuminating: I once had an immensely talented mentee put “Fruit quality” on the Y axis of their graph at this stage. I said, “Ok, but how are you going to measure that objectively? What units would you use?” Think about the conundrum here: Is a bigger fruit necessarily “higher-quality?” To whom? What about a sugarier fruit? We collectively banged our heads against these questions for 30 minutes before realizing, no, “quality” isn’t objectively measurable, and it’s not actually what we are trying to measure anyway!
I’m not picking on my student here–this is just a really good example! They say “the devil is in the details.” In the case of research, specifically, I’ve found that the devil is often in the axes labels, tick-marks, and units. If you think about it, what underlies these simple lines, words, and marks on a graph is only the entire Methods section of your project, from the levels of variation you’ll generate and observe to the tools you’ll use and their precisions to the empirical reasoning behind any proxies, thresholds, or indexes you’re using, etc. What better way to detect major “plot holes” in the eventual Methods of your study than to try graphing the results you’re hoping they will yield??
A fourth possibility, and one I regularly have encountered with gung-ho graduate mentees in particular, is that they want to draw me ten graphs, not one. Or, when prompted to “No, really, draw me just your one graph,” they start to draw one graph with 4 axes, 7 colors, 5 symbols, 4 sub-axes on the main axis, etc.
Now, here, I feel I must pause to address the 800 lb elephant in the room that I’ve no doubt let in. When I revealed my MVQ earlier, I daresay some of you, dear readers, may have instinctively objected “But some projects produce many graphs!!” Graduate theses are a great example–if an entire thesis were somehow expected to yield just one graph, you’re right, I too would be concerned! At this point, you might additionally be thinking “But graphs from projects are sometimes very complicated!” Let me validate you, if I’m reading your mind here well enough: Yes, and yes. Both of those things are true! No arguments there from me. But, at this stage in a research project, these questions may to some extent miss the point, and they may even be a “symptom” of a larger issue that, by asking it, my MVQ may help to address.
Let me (try to) explain myself. Yes, most projects create more results than could reasonably fit in a single graph. However, in some cases, the issue is just that we’re expecting to produce many specific graphs that are variations on the same general graph, as when we expect to plot a single Y against three important Xs. In these cases, we can usefully revise my MVQ to: “What is the exemplary graph you will show me…”. We can always discuss how the other graphs might deviate from the first, but having at least the one graph to frame the discussion at the design stage is what’s key!
Also, yes, sometimes projects yield many graphs, but, in some cases, this is because we are just being a tad imprecise about how we are defining a “project.” Let’s paint with an enormously broad brush for a second: To the extent a “project” is defined as a combination of one (alternative) hypothesis, one test, and one prediction–and thus one “answer” to one “question”–it should generally yield just the one graph. If we can’t limit ourselves to one (or at worst just a few) graphs to convey the results we expect to see from our one “project,” perhaps we haven’t refined our “project” down far enough; perhaps it’s more unfocused than it could be!
By this definition, I would say a graduate thesis/dissertation is often many projects tied together by a unifying question or narrative (i.e., at least one project, but perhaps more, per chapter!). So, you may be having this graph discussion with your graduate mentee a great many times rather than once! But always about just one graph at a time.
Another reason one might be tempted to draw many graphs instead of just one at this stage is that we are not arranging our predictions hierarchically enough. There are often hypotheses and predictions we have that are deemed essential to include but aren’t actually of primary interest. “Ok, ok, we know nitrogen is probably going to matter to Y, so we have to test for, measure, and maybe exclude that effect, but what we really want to know is how X matters…”
In the above scenario, could we ultimately end up producing a graph between nitrogen and Y? Sure. But is that the graph we really need at this stage to know if our project is on track? Probably not! More times than not, there is going to be “One Graph To (Conceptually) Rule Them All,” the result we’re most excited about and most focused on–that’s the one I want to see the graph for!
And if the temptation is to draw me a really complicated graph rather than many graphs, it’s still often for much the same reason–our more ancillary predictions are comingling with our focal one. We can then think about the graph we want to see at this stage as being the one that shows us what is happening when all other, less important factors are behaving as we’d expect them to.
…Or, sometimes, graphs are complicated because tests are complicated, and sometimes tests are complicated because questions/hypotheses/predictions are complicated, and sometimes those are complicated because we actually aren’t as sure of our line of inquiry as we could be. The goal, aspirational as it may sometimes be, is to craft as elegantly simple a test as we can. A (relatively) simple test should then yield a (relatively) simple graph. If one isn’t seeing a simple graph emerging from one’s test, it might suggest we need to rethink–and simplify–the test! If we can’t see any other way to rethink the test, this suggests we may need to go even further back in the design process until we figure out where all this tortuous complexity is coming from!
That last point encapsulates Value #1 of my MVQ, I think: The research design process, like research itself, is iterative. We rarely get things perfect on the first try! While one might be out there, I’ve found no single question that kick-starts the iterative refinement process faster nor better than my MVQ, especially at the design stage.
Value #2: It Puts One’s Data (and the Analysis Thereof) at the Heart of the Design Process.
I’ve said this before, but I’ll say it again: Data are not byproducts of research; they are the entire point of it. What do we do research for? I would argue it is to yield new, meaningful data that we can extract new, meaningful knowledge from.
By forcing us to acknowledge, imagine, and “pre-summarize” the data we hope to go out and collect during our test, my MVQ puts the data we want to acquire at the very heart of the design process, where it should be! We sometimes view the process of research as being largely over once we have data in hand–the things that happen after the test is cleaned up is not really research, we think, but some “post-research process.”
No! Once we have data, the research process isn’t over–it’s reaching its climax! We’re about to learn our “answer,” after all this lead-up! Just as a novelist should be considering how their story will reach peak action before they ever put pen to paper, we researchers should be anticipating the climax of our work–the process of extracting knowledge from hard-earned data–from the very beginning!
I’ll go so far as to say here that the #1 mistake I suspect researchers (especially early-career ones) make in their research is that they mentally partition these two critical phases of the research process: Data collection and data analysis. “First, I’ll go out and gather my data. Once I have them, then I’ll decide what I’m going to do with them!” This is a bad approach. This assumes that, whatever messes we make within our data set, “the stats will bail us out.” That’d be like a surgeon saying “No matter how much collateral damage we do, we’ll just fix it in post-op.” Please don’t go to that hospital.
Bottom line: We want to have as clear a sense of what data we want, what we expect them to look like, and (most pivotally) what we plan to do with them to address our hypotheses as we can get as early in the process as we can get them. Guess what: A graph necessarily suggests a data structure! Do you have grouped bar graphs in mind? Congrats–this suggests you have categorical X data (multiple layers of them, actually) and continuous Y data. This suggests you’re going to probably do a Multi-Way ANOVA of some variety. Are you picturing a line graph? Congrats–this means you have some non-independence in your X data (maybe you’re going to be doing a repeated-measures test), and if so, you’ll likely be doing some kind of Mixed-Effects model. Do you have an ordination plot in your future? Congrats–you’re going to be doing a lot of crying…
Point is: Show me your graph and I can tell you what kinds of data you’re going to want to collect. I can then tell you what kinds of statistical analyses you will probably be doing, and what strengths and limitations those tests have. For example, if you’re going to be doing a Mixed-Effects model, we’re going to want to ensure that, for each group being repeatedly sampled, we have at least 6 replicable units (don’t ask–this is just an example).
To the extent that we see our research as our livelihoods, to not be considering our data, their structure, and their eventual analysis from the outset of a project is to be playing dice on the job, IMHO. And it worsens an already big problem: A lot of researchers don’t enjoy statistics very much. From my vantage point, it’s little wonder why–they bring mistreated, frustrating data sets to the doorstep of statistics and then are, not surprisingly, frustrated by how frustrating the statistics are! I hope you can see the place where we can most easily break this cycle–draw me your graph, and I can help make sure you get data you won’t thoroughly hate analyzing!
Value #3: It Frames the Entire Project in the Lens of Communication from the Outset.
Earlier, I said “Data are the entire point of research.” That statement is true. Would I lie? Don’t answer that.
But it’s also false! There’s a second, even more “entire-point-y” entire point of research: Communicating what we have learned. In my current role within MAISRC, a charge of mine is to help our researchers communicate their results. If we came to know all there is to know about Invasive Species X–we’ve done all the tests, ran all the stats, and produced all the raw knowledge–but we never communicated this new knowledge to invasive species managers, policymakers, and the general public, all our efforts would have yielded very little, if any, progress, right?
Yeah, knowing more is fun! But guess what: We can’t (probably) take what we know with us when we die. I don’t think. I don’t really know. Don’t argue with me about this in the comments…
What I’m trying to say is that our research only yields a lasting impact once it’s been meaningfully communicated. And, while we often communicate the results of our research in words, it is also commonplace for graphs to play a huge role in that process! After all, a picture is worth a thousand words, as they say! A simple, digestible, and compelling graph is the indivisible unit of good research–it’s a kernel of raw, soy-free knowledge. Who among us doesn’t dream of just being able to flash a figure up on screen during a presentation, drop the mic, and walk away (presumably, to head to one’s Nobel Prize acceptance ceremony)? Let’s sketch out that amazing graph together from the very start of your project!
Now is a good time to address another elephant I’ve let in our room: Imposter syndrome. I will confess to having suffered from horrible imposter syndrome as a graduate student, and then again as a post-doc, and then again as a professor. I still feel it rear its ugly head sometimes! So, I know first-hand that students, post-docs, and early-career faculty members can easily develop an impulse of “I need to ensure other people think I’m smart, so they don’t think I don’t belong here.”
One unconstructive way to combat that feeling is by introducing layers of complication into our research. “If this research project is such an impenetrable thicket that even I can barely follow the thread, surely no one else will be able to either, and then no one will be able to dispute whether I belong or not!!” We might think we’ve won when we do this, but we’ve all lost, and no one has lost more than you!
We, as researchers, should be trying to gather new knowledge and then trying to effectively spread that knowledge to as many people as we can. Complex, indigestible research–presented impenetrably–might assuage our imposter syndrome and egos, but it will do little to advance our field. Show me your graph and I can tell you everything I don’t get about it. Hopefully, I’ll understand a lot! But, if not, at least we’ve got a long time to ponder together how to bring more people along for the ride.
Besides, which tends to impress YOU more: A smart person making a complex thing seem impossibly complex, or a smart person making a complex thing seem so simple, so mesmerizingly accessible, you couldn’t believe you hadn’t thought of it that way before?? Be the latter kind of smart person: Spend the time trying to distill as much knowledge as you can into one, simple, elegant graph.
A last note on this subject: If you like things pretty, a graph will certainly be prettier if you’ve been imagining it in your mind long before you ever actually sit down to make it. Just saying!
Value #4: It Provides Clarity to What Can Often be a Winding Journey.
You know how I said earlier that research can be confusing, tedious, boring, and frustrating (sometimes all at once)? Yeah, you’re welcome that I’m reminding you how hard the thing you do for a living is again. The thank-you letters are in the mail, I’m sure…
Well, you know what can make that exhausting research journey a little more bearable? When you know–with incredible precision–where your destination is and what it looks like (hint: It’s when you are communicating your graph to your audience). Research may sometimes feel like it’s never done. But a project is surely done (or very nearly so) when you’ve got your graph and you’ve shared it. For this renewed clarity, you’re welcome!
You know what also helps to make that journey a bit more tolerable? When it’s clear how you are supposed to reach your destination. Have you figured out how you will get the different X values on your graph? No? Better work on your test design today! Don’t have all the Y data yet? Guess it must be a day to go collect more data! Don’t know what to label the Y axis quite yet? Might want to read a bit more literature to polish up your hypotheses. Don’t know how to draw the error bars? Time to wrestle with the plot controls in R some more!
To the extent that your graph represents your final destination, having your graph clarifies your path: Once you have mocked up your graph, all you have to do after that is everything it takes to be able to draw your graph. “Oh, is that all, Alex??” I hear you all laughing manically. Ok, I’m not saying that this knowledge makes the process easy nor brief, by any means, but it certainly helps!
It’s also useful to remind yourself that if something doesn’t get you any tangibly closer to drawing your graph, it might be a diversion. Especially with student mentees, who haven’t gone through the rigamarole of the research process dozens of times, it can be really hard for them to stay on task because it’s pretty unclear what the “task” even is sometimes. That was surely my experience! They are used to clear direction and deadlines. But research doesn’t always afford these, and not every PI has the bandwidth to be constantly giving them. However, if you have your target graph in mind, direction and deadlines become easier to see, articulate, and quantify progress towards. You have to admit, “How much closer to drawing and communicating your real graph are you?” is a more concrete question to address than “How much closer to defending are you?”
But my MVQ yields one more benefit of this kind. If you are anything like me (and, in this and in many other regards, you shouldn’t be!), you tend to not write anything for a project until the project is “over.” That is, writing is the last thing you do, as though it too is a process separate from research, one that is only supposed to start when “collection” and “analysis” are over. I wrote all 5 chapters of my PhD Dissertation in my last 6 months. True story.
Don’t be like me: Start writing sooner. If you have questions, hypotheses, and predictions–which you should, if you’re past the research design stage–you can probably write at least the tail-end of your Introduction. However, if you also have your target graph, you can probably begin writing sizable chunks of your Methods as well. Heck, if your draft of your graph is really well-developed, you may be able to build in placeholder text, figures/figure captions, and tables into your Results section before you’ve even gathered an ounce of data!
Believe me, I know: Nothing is scarier than a blank Word document when you know, one day, it’ll need to be a full Word document. Once you have your graph draft, though, you have little excuse for leaving that Word document blank, and that’s the kick in the bum we all need sometimes!
There you have it–the single most valuable question, in my experience so far, to ask a research mentee (or yourself, for that matter). TL:DR: Show me your graph, and I can tell you an awful lot about how the rest of your project should go! May this question serve you as well as it has served me! What do you think of the case I’ve made here? Do you think I missed any benefits? Or perhaps I’ve overblown some, in your opinion? Do you have your own most valuable question (MVQ) you’ve been dying to share? Let me know in the comments!
Alex Bajcz, Quantitative Ecologist 