If you are going to claim to be an excellent educator, you need to evaluate your educational practice. This might be for a promotion or fellowship application, a conference presentation or even a peer reviewed article. However, most educators I know have had little-to-no training into how to evaluate their educational practice. This was certainly the case for me. As a research scientist with a background in microscopy I had no relevant methodologies to adapt to evaluation of teaching. Here I reflect on some of the mistakes I made early on in my educational evaluation career. Hopefully you can learn from them and save yourself wasted effort and pain!

Mistake 1: Not being specific enough about what you want to evaluate and why

Imagine you have redesigned a whole module and its assessments, and want to find out the impact of this change. That sounds great, but you are likely to be doing too much, resulting in a muddled evaluation plan. I have certainly done this, and have seen improvements to module outcomes, but could not pin-point which component of the module was responsible because my evaluation was too generic and everything was interdependent.

Start with some focus. What is it specifically about your module design that you want to evaluate? You probably want to focus on the most innovative component. Is it a particular assessment model? Or a new way of managing group work? I would focus on the aspect of the pedagogy that most reflects your characteristic approach to education.

I would also think about how distinctive that pedagogy is – particularly if you want to publish you need to focus on things that haven’t been done hundreds of times before. I’ve sat through dozens of conference talks which were effectively ‘I’ve just discovered active learning – I introduced some interactive polling into my lectures. Oh look – students enjoyed it, and grades improved but we also changed the assessment at the same time so I don’t know what caused that change’. I don’t want to discourage anyone starting out from evaluating and sharing their practice, but you are likely to generate higher impact if you focus on something more distinctive and specific.

Mistake 2: Not reading the literature before you start out

I certainly made the mistake of going straight into designing an evaluation strategy without looking at the existing literature at all. How naive and arrogant I was! As a scientist I would never have designed an experiment without reading around to find out (i) where the gaps in knowledge were and (ii) what the most appropriate methods were. Why did I think I could do this in an educational context?

There is a rich body of pedagogical research literature out there, which should be your starting point for any study. What has already been done? What are the gaps in our understanding? What methods do people use? What are appropriate sample sizes? These are all key questions you need to consider in order to design a high quality evaluation.

A word of warning – educational research literature can be pretty challenging to read if you don’t come from a closely related discipline. As an empirical scientist I found it really tough (and still do to some extent). Many of the high profile Higher Education journals are very theory driven, which can be difficult to relate to practice. To bridge the gap when starting out I would start your reading in disciplinary practice focussed journals, e.g. if you are a chemist I would start by searching through Chemistry Education Research and Practice. Find the studies which are the closest match to what you are planning to do. Identify the key features of their research design, and how your study would add to this literature rather than just replicate it.

Mistake 3: Not including any underlying theory

Really high quality educational research will probably include a strong theoretical and conceptual framework. Have I already lost you? When you are starting out this language and way of thinking can be a bit baffling, and I’m still working out how to effectively use theory in my research (which deserves a whole blog post in its own right).

In planning your first evaluation I wouldn’t get too hung up on rigorous theoretical underpinnings, but you do want to link your work to some sort of theory. Without this your evaluation will be pretty narrow in scope and interest. For example, an evaluation describing a particular pre-workshop task on a 2nd year nursing module is of limited interest to anyone not working in that context. However, if you evaluate it through the lens of cognitive load theory you allow people from unrelated areas to see the underlying relevance, and therefore make the connections to their (non-nursing) contexts. It took me a long time to understand that theory was a tool to take you from specific example to something that could be generalised and applied in other areas, and I wish I’d understood that much earlier.

Mistake 4: Not thinking about ethics alongside your research design

I trained as a plant biologist, partly so I didn’t have to consider ethical issues in my research! For me, moving into educational evaluation meant thinking about research ethics properly for the first time, and again I was very naive about using data collected from human participants. So naive I didn’t even realise that obtaining ethical permission was required! Looking back at my early studies I think I was surprisingly thoughtful about issues of e.g participant information and consent given I’d had no training in this area. I used the British Educational Research Association (BERA) ethics guide, but I am mortified how little formal oversight my early work had.

It is essential that you design your research methods with ethics in mind. This is obviously the right thing to do, but if you have any intention of publishing then journals will insist on seeing evidence of appropriate ethical oversight in your paper, typically including a named ethics committee and project code.

My main advice when starting out is to get to know someone on your ethics committee who can help you out and give advice. If your department doesn’t have a relevant ethics committee, or doesn’t usually consider ethics applications involving human research participants you may need to find an alternative in another department (or equivalent). Talk through what you are planning to do with someone who reviews or writes lots of ethics applications, and get them to help identify the issues. For example, a common research design for educational evaluation is a pre- and post-intervention questionnaire. To maximise the power of this study design you ideally want to be able to track individuals, ie compare pre- and post- responses for each student. However, this design directly undermines a plan to collect data anonymously. An experienced colleague or ethics officer will be able to make practical suggestions for how to track individuals without collecting identifying information such as student emails or id numbers.

Mistake 5: Using un-tested methodology

When I was a scientist, I would design research in one of two ways. If you wanted to evaluate a new method, you used a well established biological system to test your method. If you wanted to explore a new aspect of a biological system, you used a well established method. Using a new method on an unknown biological system was a recipe for poor research, as nothing could be validated.

The same is true in educational evaluation! However I certainly made the mistake of trying to evaluate a new teaching strategy with a new and un-tested method – bad idea all round! I see this particularly with questionnaire based studies. It is easy to think that it is straightforward to design an evaluation questionnaire, but robust survey design is *really* hard. The wording of questions needs to be so precise, the number of questions about each aspect needs to be optimised, the internal consistency of responses needs to be sufficient – the list goes on. I have basically stopped doing questionnaire based research because it is so challenging.

To make it easier I would strongly recommend finding a questionnaire or survey instrument that has already been published and validated a a reliable instrument. Ideally you would use the exact same questions, or make only minor adaptations to e.g. reflect your own institutional terminology. If you can’t find an existing survey, then you definitely need to run a pilot so you can identify issues with wording and test internal reliability first. Don’t do what I’ve done (more than once) and release a survey, collect hundreds of responses and then realise there is an issue with the questionnaire and have to throw that data away.

Mistake 6: Doing your evaluation as an afterthought

For a lot of educators, the focus is on getting the teaching done. I speak to so many people who are using interesting pedagogies, but when I ask them if they have evaluated the impact of that pedagogy they have little to say. As I opened with, if you are going to say you are an excellent educator you need evidence to demonstrate it. This means you need to plan an evaluation strategy, and ideally do so *before* you make your intervention. Doing your teaching and then thinking ‘ohh I should have evaluated that’ either means you have missed opportunities to demonstrate your educational impact, or leads to rushed retrospective evaluation strategies that are unlikely to be publishable or make much wider impact.

For example, if you want to demonstrate the impact of a particular pedagogy you typically need some sort of baseline to compare to. This could either be a previous cohort, or could be a pre/post design. In either case, you need to have started your evaluation before making the intervention. Relying only on retrospective reporting opens your evaluation up to all sorts of bias. You also need to think about how many cohorts you need to include – a journal is unlikely to be interested in a single cohort study. You may need to include multiple years, or recruit people from other departments or institutions to join in your study to increase its robustness. All of this takes time to plan, including time to get ethical approvals (see above). However if you can do this in advance you will be able to design a much more robust assessment of impact.

Mistake 7: Not planning your data analysis in advance

You’ve done 15 one hour interviews – great! Now what on earth are you going to do with that qualitative data?!

I’m not going to write a primer on qualitative analysis here, in large part because I still don’t really know what I’m doing with qual research! However it is important to realise that qual research needs to be rigorous and well planned, just as much as quant. Just cherrypicking quotes that sound good is not qualitative analysis. Most people start out with some sort of thematic analysis, for which I recommend Braun and Clarkes excellent book Thematic Analysis: A Practical Guide. However there is a whole ecosystem of different qual analysis methods, which may or may not be appropriate for your data.

Whatever your research background, I strongly recommend collaborating with someone trained in relevant methods rather than trying to teach yourself and getting it horribly wrong. If you have no qual training, find a colleague who is experienced in analysing this type of data. Equally, if your statistics training is pretty limited, collaborate with someone who understands what analyses are appropriate for surveys or other quantitative datasets. Ideally collaborate with this person *before* collecting your data so they can help inform your research design!

Conclusions

This has turned out to be a longer blog post than intended, but this perhaps reflects the number of mistakes I’ve made along the way! Starting out in educational research and evaluation is hard, particularly if you don’t come from a related academic discipline. My main advice is to collaborate with, or get advice from more experienced colleagues who can support you along the way. With support you can design robust evaluation strategies – please avoid some of the pitfalls by learning from my mistakes!

Links to useful resources

Grove and Overton (2013) Getting Started with Pedagogic Research within STEM Disciplines

British Educational Research Association (2024) Ethical Guidelines for Educational Research, fifth edition