Welcome to the latest article in our 'View from the Inside' series as we sit down with our Assessment Officer and all round question writing extraordinaire, Phil Wall.
On a day to day basis, you will find Phil managing, populating, updating and refining our question banks to provide the best possible assessment for our learners. He also supports the production of exam papers; liaises with subject experts; analyses item, paper and qualification performance; and handles assessment-related enquiries from our centres. All in all, he's a busy chap.
What are the recurring challenges you encounter when writing questions, and how do you address those challenges?
Some of the people that we commission questions from are extremely experienced in their fields, and they have a lot to bring to the table in terms of the different styles of question writing. But in order to make things consistent, we have to encourage the use of a particular style. For instance, we tend to use gender-neutral pronouns to make our questions as inclusive as possible. It’s something that’s rectified quite easily through feedback from the question writers, talking to them and coming to an agreement about what can be changed.
Also, when you’re a professional writing a question, it can be tempting to include as much information as possible in a question to give as much context so that the learner can give an informed answer. But that isn’t necessarily how you write a great multiple-choice question.
Coming up with great distracters is also really challenging for a lot of people and I can see where they come from. They know the right answer, but creating a quality distracter is an art in itself. They have to be enticing enough that a learner could genuinely choose it as the answer, and coming up with that takes a lot of consideration and imagination.
Why are there always three distracters?
...I’m not sure!
There’s probably some scientific answer that says having four options to choose from is an optimal way to write a question, but I couldn’t tell you for sure.
I suppose having any less than four answers would increase the likelihood of the learner just guessing correctly, whereas more than four can be too overwhelming. Three distractors is a nice balance.
What have you learned from your interactions with question writers so far?
I’ve learned a lot from reading all their questions. Before I started here, I had no idea you had to be 16 years old to dye your hair!
To you, what are the building blocks of a really good question?
There are three C’s in a good question: clear, concise and consistent.
When I say clear, I mean that it should be very obvious what the question is testing. If there is any distracting information in the question, that can get in the way.
Concise, in that, you should use as few words as possible in order to make the intention of the question clear.
And consistent, in the way you write the question should use consistent language.
Have you ever come across an assessment criterion that you found really challenging to create a question for?
So many of our sectors have blown my mind. There are so many factors that can go into writing certain questions: regulatory requirements, particular areas of practice, the lot. It was really new to me when working in vocational education. There’s so much to learn and know.
What is the value in putting a question to the learners in a situational context?
That’s a really good question.
The advantage of the situational/scenario-based question is that not everyone learns in the same way. Some people learn in a more kinaesthetic way, by doing and acting and speaking, so it might allow them to imagine what they would do in that situation. If they can imagine what that situation would be like, they may be able to base that on their own personal experiences.
On the other hand, you do have people who struggle to visualise things. Sometimes we get feedback on some questions where people say they could never imagine being in the particular situation described in the question. Ideally, we create questions that can extend across areas of practice. So, for instance, whether you work in a book store or a supermarket, if you’re doing a customer service qualification, you still have the experience of dealing with a customer, and we try to write towards that.
What makes a really bad question?
Superfluous information. Double negatives. Negatives in the answers and distracters as well. Lack of clarity.
Is it different writing questions for regulated qualifications than for End-Point Assessment?
A little bit. Questions for End-Point Assessment are very specialised, and so it’s vital to consult heavily with subject experts. To a certain extent, with some regulated qualifications, the knowledge and experience you need to write questions isn’t quite as extensive, so it’s easier to just start writing.
You used to be a teacher. Does that make it easier to be a question writer?
I think it gives me a certain advantage. I’m used to assessment methods and how you measure those things. I’m very used to the idea of every piece of knowledge being tested has to apply to a very specific criterion. While the answers are a bit more subjective in my old job (I used to teach English), there’s still something to measure.
So, with English, it may be split into reading and writing. When you’re assessing reading, you might measure the learner’s ability to understand the context of the work, the intent behind the words, general understanding. That’s all measured within a framework.
What’s one thing about question writing that most people wouldn’t know?
It’s a lot more scientific than most people think it is. There is a formula for a good question. We tend to see questions as open-ended, but we make them very measured and scientific.
You can measure the performance of answers if you have the data and the processes. For example, you can take the top ten percent of learners and the bottom ten percent and measure their performance in certain questions, and you can effectively assess whether particular questions are well made or not. Say you’ve got four answers, and if everyone got it right – the top and the bottom – it’s not a very good answer, is it? Because the lowest ten percent, the ones that haven’t revised, were still able to get it right. So, it’s not a good question.
What’s your one question writing pet peeve?
A really lengthy question that ends with a full stop. and the prompt for the question is something like, ‘Do they?’ Tag questions are usually unnecessary if you write the question well to begin with.
Also, ‘all of the above’ or ‘none of the above’. Those grind my gears.
Is there a gold standard for question writing that you try to write your questions against?
I went to a question writing conference in London led by Peter Bennett. He put together a pretty comprehensive guide of do’s and don’ts for question writing, and that’s the standard I hold myself to now. Whenever I’m writing a question or updating a question bank, I think ‘What would Peter do?’
What’s your favourite assessment activity?
Essays, probably. But maybe I’m just nostalgic for my university days.
To keep up to date with the latest news and tips from TQUK, follow us on Twitter, Facebook, Instagram and LinkedIn.