Module: Introduction to assessment

Welcome to the professional learning module introducing assessment. We hope you find it helpful and informative, and send us any feedback you might have. For the full list of the modules and explanation of how they work, please see here.

This course will outline some basic assessment theory and will be the first module of a number looking at assessment. During the course we will ask questions to guide and frame your thinking. At the end of the course we will ask you to complete a short written task, so you many want to take notes as you progress through the course. You can do this using our virtual notepad here, which also gives you the option of receiving feedback on your notes.

1.      What are some of the purposes of assessment?

There are many reasons we assess. Throughout the many threads in the #cogscisci email group we have seen the range of these reasons become apparent. Some teachers assess in order to identify students for specific interventions and some teachers assess in order to (try to) predict the spread of attainment when their students finish GCSEs. These are all valuable reasons but unless teachers are clear on their reasoning, the assessment design they produce might not be suited for it. Before we start, here are some questions to get you thinking:

  • Think back to last week. Where in your practice did you assess? Try to be as specific as you can.
  • For each of the examples you remember, what was your purpose behind the assessment activity?

Read this blog by Evidence Based Education. Here are some questions to frame your thinking about the blog:

  • How far do you agree with the statement that “no assessment is 100% accurate”? 
  • Does it make sense to say the answer to the assessment “What is your favourite colour?” has some inaccuracy?
  • Focus on one set of assessments done at your school (you might want to focus on January mocks for example): what purpose do these assessments serve to people in various roles in the school, MAT lead? Headteacher? Assistant Head for Data? Head of Department? Fellow colleagues? Yourself?
  • What does the word “construct” mean in the psychological literature? 
  • Think ahead to a point in time where you will formally assess. What is the construct you’d like to assess? Do you think this construct is useful to you? Is it useful to your fellow colleagues? 

Read the paper by Archer. Here are some questions to frame your thinking about the paper:

  • Do you agree with Archer’s three basic purposes for assessment? Why? Why not?
  • Archer (along with Paul Newton at the top of page 2) argues that classifying assessment along the lines of summative/formative “leaves something to be desired”. How far do you agree with Archer? How useful (or not) has this distinction been so far in your career?
  • Archer then lists Newton’s and Schildkamp and Kuiper’s interrogation of assessment purposes. Is this framework more useful to you? 
  • In the section “Assessment to Support Learning”, what dangers does the triangle metaphor imply? 

2.      What is meant by “validity”?

Validity is often referred to as the central concept of assessment. The history of validity thought stretches back to the birth of large scale testing 120 years ago. Because validity is such a large and often misunderstood topic, each reading is paired up with an accessible introduction. Read section 2 of this blog to get a history of validity thought as well as this blog by Evidence Based Education to clarify some of the ideas.

Whilst reading, here are some questions to frame your thinking:

  • Why doesn’t it make sense to talk about validity as a property of the test?
  • What are some construct-irrelevant factors when your students sit a science mock?
  • What does construct under-representation mean?
  • Think back to the last formal assessment that you administered. Was there any construct under-representation? 
  • Think back to the last formal assessment that you administered. Was there anything that introduced some construct-irrelevant variance? 
  • Summarise, from the two blogs, a definition for validity that you could give to a trainee.

Sometimes a concept is made clearer when you’re told what not to do! This article introduces a paper by Crooks, Kane and Cohen that explores some of the threats behind validity claims. After you have read the article, read the 8 page section titled “Threats Associated with Each Link” from the bottom of page page 269 to the top of page 279. The questions that follow will help guide your reading of the paper.

  • Why is a chain used as a metaphor to hold the approach of Crooks, Kane and Cohen together?
  • Read the paragraph that starts off with “The importance of the eight links…” on page 269. Could you come up with another threat for any of the links before you read the rest of the paper?
  • Do you agree with Crooks, Kane and Cohen that “for a classroom-based assessment intended solely for diagnostic and formative purposes, the aggregation, generalization and extrapolation links may be somewhat less important than other links” (page 269)? 
  • Think back to the last formal assessment that you administered. Was there a weak link in the chain? What stage was the weakest?

3.      What is meant by “reliability”?

Reliability is probably best thought about as “freedom from uncertainty”. If we have highly reliable assessments, we are likely to get the same results if we apply that assessment to the same students under similar conditions. Whilst validity can be quite discursive, reliability can get quite technical fairly fast. 

One of the biggest mistakes newcomers can make is that they confuse the term “reliability” with “dependability”. Reliability is a technical term with the restricted meaning mentioned above, however, “dependability” is a judgement made on the quality as well as the certainty we get from an assessment. 

This blog by Evidence Based Education is a good introduction to reliability. Here are some questions to guide your thinking:

  •  Reading the “sources of error” section from the blog, can you think back to your own practice and identified real situations where some of the sources of error were present?
  • What is the difference between inter-rater and intra-rater reliability?
  • What assessment scenarios have you been part of in school that involved an attention to inter-rater reliability? Intra-rater reliability?
  • How often have you implemented the suggestions in the section “improving rater reliability”?

The Institute of Education ran a study on how reliably teachers assess in classrooms. Read pages 4, 5, 6 and 7 starting with “In-depth review” two thirds of the way down page 4. Here are some questions to guide your thinking:):

  • What could the main conclusions from the first six bullet points be??
  • How do these conclusions differ from your own personal experience? 
  • From the next section reporting the validity of teacher judgement, what could be the main conclusions from the report?
  • What were the main findings from the conditions that affected the reliability and validity of teacher summative judgement?
  • How can teacher judgement be made more reliable using the findings from this report?
  • Did any of the findings surprise you? Which ones and why?

4. Bringing it together

We don’t think this is the be-all-and-end-all of assessment. Assessment is much more than what we have presented. It’s the systematic enterprise of finding what our students know and so there is much we haven’t covered. We haven’t covered motivation, ethics, comparability, the estimation of reliability, modern test theories … the list goes on. 

In future modules, we will look at how to apply some of this thinking to the classroom. Adam Boxer has brought together these principles in his blog “What to do after a mock?”. Here are some questions to guide your thinking:

  • How has Adam used the concept of validity to guide his practice?
  • To what extent have your question-level analyses run into the problems Adam describes?
  • What are your top 3 takeaways from this blog?

Our last activity is a youtube video of Dylan Wiliam’s webinar “There is no such thing as a valid test”. We hope that by re-covering some of the ground from sessions 1 to 7, you have a clearer picture of what good assessment is. 

5. End of module task

As with all modules, there is a task at the end. This task should serve not only to consolidate the work you have done throughout the module, but is also a way for other teachers to see how you have applied and used the things you have learnt. 

Please choose a task from the below and send it to cogscisci@gmail.com:

  • Write a short article about what you have learned throughout this module about assessment. You may want to focus on a few key ideas.
  • Prepare a keyword glossary of key assessment terms for your colleagues based on this module.
  • Write a reflective article about your previous practice, where you may have changed your mind in light of what you have learned, and what you intend to do in the future. 

We would love to publish anything you produce, but will obviously not do so without your permission. If you are happy for us to publish it on CogSciSci please let us know in the email. 

If you have any feedback for us, you can submit it anonymously here.

Module: How to write SLOP

Welcome to the professional learning module on writing SLOP. We hope you find it helpful and informative, and send us any feedback you might have. For the full list of the modules and explanation of how they work, please see here.

This module has been reviewed by Nick Soderstrom.

“It is virtually impossible to become proficient at any mental task without extended practice”

Daniel Willingham

This course is about SLOP: Shed Loads of Practice. It will cover the role of practice in learning and how to prepare resources that are conducive to your students conducting extensive amounts of practice. The beginning of the module is theory focussed, but as the module goes on there will be a number of activities to help you start writing your own resources. We have prepared a Google form here to help you structure your notes and thoughts throughout the unit. 

1. The role of practice

This section will consider a brief introduction to practice in the classroom, and ask you to conduct an audit of the amount of practice your students receive.

Start by reading Rosenshine’s Principles of Instruction. If you have read it before, please do so again. Focus on sections 4, 5, 2 and 9 in that order. Reading the rest is worthwhile, but those sections will suffice for this module.

  • What are the different types of practice that Rosenshine describes?
  • Your explanation sets the ground for practice. How do the best teachers prepare their students for practice?
  • Why is it important to monitor student practice?

Next, read Adam’s blog on how he uses booklets.

  • What are the different types of practice identified?
  • How much practice is provided for students?
  • How is the practice structured to support students?

Audit – how much practice do your students receive?

  • In general, how much practice do your students receive in lessons?
  • What percentage of the time over a 5 lesson sequence is spent with your students independently practising content?
  • After teaching new material, how many practice questions do your students complete independently?
  • How often do you revisit past material as a part of student independent practice? (i.e. not in something like a retrieval starter)
  • People tend to appreciate the need for practice in physical activity (like sport) more than in purely cognitive activity. Why do you think this is? How could you convince them of the need for more practice?

2. Overlearning

You can find an introduction to learning and performance by Soderstrom and Bjork here. From it, read the “introduction” paragraph which looks at the difference between learning and performance. Pay particular attention to the mention of overlearning. Next, read the middle paragraph on page 7, then read the paragraph at the bottom of page 9.

  • What is the difference between learning and performance?
  • What is overlearning?
  • What does “performance [was]…at asymptote” mean?
  • What are the ramifications for AfL of the distinction between learning and performance?

3. Drill and thrill

Students completing extensive, individual and independent practice has been out of vogue for a little while, and can be touted as boring or uninteresting. First, read Joe Kirby’s blog about the importance of drill here.

  • What justifications does Kirby give for student drill?
  • In the football video, what is the purpose of the drill?
  • How is the drill arranged so that all players participate?
  • Is the drill monotonous and boring?
  • Consider Lemov’s six rules. Which one do you think you already do the most, and which one the least?

Read Dani Quinn’s blog here. Though it is maths focussed, there is much there for us to learn.

  • How do Quinn’s justifications for drill relate to Kirby’s?
  • Look carefully at how the problem sets increase in complexity. In science, which topics/concepts are conducive to this kind of practice?
  • Quinn mentions a number of pitfalls and occasions where they “don’t work.” Which items in her list do you think are most pertinent to science teaching?
  • How does Quinn justify her claims that drilling can be joyful?

4. Spacing and interleaving

In the retrieval practice module, we discuss spacing as the act of spreading out learning opportunities on a given topic over time, and we saw it has many cognitive benefits. Interleaving is about how topics are mixed together during practice. Watch this video by Robert Bjork which introduces interleaving.

  • What is interleaving?
  • Without knowledge of the art example, would you expect the sport example to be generalisable to more concept-based areas?
  • To what extent is the art example generalisable to the classroom?

Read the first four pages of this study by Rohrer et al.

  • How do the researchers define “interleaved” practice?
  • What do they suggest as the main mechanism by which it improves learning?
  • How does this contrast with previous reading about overlearning?

Read the “caveats” section from on page 10.

  • Which caveats do you think are the most important for generalising to the classroom?
  • A small amount of blocked practice may be necessary first. How does this relate to previous reading about overlearning?
  • Feedback is clearly crucial on tasks of this kind. How can you ensure that your students receive rapid and precise feedback?

Read the last page of the study. It notes that this is a highly feasible intervention, but relies on the availability of good resources. Researchers also note that this is an effective intervention, but is not “flashy.”

  • What other possible obstacles to implementation are there?
  • Do you have resources which feature interleaving?
  • How have your beliefs about teaching and viable interventions changed based on literature such as this?

5. Guidance fading

The next reading is a chapter from Sweller et. al.’s Cognitive Load Theory. Set aside some time, as it is packed full of useful information and pointers and should frame much of your activities when designing practice. You can skip page 6-11.

Key terms to help you understand the chapter:

  • Guidance is how much support is provided
  • Worked examples are a form of high-guidance
  • The expertise reversal effect is where as students become more expert, they require less guidance.
  • Why is it important to tailor the amount of support students receive?
  • What is a completion task?
  • What is the guidance fading effect?
  • What is the difference between backward and forward fading?
  • Which is superior, backward or forward fading?
  • On page 5, what is the effect of prior knowledge on the optimal rate of guidance fading?
  • The chapter focusses on worked examples/completion tasks as a route to fading. What other routes are available to teachers in terms of the support they offer and how they fade it?

Activity:

Download the quantitative chemistry mastery booklet below, and look at page 5.

  • How has guidance been faded across this problem set?
  • Why have some questions had the answers provided with them?
  • Is there enough practice in the problem set?

6. Summary so far

At this point we pause to summarise what we have found so far:

  • Instruction begins with an explanation based on a small amount of material
  • Students receive a worked example or model
  • Students receive practice with faded support like a completion task
  • Students conduct massed independent practice of the new concept
  • Independent practice becomes more interleaved

7. Procedural and declarative: practising procedural knowledge

Procedural knowledge relates to knowledge of how to do something, like balance an equation or change the subject of a formula. Declarative knowledge relates more to knowledge of concepts or ideas, like that mass is conserved in chemical reactions, or how the tilt of the Earth”s axis causes seasons or how a root hair cell is adapted. You cannot write practice for taught material without this distinction in mind.

Procedural knowledge tends to be much easier to write practice for, but still requires skill and careful thought given our summary steps above. Tom’s exemplary blog here looks at both, but pay particular attention to his thought processes when structuring procedural knowledge.

  • Why is it important to distinguish between procedural and declarative knowledge?
  • How does Tom build complexity and fade support in his worksheets?

Look at questions 38-80 of the booklet on B1 below.

  • How are the problems sequenced?
  • In what way is complexity gradually added?
  • How is interleaving incorporated by the end of the question set?

Look again at the quantitative chemistry mastery booklet you downloaded above, and consider how all the principles already mentioned have been built in.

Look at the year 7 worksheet below:

  • In question 2, how is variation introduced into the problem set?
  • What is the effect of this variation?
  • In questions 7-11 how are topics mixed together?
  • What is the purpose of question 12 and what knowledge/skill does it build?
  • How long do you think it would take for an average year 9 student to complete the worksheet?
  • Overall, do you think there is enough practice on this worksheet?

Activity:

Choose a topic you are due to be teaching with features heavy procedural elements. Construct a worsheet to accompany your instruction featuring:

  • Worked example
  • Completion problems (or another route to fading support)
  • Extensive massed practice
  • Followed by interleaved practice

8. Procedural and declarative: practising declarative knowledge

Declarative knowledge is much harder to write practice for than procedural. Read this blog by Ruth on “SLOP-resistant” topics, and this blog by Adam on similar.

  • Why is it harder to practice declarative knowledge?
  • What simple techniques can you use to practice it?

Replacing worked examples with models:

For declarative knowledge a model answer can be a powerful discussion tool. Look at the worksheet below:

  • How would you use this worksheet in class?
  • How would you monitor the quality of students’ work as they begin to practice?
  • What other topics might benefit from an approach like this?

Other examples of practice work on declarative knowledge:

In each of the worksheets, pay careful attention to the way in which the questions are repeated and altered slightly to force students to think. Remember that the key point is that the students have to select the appropriate solving strategy from memory. In particular, look at how different topics are introduced and interleaved throughout the questions and the careful phasing of massed and interleaved practice.

Activity:

Choose a topic you are due to teach which is heavy in declarative content. Prepare a section of a booklet for that content featuring:

  • Model example
  • Faded practice
  • Independent massed practice
  • Interleaved practice

9. Final activity

In order to obtain your certificate and finish this module, you will need to apply what you have learned in order to prepare a resource. The resource should be used to accompany a significant amount of learning, so not just half an hour in one lesson, more like at least half a unit. It should account for all of the ideas above, starting with examples/models, moving on to guided practice, slowly fading that guidance until students begin independent massed practice then move on to independent interleaved practice.

Please email your resource to cogscisci@gmail.com. We will obviously not publish it without your permission, but if you would be willing for it to be published then let us know in your email and we will happily add it to this site’s resources section. Please also be aware that though we have a high standard for publication, our dedicated resources team commit to helping you improve your work to the point at which we can publish it, so if you want specific feedback do just let us know.

If you have any feedback for us, you can submit it anonymously here.

Module: Retrieval Practice

Welcome to the professional learning module on retrieval practice. We hope you find it helpful and informative, and send us any feedback you might have. For the full list of the modules and explanation of how they work, please see here.

This module has been reviewed by Pooja Agarwal

This course will outline the history of retrieval practice, the theory behind it and its application in the classroom. At the end of the course we will ask you to complete a short written task, so you may want to take notes as you progress through the course. You can use this optional virtual notepad to help you engage with the texts.

  1. What is retrieval practice?

Read this study by cognitive scientists Roediger and Karpicke. Questions which you can use to frame your thinking about the study:

  • Why did the researchers complete two different experiments? What conclusions could the second experiment offer that the first one could not?
  • What is an “educationally relevant” context, and why is it important to conduct studies on “educationally relevant” materials?
  • Participants in the majority “study” conditions outperformed participants in the majority “test” conditions in the short term, but not in the long term. What ramifications does that have in the classroom?
  • Is student cramming effective? If it is effective in the short term, how can we convince students that it is not in the long term?
  • In the conclusion, the researchers cite studies from the Bjorks about long and short term difficulties. Summarise those findings and explain how they relate to retrieval practice.

2. How do students practise?

Most students do not tend to use effective learning strategies. Read the introduction to this study on the topic by Dunlosky et al. Also read the closing remarks from page 44 until the end. It is recommended that you study the section on retrieval practice in detail, and choose two others to study similarly. 

Questions which you can use to frame your thinking about the study:

  • Out of the list of 10 strategies, which ones do you think your students use the most outside of class?
  • Out of the list of 10 strategies, which ones do you think you use most in your day to day practice?
  • Draw a sketch graph with effect along the x-axis and ease of student implementation on the y-axis. The “goldilocks zone” would be high effect and high ease. Map each of the ten strategies on your graph.
  • Which ones are in the teacher’s control? How could teachers change what they do in order to make high effect strategies easier to use? (section 7 below will provide more ideas on this)
  • Out of the two strategies you chose to read further, what surprised you the most?
  • Under which, if any, circumstances is that strategy most effective?
  • Should teachers ever use strategies that we know are less effective?

3. Spacing

Research shows that spacing retrieval over time increases its power. Efrat Furst has written a description of the evidence behind this and potential mechanisms to explain its effect here, and you should start by reading that. Read this blog by teacher Damian Benney looking at how he researched how large the spacing gaps in-between sessions of retrieval should be. Questions which you can use to frame your thinking about the blog:

  • To what extent are teachers in control of the size of the retrieval gap?
  • What are the advantages of using a routinized system like Benney’s to map all instances of retrieval across a year?
  • What are the disadvantages of the above?
  • “Lagging” homeworks and assessments are vital in implementing spacing. Which opportunities could you take to better embed spacing in your teaching?

4. Feedback

How should teachers give feedback after retrieval practice? Read the section relating to feedback in this large research summary by Roediger and Karpicke. Questions which you can use to frame your thinking about the blog:

  • For what kinds of task is feedback most appropriate?
  • In the absence of feedback, can learning still occur?
  • What are the dangers of not giving feedback?
  • When should feedback take place?

5. Anxiety

Exams are high-stakes environments that can impair our students’ performance. This study designed an experiment to establish whether or not retrieval practice protected participants from the impairments of a high-stakes assessment. Questions which you can use to frame your thinking about the blog:

  • Outline the design of the experiment and the four groups within the study
  • How was stress induced in the participants?
  • To what extent is that stress similar to the stress involved in high-stakes assessments?
  • How could these findings affect your practice as well as your recommendations to students?

6. Transfer

Transfer is the “holy grail” of education: giving students the ability to apply their knowledge to unfamiliar situations. Read Steven Pan and Pooja Agarwal’s booklet on the topic. Questions which you can use to frame your thinking about the blog:

  • Give an example of near transfer in the science classroom. In the past, how have you prepared students for such situations?
  • Give an example of far transfer in the science classroom. In the past, how have you prepared students for such situations?
  • How can retrieval practice be used to promote transfer?
  • Which approaches are less effective for promoting transfer?
  • What modifications can you make to your practice in order to better prepare your students for transfer?

7. In the classroom

Download a copy of the retrieval roulette for your specialism. See also this list of resources of teachers applying retrieval in their classrooms. At the bottom of the page are a number of resources you can use to promote and increase the amount of retrieval practice your students engage in.

Round-up and takehomes

  • Retrieval practice is the most effective method to increase your students’ ability to use information in future
  • Retrieval opportunities should be spread out over time
  • Students should always receive specific and precise feedback following retrieval
  • There are a number of ways to embed retrieval practice into your lessons, and there are many resources available to support you with this
  • Encouraging your students to do retrieval practice is not enough. You must provide them with clear and simple routes to doing so

8. End of module task

As with all modules, there is a task at the end. This task should serve not only to consolidate the work you have done throughout the module, but is also a way for other teachers to see how you have applied and used the things you have learnt. We would like to publish your task when you are finished, but if you would rather that we didn’t then please just let us know when you submit it.

Please choose a task from the below and send it to cogscisci@gmail.com:

  • Retrieval overview: write a blog or poster to be used by other teachers to quickly learn more about retrieval practice
  • Student resource: write a document aimed at students and parents to promote retrieval practice by the suggestion of some simple resources and routines
  • Teacher resource: design a resource for teachers to use when implementing retrieval in their classroom
  • What I’ve done: write a blog article about your experiences of retrieval practice and how you have implemented it in your classroom
  • Pushing it further: what kinds of topics have you used retrieval practice for? In which circumstances have you had more or less relative success? Write a blog outlining your findings.

If you have any feedback for the CogSciSci team on this course or anything else on the site, please use this form to submit it anonymously.

Create your website with WordPress.com
Get started