Lessons Learned Using Oral Reading Fluency from Curriculum Based Measurement as a Practical Driver Measure
In a previous article we shared our insight that the ideas and measures developed in Curriculum-Based Measurement (CBM) can be applied to continuous quality improvement in education.
The lessons learned that we are sharing here come from our work developing and implementing the CBM measure Oral Reading Fluency (ORF) with the Baltimore Secondary Literacy Improvement Community (BSLIC), a networked improvement community that worked on secondary literacy. This case study discusses this work in more detail.
Many of these lessons may seem to be quite obvious. They certainly did to us in retrospect as we were reflecting and writing. At the time however these lessons were not yet learned and thus were not so obvious.
A meta-learning, if you will, for us from developing and implementing ORF as a practical driver measure has been that the disciplined inquiry and cycles of learning that are hallmarks of improvement can be directly applied to the development of measures and the measurement system of improvement surrounding them.
Practical measures may at some point be able to be used in a general off-the-shelf manner. This is perhaps a worthy goal for us to strive toward as it may help improvement become more accessible to a wider audience in education.
However we have learned that measures like ORF, with obvious practical attributes, become more practical and thus more powerful for improvement when we turn the improvement sensibility and tools on them and our practice in designing a measurement system.
Again, sounds painfully obvious in the writing but assuredly is less so in real-world practice.
While a measure may be practical in theory that does not make it so in practice
Candidate practical measures should be subjected to their own improvement learning cycles.
ORF was initially pilot tested with a small number of improvement fellows and a small number of students. Only through safe-to-fail tests under real world conditions is it possible to determine if a measure is practical in practice and to improve its practicalness. These small tests provided important learnings for fielding ORF at scale.
We were able to discover the range of administration times that could be expected in these small tests. We learned that ORF could be collected virtually but was fragile to technological issues.
Perhaps most importantly, we learned that there were benefits beyond just data collection of ORF being administered by fellows versus other support staff. Administering ORF gave fellows direct knowledge of each students’ fluency by hearing them read aloud which enriched the fellows’ pedagogical and improvement work.
The cadence of practical measurement should match the cadence of learning cycles in the improvement work.
ORF can theoretically be collected on a weekly cadence however it would not be practical to do so given the time costs of administering to more than one student.
In BSLIC learning cycles were organized into multi-month action periods that were composed of two week blocks of fellow improvement work and group reflective huddles.
For this operational cadence a monthly collection of ORF provided sufficient data for improvement progress monitoring, feedback to fellows and was practical to collect.
Practical measures are often considered ‘noisy.’
There can be significant variation in practical measures like ORF. In traditional research this is often considered to be ‘noise’ to be controlled or removed.
This variation is important and should not be ignored. Attending to variation is a core principle of the science of improvement.
In practice, though, it can be a difficult lesson to learn as it is counter to the traditional inclination to see variation as a problem. This is especially true with substantial or unusual variation which can occur with practical measures.
Student ORF performances can vary greatly across a school year. Sometimes a student’s reading rate will rise excitingly and other times it will fall unexpectedly. One interpretation of this variation is that the student’s true reading rate is being obscured by ‘noise.’ In this framing, the noise is seen as a nuisance to be removed if possible or, more likely, modeled away as statistical uncertainty.
When using ORF as a practical measure of improvement, an alternative framing is possible which reverses the traditional inclination to label unexplained variation as noise. Under this interpretation, all variation is considered an important source of information to be explored further.
Let’s consider an example. Despite the work of researchers to normalize the difficulty of passages used in ORF, there can be unexpected variation, both positive and negative, that may be due to some passages being easier or harder for some students.
Naturally improvement teams should try to reduce the variation by changing out poor performing passages over time. Importantly though, improvement teams should also investigate why some passages are harder or easier for students. Learning from these opportunities can provide valuable new insights on passage elements that are particularly challenging for some students. Digging into this variation could lead to a better understanding of another facet of the difficulties that students might face on reading tasks both on standardized assessments and in authentic situations. This is an important learning that could easily be thrown away as noise.