The inaugural webinar for our Results Matter: Closing the Achievement Gap webinar series was led by Results Driven Accountability (RDA) expert Dr. Alan Coulter. During “Results Driven Accountability: Where Were We? Where Are We? Where Do We Go Next?” Dr. Coulter provided an overview of the Individuals with Disabilities Education Act (IDEA) and its predecessors and the current approaches to ensuring accountability. After the webinar, our co-founder and co-CEO Clay Whitehead and Dr. Coulter answered questions from district leaders, educators, and clinicians. Below is a summary of their discussion.
Clay: Thank you for making time to be here. The first question is very simple.Where do we begin with RDA from a tactical point of view?
Dr. Coulter: I think it depends on if you are at the district level, school level, or if you are an individual provider of service. That is going to change how you think about Results Driven Accountability. However, you always begin with the data. For example, we talked about the data analysis that went on to establish the State Identified Measurable Result (SIMR). If people look at the SIMR for their state and it doesn’t match their own particular needs in terms of what they’re trying to achieve with students, then they need to select a result to use as a goal. This goal will act as somewhat of a “North Star” as they think about how they’re providing services.
Now, while we’ve had a considerable emphasis throughout this webinar on the word “result,” and that does not in any way depreciate or minimize the importance of using evidence-based practices to get there. Therefore, where should you begin? You begin with data that reflects a measurable result.
Clay: What data do we need to be using to set up our students for success?
Dr. Coulter: Let’s say you’ve selected the data that is going to be your measurable result. You then also need to come up with at least one measure that will show progress. In our business, we really have to select measures that are reliable and valid. Professionals typically use a mixture of data, as well as professional judgement, but there isn’t room for measures that are unreliable. We need measures that produce the same metric regardless of who is doing the administration. Once you have a reliable measure, you want to ensure it’s going to be a valid indication that things are progressing. For example, a speech-language pathologist is going to want to look at production and vocalizations, etc., while other individuals may want to look at literacy, social goals and more.
There are really two types of data you need to collect. You need a reliable measure of a result and you need a reliable measure that tells you whether you’re making the necessary progress to meet the result you want.
Clay: If I’m a therapist and there’s all this emphasis being placed on result-based achievement, how can I translate this into actual real life learning and success?
Dr. Coulter: That’s a great question because the most important thing to remember about Results Driven Accountability is that you shouldn’t try to go at it alone. For instance, I remember when there was a big push around literacy and I was so proud of the American Speech-Language and Hearing Association (ASHA) when they came out advocating for literacy. In other words, all the people that serve children, not just teachers or families, should be working toward Results-Driven Accountability.
If you’re a therapist working toward Results Drive Accountability and results-based achievement, use your national association as a resource and reach out to establish relationships with colleagues. Make connections with people who can work with you on the students you are serving so that everybody knows your goal. Find out what their goals are and collaborate to reach those goals.
Results Driven Accountability can be a very lonely venture if you’re not, in fact, reaching out to colleagues and getting their support. I also think that people need to remember that, from a national level, Results Driven Accountability is always about the large numbers. It’s this concept of what I call “altitude” for therapists, school psychologists, or people providing any kind of services to children. Result Driven Accountability is only going to succeed if we remember that classrooms and schools are really the meaningful unit of change. That’s why every individual listening to this webinar needs to know that they’re the ones who make Results Driven Accountability work. The people who look at these numbers at the district, state, and federal levels, they’re dependent upon teachers, therapists, and so on, to do the results-driven work using evidence-based practices that will compile into something that we’ll all be proud of.
Clay: What is the best evidence-based practice for measuring results? Also, how do states’ measures of accountability compare?
Dr. Coulter: Just to clarify, when we say “evidence-based practice for measuring results,” we are referring to progress monitoring practices. Because as we mentioned earlier, this progress monitoring is how you ensure some sense of whether or not things are working.
On the resource handout, we list five different clearinghouses where you can find evidence-based practices. There’s the Pew Charitable Trust’s Results First Clearinghouse Database, Johns Hopkins University’s Best Evidence Encyclopedia, the What Works Clearinghouse, the Intervention Central database, and the Substance Abuse and Mental Health Services Administration’s (SAMHSA)’s Guide to Evidence-Based Practices. You can also look at the recommendations made by ASHA and the National Association of School Psychologists (NASP). They all constantly try to put evidence-based practices in front of people.
If you follow the evidence-based practices listed in these databases the way they are intended with the set of students they are designed for, you can have some assurance that you will get the intended results because you followed the implementation with fidelity.
Let’s shift now to the next question and this is a tricky one. How can we compare states in terms of their measures of accountability?
Typically, people truncate that question by only thinking about the achievement measures that a state is using. We know from the National Center on Educational Outcomes, Education Week, and a number of other sources that it is very difficult to compare student achievement state-by-state. There are still so many different measures of student achievement and we have variances in reliability, the way data is collected, and participation rates of students, especially students with disabilities, in measures like the National Assessment of Educational Progress. It’s almost impossible to get an equitable measure of student achievement.
Comparing accountability state-by-state is actually not the best way to think about it. For example, the Office of Special Education Programs (OSEP) does not just look at achievement; they look at a number of other factors. My recommendation for comparing measures of accountability is to refer to OSEP’s website for their rating of your state’s accountability system. This will show OSEP’s evaluation of how well your state did on the implementation of the Individuals with Disabilities Education Act. This is probably the most equitable way to compare accountability across all states.
When you do that, remember that this is just a state viewpoint. It’s not a viewpoint about a singular district, school, or student. It is a very high-altitude assessment of accountability.
Clay: What is the best way to show student progress?
Dr. Coulter: The most important thing to think about is how you are going to depict your measures. There’s a whole science around how you show your measures and progress. We are much faster at interpreting data when we can see it graphically. For example, progress over time is best shown in a line graph.
Clay: What strategies are most effective in closing the achievement gap for students with special needs?
Dr. Coulter: If we are talking about an achievement gap in terms of literacy or math, we know a number of evidence-based practices we can use. However, I believe we need to pay more attention to the intensity and duration of this intervention. When we are making decisions about how long and how often students need intervention, we should be basing this off of the students’ needs, not on administrative conclusions. If we don’t consider students’ needs, we’re not going to close the achievement gap for most of the students we serve. We have to get better about making decisions based on individual needs.
Clay: We have time for one last question. Research currently trashes remediation as a way to close the achievement gap. Can you tell us what’s going on there?
Dr. Coulter: “Remediation” is a colloquial term and it doesn’t really have a reliable operational definition. I think we’re much better if we don’t use the term “remediation.”
When we talk about interventions, I think we can see that there are interventions that definitely make a measurable difference. There are other popular interventions that, while they might be easy to do or they might be something that we’ve used for a long time, science has proven to be ineffective. We need to be very careful about how we define our terms, and we have to go back to the science to see which interventions have produced results because these are the practices we need to keep replicating. We need to stop using practices that are not producing the science-based effect and start using those that will.
To watch Dr. Coulter’s entire 90-minute webinar and download the accompanying slides and resource handouts, click here. To register for Dr. Kimberly Gibbons and Dr. Mary Morningstar’s upcoming “Results Matter” webinars, click here and here respectively.
