This CNN Business blog post is full of so much nonsense.

“How to prepare children for the jobs of the future”

Specific examples of foolishness:

“65% of the children entering primary school in 2017 will have jobs that do not yet exist and for which their education will fail to prepare them”
I’ve seen this dopey statistic repeated in many places. At least the author cites a report in the blog post, but the article cited just makes the claim without rationale or references to research. The report implies that this claim is related to automation of some jobs but “some estimates have put the risk of automation as high as half of current jobs, other research forecasts indicate a risk at a considerably lower value of 9% of today’s occupations.” Hmm. An estimate of 9-50% is… quite a range. The author also guarantees that current education will “fail” to prepare students for these future unnamed mysterious jobs. This ability to peer into the future is impressive!

Here’s a series of other claims without rationale or evidence from the blog post:

• About a toy that teaches coding: “It’s a skill that you can apply to anything: you basically learn to think in a very logical and rational manner.” Coding is a great skill and it’s a great idea to get students involved in it. But it’s NOT some magic “all-purpose” reasoning skill that you can apply to any context. Critical thinking is context dependent: students learn to think like mathematicians, historians, artists, citizens, etc., and these are different skill sets.
• Wowza. Please count the buzzwords in this claim: “Kano offers paradigm game-changing opportunities for teaching computer science…(It) also enables project based learning opportunities to extend collaboration, creativity, communication and critical thinking skills.”
• “Danish companies… have been working together to introduce sensors into the classroom. They developed… a small white box that monitors noise, temperature, air particles and CO2 levels. Data is then fed to a smartphone app, so that a teacher or facility manager can monitor the environment and make sure it is as comfortable and productive an environment as possible.” I’m not sure what to say about this one. If CO2 levels get too high, does the teacher ask students to stop exhaling?

Have any of you heard claims about current education not preparing students to “most” future jobs? If anyone reading this knows any research that this “65%” claim is based on please let me know. I can’t imagine how anyone would do research that would help establish this prediction? Any thoughts?

(note: I spotted this article in Dylan Wiliam’s twitter feed: https://twitter.com/dylanwiliam/status/1157307029431771136)

Last day! And it was really a half day: one breakout session first thing in the morning and then the final keynote speaker.

Those of you who know me won’t be surprised that I filled my half day with Dylan Wiliam 🙂 The morning breakout session was a Q&A. Each table generated one question to ask him and he used the whole 1.5 hours answering these 5-10 questions.

I asked him two questions:

Question #1: Let’s say a teacher wants to change ONE teaching habit this year (knowing that habit change is difficult and takes a long time to do), what one habit related to formative assessment would you advise that teacher to change? D. Wiliam’s Answer: [pause…pause…] wait time. [laughter from the crowd]. He explained that changing wait time is hard to do and potentially very useful. He clarified something I’ve never heard before about wait time: we think of the wait time that occurs after we ask a question (he called this “thinking time”), but there’s also wait time that occurs after a student answers (he called this “elaboration time”). He recommended that we focus on this second aspect of wait time. When a student responds, wait before you indicate “correctness.” Have that student keep thinking. Get other students to comment on that answer. Get everyone thinking about that answer (deep processing!). He cited a study by M. Rowe “Slowing down is a way of Speeding up” that I need to find and read.

Question #2: For this question, I actually… lied a bit. Yep, I lied to Dylan Wiliam. Please don’t tell him. I told him that I was a former high school psychology teacher (true) and that I was working with a group of other psychology teachers (partial lie – I’m working with a bunch of former students of mine who now work as teacher coaches, district office admin, etc. ) to develop professional develop stuff based on the 3-box memory model – I asked if he knew of other work like this and if using this model seemed like a good idea (note: I “lied” b/c I didn’t want to spend the group’s time listening to me explaining how this group got together, what our roles are, etc.) D. Wiliam’s Answer: he said he thought it was a good idea to use the 3 box model and that the tricky decisions will be how far to go into the details. He cited a potential point of disagreement between Bjork (desirable difficulties) and Sweller (cognitive load theory). He advised us to make a “clean” version of the model and use that as a place to hang concepts and research on to (and that’s our plan!). I got the idea to ask Wiliam this question because of a slide he used very briefly during one of his presentations:

It’s great to get some “affirmation” from D. Wiliam that our plan to base a chunk of professional development on the 3 box memory model might be a great idea! (I bet I’ll write about these plans on this blog in the future)

D. Wiliam’s Keynote: the last event of the conference was a keynote address called “Negotiating the Essential Tensions.” It was good, but… too ambitious? Wiliam tried to cover a LOT of ground, shared a LOT of research, and I took a LOT of notes. I think I wish he would have focused on a few issues and gone deeper. But: he’s Dylan Freaking Wiliam, and it was a fascinating, enagaging presentation!

One aspect of the keynote that will stick with me: his cautions about jumping to “sweeping” (and standardized) answers about grading. He used some theory, analysis, and research to eloquently review some of the very complex issues involved in grading. Highlights:

• NO grading system is perfect, which means that every system involves trade offs. If we’re not clear about what those trade offs are and which ones we’re choosing, we’re not making informed choices about our grading system.
• Educators have been worried about the validity of grades for a LONG time (and Dressel 1957 ).
• There are important interaction effects between feedback and grading (Butler 1988)
• Everything is contextual: there are realistic scenarios when feedback might be more useful if it violates some of the “rules” we learn about feedback: sometimes feedback might be more useful if it is LESS specific, LESS timely, and LESS constructive. Context = important. The purpose of feedback is to help students improve performance on a task they haven’t done yet, not the task they just completed. He says Hattie doesn’t make this point, and he should (there’s a Kluger and Denisi meta-analysis that makes this point, but I haven’t found it yet).
• The old phrase “You don’t fatten the pig by weighing it.” is … wrong. Retrieval practice research indicates that students learn from tests. Frequent, low or no stakes testing is a really good idea (and might be one of the most powerful things we can do). Roediger and Karpicke 2006 and Bangert-Drowns, Kulik, and Kulik 1991

Whew! It was a great conference, I’m glad I went, and I’m walking away with a bunch of ideas. Thanks to my district for supporting my travel, and thanks for reading all this!

I continue to be a lucky fellow! Day 2 of the conference started with a keynote from Dylan Wiliam, one of my education research/writer heroes. He used his keynote to highlight and expand on the themes he wrote about in Creating the Schools our Children Need

If you haven’t read it yet, get to it! One of my favorite education books. In the book and his keynote talk, Wiliam walked us through research about why many current large scale education reforms (like hiring “smarter” teachers, firing “bad” teachers, funding voucher/charter schools, copying education systems of other countries, and offering more differentiated/personalized instruction) aren’t effective. The rest of the book and talk was devoted to what might work: investing in current teachers, promoting knowledge-rich curricula, and implementing effective formative assessment processes. Wiliam acknowledges that this is tough work (it involves changing habits, which is always tough) and argues (with evidence at every point) that teachers need choice, flexibility, chances to take small steps, and supportive accountability as they figure out how to integrate “short cycle” formative assessment habits in their teaching. If you ever get a chance to hear Dylan Wiliam present, do it! The keynote was filled with data, analysis, and clear thinking.

The rest of the morning and part of the afternoon were devoted to breakout sessions. I got to hear Jay McTighe talk about designing performance assessment tasks. He includes multiple resources on his website and he’s worked with many (many!) districts. I liked hearing about his practical experiences and he provides multiple intriguing examples of performance assessments. One thought/concern: I didn’t hear much during the presentation about research on performance assessments as an intervention, and I’d like to hear more. I think some of the research about how effective different kinds of performance assessments are would help refine the idea quite a bit and help teachers/administrators figure out how to use the idea most effectively.

And a specific/related concern: there is a “debate” between folks who say they advocate “direct instruction” and “discovery/problem based/inquiry based/project based/etc.” learning. I know some teachers who identify as “project based” teachers. During his talk, McTighe talked about “authentic and inauthentic” tasks. I don’t think McTighe thinks about it this way, but I worry that some teachers/administrators see information about authentic tasks, performance, assessment, DoK level 3 and 4 tasks, etc. and they wrongly assume that these kinds of assessments and activities are “better” or more important than other kinds of activities and assessments. This is a generalization, but I think this “debate” is a false dichotomy: we shouldn’t be thinking about direct instruction or problem based learning, etc. Underneath it all, the reality of learning is that for every learning task, students need to be able to retrieve some skills and information from their long term memory and get it into their working memory in order to do the cognitive task we asked them to do. If those skills or information aren’t in their long term memory or they can’t retrieve them, then we need to help them with that, and that probably means some direct instruction. Despite the title, I think this article by Kirschner, Sweller, and Clark shows convincing evidence that we need to be thoughtful about what skills and knowledge students need before they try to tackle important “inquiry based” tasks.

Students absolutely need chances to use skills and knowledge in order to answer important questions, and problem based learning etc. activities can be great ways to do that. But it’s not an either or. It’s silly to talk about direct instruction or problem based learning being better or more important. I would like to hear J. McTighe talk about this underlying learning reality more.

Then I went to hear S. Brookhart talk about “Comprehensive and Balanced Assessment Systems,” a paper that was developed based on last year’s LDi Formative Assessment conference.

This paper is worth reading: Sue led us on a group discussion activity that challenged us to think comprehensively about our district’s assessment system. We examined how comprehensive (what is measures at what levels where) and balanced (what data are returned from what assessments appropriate for what purposes) our systems are and identify gaps. I think this activity would have been more effective if Sue would have been able to model the analysis/discussion on an example, but it was a good discussion.

Ready for day 3 tomorrow!

Well. I took about 17 pages of notes from day 1 about the action-packed day 1 of the LSi Formative Assessment conference, so this summary is going to be a bit… challenging, I think. Here we go:

Day 1 was technically a “pre-conference” event, but it felt like most of the folks attending the conference (about 300?) are here already. Tom Guskey led a day-long talk/discussion about grading practices, and it was wide-ranging and useful. I like Dr. Guskey’s writing and thinking, and his 5 hour presentation addressed many of the tricky topics involved in classroom grading.

(note: these are slides from Dr. Guskey’s website – I don’t think they are exactly the slides he used, but they are close)

Grading and Reporting Effective Policies and Practices
Developing Standards-based Grading & Reporting Systems

I liked Dr. Guskey’s realistic emphasis on the complexity of discussions about grading practices. He emphasized that these discussions are personal, important, emotional, and touch on many aspects of school and district “systems.” He recommended that schools/districts start by discussing the purpose of grades, and think about the various ways we communicate that purpose: as you discuss purpose(s), think about how you communicate this in the gradebook, the report card, and the transcript. He didn’t say this, but it feels to me like we need to be honest with ourselves about how much “courage” (or “capital?”) we have at every level. If we want to talk about changes in grading, we need to ask ourselves if we have the will and autonomy to implement changes in our online gradebook system? In our report card? In student transcripts? If so, cool (maybe). If not, maybe we should wait until we have the will and autonomy to do so.

At every stage in the discussion, Dr. Guskey emphasized that we need to start with the PURPOSE of grades. There are several possible purposes:

1. Communicate to parents
2. Provide info to students about proficiency
3. Select groups of students for instruction
4. Provide incentives for students
5. Evaluate effectiveness of programs
6. Document effort or responsibility 

Grades can’t do meet all these purposes! There are trade-offs: if you want grades to accurately communicate about proficiency, your grades may not be able to evaluate the effectiveness of programs (you may not have the range/variability you need for that purpose), or document effort or responsibility. He recommended that we pick a few purposes, and be honest about the trade-offs.

Dr. Guskey then led us through some of the advice in his excellent book On Your Mark. Short summary: percentage or points grade based systems (the most commonly used systems in secondary schools) are… limited. This one hits pretty close to home for me: our district uses a percentage based system, and we’re not changing any time soon for various reasons. His arguments are compelling and well supported. I think his recommendation – changing report cards to allow teachers to assign separate grades/marks for Academic Proficiency, Process (work/study habits), and Progress (growth over time) – make a LOT of sense. And I don’t think my district will be able to do this any time soon. He ended this section of the talk with some great examples of how districts in Kentucky have been able to change their report cards to be more standards based and “humanized.” I admire their work. Example:

Here are T. Guskey’s 15 steps for developing a standards based reporting system. Good advice, I say:

1. What is the purpose of the report card?
2. How often will the report card be completed? (parents want it more often, teachers want it less often)
3. Will the report card include individual standards for each grade level, or strands applicable across grade levels? (recommendation: put strands on report card, and standards in the gradebook) 
4. How many standards/strands will be included for each subject area or course?
5. Will they be end of year or benchmark standards? (if you choose end of year, student performance won’t look good at the end of 1st quarter) 
6. What product strands will be reported? (when you are developing rubrics, don’t worry about proficient – tell me what you expect at the top Rob – whoa!) 
7. What process strands will be reported? 
8. Will progress be reported?
9. How many perf levels will be reported per standard?
10. How will levels be labeled?
11. Will teachers comments be included? Class and student?
12. How will information be arranged on the report?
13. What are parents expected to do with the info? 
14. What policies need to be revised in order to support the new report card? 
15. How will parents/families be involved in revisiting the report card?

The day ended with a fabulous “dream team” panel: T. Guskey, D. Wiliam, S. Brookhart, and J. McTighe. Their discussion was wide ranging, fascinating, and too complex to summarize here, so I’ll just list some of my favorite ideas:

• D. Wiliam: It’s easy to be critical of grading systems, but if you look at them from the inside, you understand why.
• J. McTighe: Think about means and ends – practice is a means to an end. You need to be really clear about the end, and even if the practice is different than the end (e.g. running through tires) You give them feedback that relates to the eventual goal, not that specific practice. Don’t give them feedback on how well they run through the tires, give them feedback that will help them run better in the game based on what you saw after they run through the tires
• D. Wiliam: When teachers observe other teachers, the 1st question is “how’d I do?” The best administrators respond “How well did you think you did?” If they know that, your job is done. The goal if feedback is to make itself redundant.
• T. Guskey: be careful with the word potential. You don’t know what a student’s potential is. D. Wiliam: why be limited by your potential?
• S. Brookhart: I would be happy if districts just took the current system and made grades more reflective of learning. If you can’t do SBG, just do what you can do well.
• J. McTighe: what is WRONG with SBG? They are still based on grade level standards, and those grade level standards are norm referenced. Let’s do what they do in athletics or music – some up with a learning progression 

There’s a lot more to say, but I don’t think I’ll say it here. I’m excited for day 2!

I’m a lucky fellow: I get to attend the 2019 LSI Formative Assessment Conference at the University of Maryland.

Some of my favorite researchers/educators/writers are speaking at the conference. D. Wiliam, S. Brookhart, and T. Guskey are excellent scholars and writers, and I think they consistently translate research into usable ideas for K-12 folks. I’m not as familiar with J. McTighe’s work yet (other than Understanding by Design), but I’m excited to learn from him.

I’d like to post more often to this blog, so I’m officially challenging myself to “live-blog” (almost) this conference: I’m going to post something to this blog every night based on what I learn at the conference. Hope it’s useful for someone out there, and wish me luck!

[NOTE: an expanded version of this post appears in McEntarffer, R. (2021). Replacing the term formative assessment: A modest proposal. In S. A. Nolan, C. M. Hakala, & R. E. Landrum (Eds.), Assessing undergraduate learning in psychology: Strategies for measuring and improving student performance (pp. 57–65). American Psychological Association. https://doi.org/10.1037/0000183-005]

I’m worried about the term “formative assessment.” The term refers to an important idea – using assessment data DURING learning to make a change. But so many people use the term so differently, I fear that the important, core idea got lost.

In my district a decision was made quite a while ago to include the term formative assessment in on-line gradebooks. I understand why that decision was made, and well-meaning folks did it for good reasons. But one of the side effects was that the term “formative assessment” now means “an assignment that is worth fewer points” to students.

Dylan Wiliam is one of my favorite education researchers, and one of the people who “originated” the term formative assessment. A while back, someone asked a him “What’s the biggest mistake you made as a researcher?” on twitter, and his reaction was fascinating:

I like this idea: maybe including the word “assessment” in the term “formative assessment” wasn’t a great idea in the first place. Here’s my modest proposal: let’s replace the term formative assessment with two terms: responsive teaching, and student practice.

Responsive teaching could refer examples of teachers using assessment data (exit tickets, short quiz results, etc.) to make a change in their teaching. Student practice could refer to any time students use feedback to revise their work, try again, etc.

I don’t expect many folks to really stop using the “formative assessment,” but I think the terms Responsive Teaching and Student Feedback might be more descriptive and clear in most situations. Below is a slide I’ve used during discussions about all this – please feel free to steal!

(Note: thanks to Alex Bahe for letting me swipe a couple graphics)

UPDATE: An expanded version of this blog post appears as a chapter in
Assessing Undergraduate Learning in Psychology: Strategies for Measuring and Improving Student Performance, – Nolan, S. A., Hakala, C. M., & Landrum, R. E. (Eds.). (2021). Assessing undergraduate learning in psychology: Strategies for measuring and improving student performance. American Psychological Association. https://doi.org/10.1037/0000183-000)  – see chapter 4 “Replacing the Term Formative Assessment: A Modest Proposal “

UPDATE (April 2021) I just stumbled across this article: “Formative Assessment” on the NPJ Science of Learning blog (where I also blog sometimes!) In this blog post, Dylan Wiliam says:“we realized that using the term ‘formative’ to describe the position an assessment occupied in a course of study, or the assessment itself, represented… a “category mistake”—ascribing to something a property it cannot have—since the same assessment procedure could yield evidence that could be used summatively or formatively … there is, therefore, no such thing as a formative assessment. There are, however, assessments whose results can be used formatively. ” 

[Note: this blog post original appeared on the Noba Blog]

In a 2006 article Wylie and Ciafolo describe a technique called “single diagnostic items” that may be a great tool for teachers to use to gauge the impact of classroom demonstrations. Single diagnostic items focus on one important concept and “diagnose” student misconceptions about that concept. Imagine using a single item to determine what your students are learning! Wylie and Ciafolo define these items as “single, multiple choice questions connected to a specific content standard or objective. They have one or more answer choices that are incorrect but related to common student misconceptions regarding that standard or objective” (p. 4). The incorrect responses indicate a specific misconception about the concept, so that student responses identify specific misconceptions.

I wanted to see how single diagnostic items worked in a real classroom so I asked an instructor of an introductory psychology class at a local small liberal arts college for permission to work with one of her classes. She and I decided to focus on the topic of working memory. The text for the course did not cover this topic thoroughly and the instructor had not yet discussed this topic with the class.

My experience with single diagnostic items in the classroom

After introducing myself and explaining the goals of the research project, I asked the class to respond in writing to the prompt: “In a few sentences, please briefly describe working memory.” Then I conducted a working memory demonstration: Students closed their eyes and mentally counted the number of windows in their house. After they finished, they closed their eyes again to “count the number of words in the sentence I just said.” After they finished this task, students indicated whether they had to use their fingers to count when I asked them about the number of windows in their house (none of the students raised their hands). Then I asked how many used their fingers to count the number of words in the sentence (almost all the students raised their hands). Then I projected a single diagnostic item on the screen:

Why do most people use their fingers when they count the words in the sentence, but not when they count the windows?

• A. Windows are visual, and visual things are easy to process.
• B. Most people are visual learners.
• C. The windows are in long term memory, but the words are in short term memory.
• D. Familiarity – I’m more familiar with my windows than I am the words in that sentence, so that task is harder.
• E. I can picture the windows but I can’t picture the words, and that has something to do with it.
• F. Working memory must process words and pictures differently.

Students then indicated their response to this item (using their cell phones and the website Poll Everywhere: http://www.polleverywhere.com/). We briefly discussed the diversity of their responses, shown here:

In our discussion the students pointed out that at least one student in the class chose each of the possible responses. We discussed the frequency of the different responses : most students chose answer C (“The windows are in long term memory, but the words are in short term memory”) or answer E (“I can picture the windows but I can’t picture the words, and that has something to do with it”). We briefly discussed the diversity of responses and concluded that the data indicate that the class doesn’t yet have a common explanation for why the word counting task required almost everyone to count on their fingers and the windows counting task did not.

Then I explained the origin of the task: Baddeley and Hitch (1974) established that working memory is an active system made up of separate elements that deal with different kinds of information differently. To complete the “counting the windows” task, first working memory determines that the windows need to be pictured and then counted (“central executive” function). Then working memory activates the element that handles words and numbers in order to count the windows (“phonological loop”), and the element that can picture each window visually (visuo-spatial sketchpad”). When faced with the “count the number of words in the sentence I just said” task, the central executive encounters a problem. The phonological loop has to repeat the words in the sentence, but the visuo-spatial sketchpad can’t count, so most people have to use their fingers to complete the task.

After explaining the working memory research and terminology to the class, the students again wrote answers to the writing prompt “In a few sentences, please briefly describe working memory. “ They also again used their cell phones to vote on the correct answer to the diagnostic item:

The class discussed these data and agreed that the memory demonstration and explanation changed their conceptions and understandings about the nature of working memory. Almost everyone in the class agreed in the end that answer F “working memory must process words and pictures differently” was the most correct answer. We discussed the two previous most common answers (C and E) and the class was able to describe in what ways those responses were correct and incorrect.

Later I analyzed the students’ written responses to look for other evidence of changes in understanding of the working memory concept. I created a short rubric to use to score students’ pre and post writing responses:

Each student response was scored by me and a colleague who did not know which responses were “pre” and which were “post.” These scoring data also indicate changes in understanding the working memory concept.

Single diagnostic items like this one could be used to assess the effectiveness of the classroom demonstration about operational definitions. These “effectiveness data” could be used to make decisions about which demonstrations are most effective and which need to be modified. These same data could have multiple formative purposes: Teachers can regroup students into discussion groups based on their responses and ask groups to process the rationale behind their answers. Heterogeneous discussion groups might be useful, each student discussing their different answer with the goal of the group moving toward a consensus conclusion. Teachers could use the two most common answers and use other classroom demonstrations/activities to focus on those misconceptions directly. All these formative uses of the assessment data share a common characteristic: data from this one item are used to focus specifically on student misunderstandings about this important concept. This focus on the misconceptions these students demonstrate address student thinking actively and directly. The assessment data informs instruction by the teacher and metacognition by the students.

How to develop Single Diagnostic Items

Developing single-diagnostic items does require teachers to invest time in the item development process, but can save time in the classroom by efficiently providing valuable information about student misconceptions. One item-develop process is described below

1. Gather teachers who teach the same/similar content. Writing single-diagnostic items requires “deep” content knowledge, and is best done with a group of experienced teachers.
2. Choose a “big idea” to focus on. Single-diagnostic items take a while to write, so the group should spend its time focusing on an idea/concept/etc. that is a “big deal.” Some authors call these “hinge” or “threshold” concepts: ideas that students need to understand well in order to make progress in the discipline.
3. Ask the group to list misconceptions about the “big idea” (another way to phrase this task is to ask “How do students go wrong about this idea?) List all the misconceptions the group develops, then look at the list and collapse any similar ideas into appropriate categories.
4. Write a stem for the single-diagnostic item that will require students to use the “big idea.”
5. Write options for the single-diagnostic idea, one option per misconception and one possible correct answer. Note: multiple correct answers can be included, and the group should end up with one (and only one) option for each misconception. Ideally, if a student chooses an incorrect option, teachers should be confident the student did so because they are laboring under that specific misconception.
6. Test the item with real students. Participating teachers should use the item in class, and ask students who choose an incorrect response WHY they chose that response to test the relationships between incorrect options and misconceptions.
7. Revise based on feedback.

References:

Baddeley, A. D., & Hitch, G. (1974). Working memory. In G.H. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory (Vol. 8, pp. 47-89). New York, NY: Academic Press

Wylie, C., & Ciofalo, J. (2006). Using diagnostic classroom assessment: One question at a time. Teachers College Record, Jan. 10, 2006, 1-6.

This quote from Yana Weinstein (@doctorwhy) is important , I think. But I’ve been having trouble talking about it.

There’s a lot in this quote. Piles and piles of cognitive (and developmental) psychology indicate that humans use concepts and categories to understand the world. We just do. Our brains are meaning making and pattern finding machines. And it’s not a bad thing: generally making categories is an efficient, adaptive way to operate in the world.

BUT when we deal with other people, our categorization habits can get in the way. Badly.  Categories quickly lead to stereotypes, which can easily become prejudice, and if we have even a bit of power over someone else, prejudice becomes discrimination. And this all might happen without our awareness.

My favorite part of the quote is the last part: “The moral solution is to intentionally override the tendency to categorize individuals in the same way that we characterize other items that we encounter.” That sounds very matter of fact and clear, but underneath that statement is something profound, and inspiring, I think. When we deal with our fellow human beings, the other folks in our human family, we need to try to consciously “override” what our brain wants us to do at first: categorize someone. We should try to NOT judge their actions based on the categories and expectations built up based on our past experiences. We need to stop those immediate thoughts, remember that we’re thinking about another human being, and do our best to resist the influence of our internal categories (and stereotypes).

I expect there are many, many studies that show how unlikely this is. I’m certain that it’s difficult, and it may even ultimately be impossible. But I don’t want to think about that yet. Maybe it’s worthwhile thinking and talking about how we might at least try. What can help us interrupt the quick flow toward judgment? How can we try to override?

Stumbled across this site recently. Argh.

http://honestygap.org/what-is-the-honesty-gap/

The site is well designed, professional, and the statistics and analysis look convincing. And they get their NAEP score analysis almost exactly wrong. The purpose of the site is to argue for this claim:

“Frequently, states’ testing and reporting processes have yielded significantly different results than the data collected and reported by the National Assessment of Educational Progress (NAEP). The discrepancy between NAEP, the Nation’s Report Card, and a state’s claim is what can be described as an “honesty gap.”

They go on to provide state by state evidence for this supposed Honesty Gap:

The implication is that states are lying about student proficiency because their proficiency rates are so much lower than the NAEP proficiency rates. BUT this analysis, and this website, doesn’t include a crucial detail (and I suspect they know about this detail and they purposely fail to include it): the way the NAEP developers use the term “proficient” is VERY different from the way that term is used on the state achievement tests they are comparing NAEP scores to. Here’s a summary (from this Washington Post article) about this important difference:

“Oddly, NAEP’s definition of proficiency has little or nothing to do with proficiency as most people understand the term. NAEP experts think of NAEP’s standard as “aspirational.” In 2001, two experts associated with NAEP’s National Assessment Governing Board (Mary Lynne Bourque, staff to the governing board, and Susan Loomis, a member of the governing board) made it clear that:

‘[T]he proficient achievement level does not refer to “at grade” performance. Nor is performance at the Proficient level synonymous with ‘proficiency’ in the subject. That is, students who may be considered proficient in a subject, given the common usage of the term, might not satisfy the requirements for performance at the NAEP achievement level.'”

So the “Honesty Gap” argument falls apart before it begins: you can’t compare “proficiency rates” between the NAEP test and state tests, because the term “proficient” is defined very differently. I’ve seen this same argument on other sites (often promoting charter schools) and it’s just wrong wrong wrong. The only Honesty Gap the site argues for is their own dishonest use of NAEP data. Knock it off, please.

(originally posted at http://psychlearningcurve.org/summer-reading-list/ )

POSTED BY: ROB MCENTARFFER, PHD MAY 23, 2016