We're excited to kick off the New Year with the unique perspective of Jason Torres Altman from TerraLuna Collaborative. Jason has devoted his entire career to evaluating education programs that address inequity in schools and communities. In this conversation with Elana Leoni, CEO at Leoni Consulting Group (LCG), Jason discusses the power of data in education and EdTech, the expected results of evaluation, and the limitations of what's possible. With his grasp of details, compassion for the disadvantaged, and characteristic blend of humor and humility, he makes the wonky science of evaluation understandable. As always, we invite you to listen to the entire episode. Meanwhile, here are some of the highlights that we think are particularly interesting.
What People Don't Get About Evaluation
As someone who's coordinated evaluations for states, programs, organizations, and EdTech companies, as well as being an evaluation industry thought leader and conference presenter, Jason hopes that we can lay to rest some of the common misconceptions about evaluation.
On the foundation side, those who give the funds are eager to prove that their work is making an impact. Yet for all this eagerness, it's hard to get a meaningful evaluation when the funding is limited to a one-year cycle, as so many initiatives are. "It’s not likely that any intervention or program that was started with this funding would be able to show anything more than the shortest of short-term outcomes and be able to show that after one year," said Jason.
On the programming side, evaluation is often an afterthought to the program, an add-on that's underutilized and sometimes even ignored. As a result, Jason noted, "when evaluation is treated as an afterthought, [brands and programs] miss out on the parts where they could’ve been really honing in on the why, or learning more about the way they’re doing things, what’s working for who, and when." In his experience, by the time an organization is ready to hire a consultant like him, the report has been produced, packaged, and sent off to a funder without some of the compelling data that might have made a difference.
For the people whose program is being evaluated, Jason characterized a common misconception with the example of putting a white, male evaluator in an urban public school setting. "The misconception that an educator might have is that the evaluator is scary, and that they’re trying to do something to them," he cautioned, "so as an evaluator, you have to be really careful about how you set that lens, and have folks understand that you’re looking at a more expansive view of whether or not the program’s doing what it’s supposed to be doing, and what it’s doing best."
Bringing the Team Into the Mission
"The data doesn’t actually make the evaluation," Jason told us. In the end, it's about more than just the numbers. One powerful impact of evaluation is how it can bring a team together around a mission when the data validate their efforts or challenge them to rethink their approach.
Given this level of teamwork, Jason reflected, "Most nonprofits are very much startups. They’re young, they’re aspirational, they don’t have a lot of seed funding. And for any startup or nonprofit who is starting to generate ideas and debate the merits of whether a theory may be practical or off base, I would encourage folks working in the public PreK-12 space to engage active educators who know what it’s like to be an educator right now." He explained that as subject matter experts (SMEs), educators can help program developers understand potential breaking points in their theory and identify fundamental problems in their program or product that should be addressed when aiming for outcomes that will be useful for teachers.
"In the EdTech world," continued Jason, "there is already a culture of using data and trying to make data-informed decisions, or at least having the data available on a spreadsheet while you make a decision because it’s what you wanted to do anyway." But whatever outcome your team is seeking, he recommends beginning the evaluation process with a clear plan. What questions are most important for you to answer? Who can best help you understand those answers? When is the best time to ask those questions of those people?
Because many nonprofits and startups are limited by budget, time, and capacity, they might not be in a position to hire a consultant like Jason. However, they can still look at theories of action and theories of change on their way to discovering the likely impact of their product or initiative. Jason believes that practical learning is one of the best ways for organizations to truly own what they study, and he offered suggestions for how to proceed without a professional evaluator (see resources below).
A Word of Caution for Educators
As any educator can tell you, the education space is flooded with product offers. It can be an overwhelming task to sift through all the emails and phone calls to find just the right tools for a given classroom or school. We asked Jason for insight about how the average educator can even begin to evaluate a product without fully grasping whether it's solid in terms of the research and impact behind it. He began by pointing out that many school systems have someone at the district level with the role of evaluating new programs and curricula, which is often prompted by questions from individual educators.
"It’s sometimes hard to tell where internal evaluation or research ends and the marketing starts," Jason went on, "so I think educators need to start with the end in mind, and then you can probably send a lot of that information to the trash, because it isn’t designed to help you do that thing. Folks need to be technologically literate and maybe a little bit evaluation literate as well, and look for warning signs. Statistics that might be cited, do they seem legitimate? Do they seem plausible? Does it seem too good to be true? Where was the study done? Do you have information about which classrooms or students were benefiting from this intervention or whatever? Can you see those? Was it a large sample or was it just five people?"
If an educator has the opportunity and time, Jason suggests becoming part of an evaluation process, joining a pilot program, and helping to collect the data that will fine-tune the tool. But since that's not always possible, he cautions educators to tune out the sales pitches and seek tools that they know have been tested with "valid, reliable, gold standard-type research, and there’s proof that it worked."
Are We Making a Difference?
For every practice that's abstract to many people yet absolutely essential for improving how things are done, we think there should be spokespeople like Jason who can put things into friendly, approachable terms. We asked him what has kept him inspired during his 20-plus years as an evaluator.
"I’m so privileged to have this role," he admitted, "to help folks who are doing something a little bit extra, not just the core curricula. If there was no evaluation happening we would be awash with great ideas, but would have no idea which ones actually were going to make a difference for teachers and students. There are millions of teachers out there every day giving far more of themselves than any one should ever ask. And they need to have access to the very best tools to use with their students. And I’m able to play a small role in that, and my job is so much easier than having to do any of that, right?"
Take a look at the full transcript of Jason’s podcast episode.
Note: This interview was originally recorded on December 10, 2021, as part of the All Things Education and Marketing podcast hosted by Elana Leoni.
Resources Mentioned in this Episode:
TerraLuna Collaborative (consulting firm specializing in evaluation, facilitation, and research services to support systems change initiatives.)
Michael Quinn Patton's Evaluation Flash Cards 5 Cartoons (Evaluators are like court jesters)
For companies interested in getting started in evaluation:
Get in touch with Jason: email
Episode Skeleton (use this to jump to the parts you want to listen to):
[00:53] Jason’s introduction.
[03:08] Jason discusses his career path and the work of an evaluator.
[07:54] What are some misconceptions of evaluation?
[12:02] “Evaluation is sometimes done when it has to be done.”
[16:20] Jason talks about founder's syndrome.
[17:38] Evaluators as court jesters.
[18:30] Working with decision-makers as an evaluator.
[19:29] Jason talks about the role of an evaluator.
[20:19] What does that mean on a theory of change and theory of action level?
[22:24] Evaluation puts the impact first.
[25:04] This (Q1) is a great time to bring your team together back to the company mission.
[26:10] The educator’s role in evaluation.
[30:29] How can companies (even with limited resources) get started with evaluation.
[35:41] How can educators evaluate whether a product will work for them?
[41:55] What keeps Jason inspired to be an evaluator.
[44:10] How to get in contact with Jason.
Elana Leoni, Host
Elana Leoni has dedicated the majority of her career to improving K-12 education. Prior to founding LCG, she spent eight years leading the marketing and community strategy for the George Lucas Educational Foundation where she grew Edutopia’s social media presence exponentially to reach over 20 million education change-makers every month.
Jason Torres Altman, Guest
Jason Torres Altman provides evaluation support to entities providing advocacy for, policy support for and programming in K-12 public schools, extension services, and community development. Since his career began in 2002, in all project work he emphasizes elevating the voices of the most affected, and often least empowered stakeholders as well as deeply considering nuance and context at the local level. His desire is that evaluative support can contribute to the changing of hearts and minds about what is important in local communities and the responsibility to serve local communities in the way that they choose to be served. Jason is a published professional with more than 40 academic papers, reports, and journal articles, and more than 80 academic presentations to his credit.
About All Things Marketing and Education
What if marketing was judged solely by the level of value it brings to its audience? Welcome to All Things Marketing and Education, a podcast that lives at the intersection of marketing and you guessed it, education. Each week, Elana Leoni, CEO of Leoni Consulting Group, highlights innovative social media marketing, community-building, and content marketing strategies that can significantly increase brand awareness, engagement, and revenue.
Rate, Like, and Subscribe
Let us know what you thought about this episode by rating and reviewing our podcast. Click here, scroll to the bottom, tap to rate with five stars, and select “Write a Review.” Then be sure to let us know what you loved most about the episode! Also, if you haven’t done so already, subscribe to the podcast to be notified when we have more content to share with you.