June 2, 2026 · Episode 314
AI in Higher Education: Bias, Procurement, and Human Oversight
38 Min · By Dr. Drumm McNaughton
AI procurement in higher education can scale bias fast. Learn what presidents and boards should ask about data, validation, ethics, and human oversight.
Most higher ed AI procurement skip the four questions that determine whether the tool will scale institutional bias. A framework for presidents and boards on what to ask before signing.
As a high schooler with straight A’s in math and science, Dr. Karen Panetta was told by her school’s career assessment that she could sell makeup or be a cook. A male friend with lower scores got engineer or politician. Dr. Drumm McNaughton, host of the Changing Higher Ed® podcast, was told by his post-Navy assessment to become a forest ranger. No AI was involved in either case. Just rules-based systems applying gender and biographical constraints.
That same logic now sits inside AI tools landing in admissions offices and HR systems across higher ed, and McNaughton names the difference that should keep boards awake: AI does not eliminate human bias, it removes the human accountability that used to make it correctable. The candidate who hears “the AI said no” with no other explanation is in the same position Panetta was in at sixteen, except the institution now has fewer ways to defend the call.
Procurement is where the response begins. Before signing any AI contract, higher ed leaders should be able to get specific answers to four questions:
- Where did the training data come from?
- Was it ethically sourced?
- How was it validated, and by how many experts?
- Who is the human in the loop when the system gets something wrong?
Panetta gave one example of the validation question in action. Six doctors looked at the same medical image during a cancer detection project. They agreed on whether it was cancer. They disagreed on every grade beyond that. If domain experts cannot agree on ground truth, the AI trained on their assessments cannot produce a reliable grade either. Most products in front of higher ed buyers do not carry that level of validation history, and most vendors do not volunteer it.
The Myth of the Magic Box
AI is widely treated as a black box that takes a question and returns an answer. Panetta argues the reality is the opposite. AI output is only as good as what she calls the ground truth, known data with known outputs that an expert has vetted. Scrape the open web with no provenance and no expert review, and the result is garbage. Garbage in, garbage out.
Her sharpest analogy on this point came through a Dracula reference. Villagers used to fight Dracula with garlic. The garlic loses its potency overnight. AI works the same way. If enough people repeat something often enough, AI’s interpretation of the truth ages with the consensus. History gets rewritten in the model’s worldview, not in the historical record. That is a problem for any institution treating AI output as authoritative without checking what the consensus was built on.
It is also a specific problem for graduate research. Panetta has watched students and researchers ask AI for a unique contribution and accept whatever comes back. The trouble is that the same suggestion is going out to thousands of other users asking the same question. A unique contribution by definition cannot come from a system handing the same answer to everyone.
McNaughton framed the institutional implication through Bloom’s Taxonomy. The lower levels, knowledge and comprehension, have been reshaped by AI. Students can retrieve information instantly. The value has moved up the pyramid to analysis, evaluation, and the ability to read something and judge whether it makes sense. Foundational knowledge has not become irrelevant. It has become the prerequisite for everything that matters above it, and the faculty role is shifting toward exactly the mentoring that AI cannot do.
Bias at Scale: Why “The AI Said No” Cannot Be an Institutional Answer
The personal stories in the opening are the visible tip of a much larger institutional problem. Bias was in human decision systems long before AI existed. The risk now is volume and velocity. An admissions team that quietly carried bias in a small number of hand-reviewed files can now deploy that same bias across every applicant in the pipeline through an AI screen, faster and at lower cost, with no human signature on the rejection.
Panetta surfaced the variance that already exists inside graduate admissions, even without AI. When she reviews applications, she weights projects, volunteer engagement, and how a student learns. Other faculty on the same committee weight brand-name undergraduate institutions, published work, and traditional credentials. Both are defensible perspectives. But if the second set of weights is what gets encoded into an AI screen and the first is not, the institution ends up with a narrower applicant pool than it intended, and it cannot easily see what it lost.
The institutional fix is not to bolt on a bias detector at the end. The fix is to keep a human in the loop on every consequential decision, and to require that human to be able to explain the call out loud. If they cannot, the institution is one lawsuit away from being asked to defend a process it does not actually understand.
What to Ask Before You Sign: The Procurement Framework
The four questions in the opening are the procurement floor. They look simple. Most vendors cannot answer them in any specific way, and a no-answer is a meaningful answer.
On data provenance, the test Panetta runs is whether the data fits the population it will be used on. Comparing World War II smoking-and-nutrition data to a modern population that exercises and reads labels produces conclusions that do not apply to either group. Comparing nutrition outcomes from a developing country to Boston is the same problem in reverse. Data ages, and it ages unevenly. Population, time, and context all matter.
On ethical sourcing, Panetta pointed at health advertising as the everyday case. Open any health ad on the web, and there is a real chance you are looking at a real person whose backside is in the image. Did that person consent? She asks the same question about photos of the elderly in medication ads. AI-generated imagery should carry a declaration that it was generated. There is no equivalent of a grade-A stamp telling viewers what they are looking at.
The procurement bottom line is direct. Do not believe anything that claims 100 percent accuracy. Ask how many test cases were run, what populations were covered, and where the training data came from. If the vendor cannot give specifics, pass. Panetta also gives IEEE credit on this point. With roughly 500,000 technical professionals globally and no corporate paymaster, IEEE is working on AI standards in the public interest, not on behalf of any one company. Institutional buyers should be asking whether the products they are evaluating align with that kind of independent standards work.
IRB and Do No Harm Apply to AI
McNaughton and Panetta closed quickly on the point that the principles behind Institutional Review Boards already give higher ed a working framework for AI. Real research has been IRB-approved for decades. The first principle is do no harm. If an institution is using AI, Panetta argues, it owes its students and faculty the same level of design rigor that gets applied to any other research involving human subjects.
She extended that beyond traditional research. She works on a conservation project that tracks elephant health and movement for park rangers. If the system is not wrapped in serious security, poachers can use the same tracking output to find elephants faster than rangers can intervene. Every kind use of technology has a misuse case sitting next to it. Building those misuse scenarios into the design process is, as she put it, a whole career in itself.
For higher ed leaders, that means asking the IRB question about every AI deployment, not just the ones with obvious human subjects. Who could be harmed if this system fails. Who could be harmed if this system is misused. What protections sit between those failure modes and an outcome the institution will have to answer for.
Teach AI to Everyone, Not Just the Engineers
Panetta is clear that AI cannot be left to engineers. The practical reason is that every discipline will use it. The more interesting reason is that the questions engineers do not think to ask are precisely the questions higher ed needs answered.
Faculty in English, history, and the arts are the ones who can identify where AI is flattening creativity, where it is averaging students into a generic voice, where it is missing the right question entirely. Panetta is actively engaging arts and sciences faculty at Tufts because she believes the technical disciplines need the humanistic disciplines to build what she calls smart hooks into AI for explainability and protection. McNaughton pointed at the same dynamic from the strategic side: knowledge has become specialized enough that complex problems now require interdisciplinary teams to solve, and that has to be reflected in how institutions organize their AI work.
Panetta’s concern, in her own words, is that we are all going to sound the same. Gen Alpha is growing up with AI as default infrastructure. If students learn to outsource the work of finding their own voice to a model that gives the same suggestion to every other user, the output of higher education converges on whatever the model considers normal. The institutional question is whether faculty are teaching students to use AI with skepticism and an awareness of their own voice, or whether the institution is quietly subsidizing that homogenization.
Foundational Knowledge Still Matters
The temptation in this environment is to skip the fundamentals because students can retrieve information on demand. Panetta pushed back hard.
She calls the current moment a Back to the Future technology cycle. Theories engineers could not implement decades ago because the computation and materials did not exist are now realizable. Digital twins, nanomaterials, simulation at billions-of-experiments scale. None of that leap is possible for a student who does not understand the underlying theory. AI accelerates the work for someone who knows the foundation. For someone who does not, it produces output they cannot evaluate.
She also pressed on something faculty hear and student services teams already know. Students need to fail early and often in low-stakes settings, so they learn how to recover. Her framing was direct: if you lose a game, you do not quit the sport. You train harder, you adjust, you try strategically different things. Engineering, in her phrasing, is our sport. The same is true for any discipline that requires building real competence.
Data Ownership Is an Institutional Strategy Question
Panetta’s research touches sensitive datasets, including facial recognition for human trafficking prevention and dental imaging for training new dentists. Some she has made public. Others she holds back because the misuse risk outweighs the open-source benefit.
The institutional analog is that universities should be deliberate about which AI platforms they allow their data to flow into. Several campuses have already restricted certain large language models because research data uploaded to those tools becomes training data for someone else’s downstream product. Other institutions are building their own AI infrastructure or contracting with providers who guarantee that uploaded data does not leave the institutional perimeter.
This decision is not academic. Once research data, student data, or proprietary institutional data is fed into a public model, control over that data is gone. Institutions that have not made an explicit policy decision on this have made one by default. Panetta’s point on human data was sharper still. When there are people in the data, that data has to be the most protected class the institution holds.
Three Takeaways for University Presidents and Boards
- Do not be afraid of AI, and do not let any one constituency own it. Teach students to use AI responsibly. Make sure faculty in every discipline, not just engineers, are learning it and discussing how it should be used ethically. The questions humanities faculty raise are the ones engineers need to hear, and the conversations that emerge from that mix are the ones that produce smarter institutional policy.
- Do not use AI outputs to determine the future of faculty careers or student admissions. The temptation to automate promotion review, tenure screening, or admissions filtering is strong. The risk is that the AI encodes whatever bias the humans behind it brought in, and the institution loses the ability to defend the decisions it makes. Keep a human in the loop on every consequential decision, and require that human to be able to explain the call out loud.
- Do not believe anything that claims 100 percent accuracy. When a vendor pitches an AI tool, ask how many test cases it was validated on, what populations those cases covered, and where the training data came from. If they cannot give specifics, pass. The sticker price at the top of the spec sheet is not the story. The data behind it is.
Quick Overview
- AI does not create bias. It scales whatever bias is already in the institution’s decision systems, at the speed and volume the institution chooses to deploy.
- AI does not eliminate human bias. It removes the human accountability that used to make bias correctable.
- Most AI products in front of higher ed buyers cannot answer four basic questions: where the data came from, whether it was ethically sourced, how it was validated, and who is the human in the loop.
- Six doctors agreed on whether something was cancer. They disagreed on every grade beyond that. If domain experts cannot agree on ground truth, the AI trained on their decisions cannot either.
- Don’t believe any AI product that claims 100 percent accuracy. Ask about test cases, populations, and training data. No specifics is a meaningful answer.
- Garbage in equals garbage out. Scraped open-web data with no provenance is not a foundation for serious work.
- AI’s interpretation of truth ages with the consensus. Enough repetition rewrites history inside the model.
- A unique research contribution cannot come from a system handing the same answer to thousands of other users.
- IRB principles, especially do no harm, give higher ed a working framework for evaluating AI deployments.
- Every kind use of technology has a misuse case sitting next to it. The misuse case has to be designed against.
- AI cannot be the exclusive domain of engineering and computer science. Humanities and arts faculty are essential to identifying where AI fails.
- Gen Alpha is growing up with AI as default infrastructure. The risk is that students learn to outsource the work of finding their own voice and end up sounding the same as everyone else.
- Foundational knowledge has become the prerequisite for evaluating AI output, not an obsolete burden.
- Students need to fail early and often in low-stakes settings. Engineering is our sport, in Panetta’s framing.
- Data ownership is an institutional strategy question. Universities that allow data into public AI tools have made a policy decision by default.
- IEEE, with 500,000 technical professionals globally and no corporate paymaster, is working on AI standards in the public interest. Institutional buyers should be asking whether the products they are evaluating align with that kind of independent standards work.
About Our Guest
Dr. Karen Panetta is an electrical and computer engineer, inventor, and Dean of Graduate Education for the School of Engineering at Tufts University. An IEEE, NASA JOVE, and AAIA Fellow, her research spans AI, machine learning, automated systems, and visual sensing, focusing on algorithms and embedded systems for robot vision and biomedical imaging. A celebrated educator and researcher, Panetta received the Presidential Award for Excellence in Science, Math, and Engineering Mentoring from President Barack Obama. Dedicated to empowering women in STEM, she founded the “Nerd Girls” program, serves as editor-in-chief of IEEE WIE Magazine, and co-authored “Count Girls In.” Driven by a passion to use engineering for humanity, she has advised world leaders and spearheaded global humanitarian projects aimed at improving lives, building thriving communities, and protecting the environment.
About the Host
Dr. Drumm McNaughton is the founder, CEO, and Principal Consultant at The Change Leader, Inc. A highly sought-after higher education consultant with 20+ years of experience, Dr. McNaughton works with leadership, management, and boards of both U.S. and international institutions. His expertise spans key areas, including accreditation, governance, strategic planning, presidential onboarding, mergers, acquisitions, and strategic alliances. Dr. McNaughton’s approach combines a holistic methodology with a deep understanding of the contemporary and evolving challenges facing higher education institutions worldwide to ensure his clients succeed in their mission.
Read the Podcast Transcript →
Transcript: Changing Higher Ed podcast 314 – with guest Dr. Karen Panetta
Introduction to Changing Higher Ed®
Welcome to Changing Higher Ed®, a podcast dedicated to helping higher education leaders improve their institutions. With your host, Dr. Drumm McNaughton, CEO of The Change Leader, a consultancy that helps higher ed leaders holistically transform their institutions. Learn more at changinghighered.com. And now, here’s your host, Drumm McNaughton.
[00:00:31] Introduction
Drumm McNaughton: Thank you, David.
My guest today is Dr. Karen Panetta. Karen’s an electrical and computer engineer, she’s an inventor, and she also happens to be the Dean of Graduate Education for the School of Engineering at Tufts University. She’s a fellow at IEEE, NASA, JOVE, and AAIA. Her research spans AI, machine learning, automated systems, visual sensing.
In short, she’s a renaissance woman who can really talk about AI in a way that people can understand it, especially if you’re not a geek. She received the Presidential Award for Excellence in Science, Math, and Engineering Mentoring from President Barack Obama, and she joins me today to talk about AI, education, and how we need to teach and use AI in our schools today.
Karen, welcome to the show.
Karen Panetta: Thank you for having me. Excited to be here.
Drumm McNaughton: Thank you. I am excited to have a good conversation with you. When we were talking a few days ago, you really struck me as a renaissance woman, doing so many different things. Please tell me, tell the listeners a little bit about your background.
Karen Panetta: Sure.
[00:01:44] Dr. Karen Panetta’s Background
Karen Panetta: Well, first of all, a lot of people call me a pioneer, but sometimes I like to think of myself more as a warrior, because I’m, treading waters that no one’s ever looked at before. So I am a distinguished professor and dean at Tufts University, but I came from industry. I am the co-inventor of the first, and I have been working with simulation and imaging processing.
And when I came to Tufts, I was the first woman, in, as a professor there in engineering, so that was a kind of a different experience.
[00:02:16] AI Across Domains
Karen Panetta: But I am now in AI, artificial intelligence, and what I do is I like to say I build AI engines and I put them into different vehicles, whether it’s dental for health or medicine, even conservation for animals as well as traffic management. And I’m even doing it underwater.
Drumm McNaughton: Okay. I’m not sure I want to…
Karen Panetta: Ocean conservation is absolutely, well, looking at our oceans, mapping our oceans, understanding what’s there. It’s real exciting, and I think it’s our new frontier, actually.
Drumm McNaughton: Yes, absolutely. My, experience on oceans was flying over the top of them when I was in the Navy. Hopefully, we didn’t get into any of them. But you- you’ve also done a lot of work. You’ve been recognized by IEEE, other things. That’s an amazing career.
[00:03:11] IEEE Global Impact
Karen Panetta: Well, IEEE, Yes, I, IEEE has really been a partner in my entire career journey. because you have to remember, when I came from industry into academia, that’s a very unusual switch. So IEEE is the world’s largest organization for technical professionals, and we have 500,000 members around the world, which means when I want to do a project on autism in children in India, I’ve got thousands of helpers over there ready to jump on board, and that’s exactly some of the projects that we’ve done.
So I tracked autism in young girls in India. I’ve worked with zoos in African countries. I’ve done power for orphanages in Malawi and other third world countries to help bring electricity and health and fresh water. So, not typically the things you see with electrical engineers, but I think that that’s what we’re really trying to promote, is just how interdisciplinary it is to be an engineer and understanding how you can solve, make huge impact to other people’s problems.
Drumm McNaughton: And because knowledge is becoming so specialized, it requires a team to work on some of these complex problems we have.
Karen Panetta: Absolutely, yes.
[00:04:25] Designing for People
Karen Panetta: I think that that’s the first lesson I try to teach my engineers, is you think because you’re an engineer, you already know all the solutions for everybody’s problems. While we know technology, you really have to understand who or what the client that you’re serving and the populations.
Some- we’ve seen places when we’ve brought in really cool technology, well, if it’s super expensive, can’t be maintained, or it’s intimidating, they won’t use it no matter how useful it is. So to leave a long-lasting impact, it has to be conducive to the people that you want to use it. And it also has to be low cost because if you go implement something that is worth more if they steal it and take it apart and sell the parts, you’ve defeated the entire project.
Drumm McNaughton: Exactly.
[00:05:08] Tufts Interdisciplinary Culture
Drumm McNaughton: And so speaking of interdisciplinary, Tufts is an incredible institution. My uncle actually graduated from Tufts many years ago as a chemical engineer. Tell me a little bit about Tufts and how this interdisciplinary work works, especially with AI.
Karen Panetta: Oh, sure. Well, your uncle must be filthy rich now because Tufts graduates are, make a, r- really do very well, become very prominent leaders in industry, even if they don’t stay in engineering. So that’s part of the recipe and hallmark of Tufts, is right now- It’s not just about the technology skills, but the well-roundedness, the liberal arts.
We have a lot of students that study multiple disciplines, but the beautiful thing that I really like about Tufts is the way I can integrate with the dental school, the vet school, the school of nutrition, medical school. It doesn’t matter what, if I’m an engineer, I can pick up the phone and, or excuse me, Zoom with anybody to have meetings and bounce ideas off of people.
[00:06:09] Elephants Drones and Conservation
Karen Panetta: I once received a phone call saying, “We want you to work with elephants.” And I was like, “I love elephants. It’s the Tufts mascot, but what’s that got to do with electrical engineering?” And that was conservation. So I did, I owned the first drone on the campus.
I learned how to work with drones from my IEEE colleagues in robotics and automation society. And I started an animal conservation project that now is recognized for being able to detect and recognize animals, and helping vets with diagnosing their health remotely so that they don’t have to anesthetize them or get squished or anything else.
Drumm McNaughton: And you use AI with all this, correct?
[00:06:50] Computing Power and Digital Twins
Karen Panetta: Now I do. Back then, we didn’t put, we didn’t have the computational power that we did now. So if you think back to my days when I was doing the digital twin, I had one CPU, and I was doing millions of experiments on time-shared computers. Today, because of the power of computing, we can do billions of experiments, and we’re looking at is all these different scenarios to expedite solutions or try out new ideas.
And the AI now helps us with detection and recognition. It’s also helping us find new solutions from medicines, as well as analyzing data that was just too cumbersome for us to do. But I think I have found it really helpful for, again, with my digital twin background, to help us emulate physical experiments that before used to take lots of time, money, and resources, and now we can model it and put it right into AI, and it can give us some really tangible outputs.
[00:07:47] Can AI Innovate
Drumm McNaughton: Help me out, because this is one of those curious things that I’ve had. My understanding of AI is it, it does tremendous research, it can look into nooks and crannies that many humans don’t know to give solutions. But does it think for itself and take you to that next step that’s needed for innovation?
Can it do that?
[00:08:08] Myth of the Magic Box
Karen Panetta: So, this is part that I would call it the myth. People right now, they think it’s this magic black box that, you ask it a question, it comes out and gives you the answer. And really what we have to understand is- The data that you feed it or, what I call the ground truth, is meaning, known data, with known outputs of what the responses should be.
You have to have that at this foundation. If you don’t and you’re just scraping the web and, getting data from anywhere, you’re going to get garbage. Garbage in equals garbage out. So, what I’ve been trying to push is that, people have to understand that AI’s not this magic box. It really requires significant knowledge of your data , and the context of the data.
Because I can put in data, have it give me something that mathematically looks, oh, that, all right, that makes sense. But for the situation, it might not make any sense, and that’s why you still need experts, and humans in the loop, that I would say. So we’re not there yet, but, it’s being portrayed that way, but we’re not there yet.
Drumm McNaughton: Yes. And that’s a critical piece that you mention as far as where we are with AI is there’s a lot of benefits to it, but it requires the human in the loop, especially when it comes to ethics, and we’re going to get into ethics in a little bit.
Karen Panetta: Yes.
[00:09:24] Deepfakes and Public Awareness
Karen Panetta: I think that one of the, promotions that I’ve been trying to do, and my colleagues and I at IEEE, is we understand that part of our job is, the public imperative to help explain technology to everyone, not just other nerds like me. You have to really be able to talk to children, and that starts from birth.
People always think, oh, you have to wait. No…. We’re in a renaissance right now, similar to when we first had the internet come out and we had all the cyber phishing and things like that. And now with deepfakes and all those other types of high-end AI that can help manipulate and convince people of, I would say that something that is really isn’t.
If you could hear the voice of someone that you love, and it might sound legitimate. So, we’re really trying to train young people as well as the entire public that there’s cautions. You have to proceed with caution, just like you did when the internet first came out, or, and even when the telephones, right?
Remember your telephone used to come up and have caller ID and it was, you were late on something, the police would come in and get you.
you,
Drumm McNaughton: Oh, Yes.
Karen Panetta: It looked legitimate, but now we’re like, “Yes, that’s spam.” Well, we haven’t got that same level of awareness in our public yet, and that’s, I think, the first challenge that we’re trying to overcome.
The second thing is just like, when the internet came out, the laws, the rules, the safety, the guidelines tremendous lag behind it. So, we’re seeing that now. You can’t create the laws fast enough to compensate for what’s going on. And then finally, Yes, you think about how much work there is to catch up.
So for everything we’re doing, there’s always somebody behind you trying to, undo it. So you always have to keep up with the bad actors as well. So, we’re going to be getting our steps in trying to keep up with this.
Drumm McNaughton: Our digital steps in.
Karen Panetta: Oh Yes, our digital steps.
[00:11:15] Three Ethics Questions
Drumm McNaughton: So this is interesting because there are all sorts of ethical things that the mainstream media is just now starting to come out. They’ve been glossed over. For example, a newcomer to AI, there’s three things that they really need to note, isn’t there?
Karen Panetta: Yes, yes. They need to know, one, where did the data come from? Was it ethically sourced? And, if I use this data, is it going to actually harm somebody’s reputation or is it going to make an impact on someone’s career? And you have to also be able to know when you see a result of AI, you also have to ask, well, how many test cases was this and where was this data sourced from?
So if we’re trying to do nutrition in a third world country and comparing them to Boston, Massachusetts, that’s not even going to apply. So you have to understand that. And data ages out, right? So, if we’re trying to look at data sets of people from World War II where everybody smoked versus today where everybody’s on the treadmill and everybody’s trying to be healthy, it doesn’t make sense.
But I want to bring up a point about just ethically sourcing because people don’t even know what that means. If you see a picture, and I think about this all the time, I look at websites and you see all these ads for, weight loss and others like that, and they show you somebody’s backside, and I ask that question.
I’m like, does that person give permission to have their picture, even though you’re not showing their face, is it really okay to do that? And if it was my backside, I would know it and I would not want it out on the internet like that. So that’s an example of “is it ethically sourced?” And even now I see a lot of things like showing the elderly and saying, oh, ads for different types of medications and things.
And I’m like, “did they really take a picture of somebody in a hospital? Is this digitally generated?” So there should be some sort of declaration that this is artificially generated, but we don’t have that kind of stamp of approval or, grade A, AI.
[00:13:13] Algorithms Headlines and Bots
Drumm McNaughton: It strikes me as very similar to those people who get all their news from social media instead of reputable news sources, and I’ll, mention NPR as being one of those. They report on both sides of the issue. Social media, because of their algorithms, they continue to feed you information that they think you like based on your previous clicks.
Is that really the truth?
Karen Panetta: Right. I think a lot… Well, right now what we see a lot in the news is headlines, headlines that are meant to grab you and suck you in. And they’re going to feed you more of what they think you want to see. So sometimes when we’re doing research, I, we’ve been doing some research, and all of a sudden I’m getting these websites coming up, and I’m like, “How did I get myself involved in this?”
And it was because I was doing research on that area for some reason, and then I get s- and I was like, “I don’t want to see this. This is not what I wanted to see it.” but you have to think about those key questions, ” where is this data coming from?” Because we have seen where millions of bots can influence and, say that, “this poll says this.”
It can influence those things, and if we can’t track it back and make it explainable, then it’s not real. So right now it’s a huge challenge because if millions of people think it’s true, then guess what? We’re rewriting history, and AI’s going to say it’s true. And I always give this analogy in my talks.
I always say, if I were to ask, “Does Dracula have an aversion to garlic?” People would say, “Yes.” Does that mean Dracula’s real? we could rewrite history, and we can also see that it could actually hurt people. And I think we’ve seen some of the deep fakes that have actually hurt celebrities, but they have the bank account to track these perpetrators down and get it removed.
You and I, they put it up there, we’re dead. We got nothing.
Drumm McNaughton: Oh, Yes, no, no kidding.
[00:15:04] IRB Security and Do No Harm
Drumm McNaughton: And one of the things that you and I have talked about before is real research is IRB, Institutional Research Board approved. Every research higher ed rin-
Karen Panetta: Do no harm. That’s right. Do no harm.
Drumm McNaughton: Do no harm. Most IRBs focus on interaction with humans, but it goes beyond that, especially with you, the research that you’ve done on elephants.
Karen Panetta: Yes.
Drumm McNaughton: That was an interesting example.
Karen Panetta: Because if I’m tracking for park rangers the health of elephants and I’m looking at their roaming and where they are and their health, if I don’t put a layer of really good robust security around that, well, then the bad actors can come in and find the elephants and by the time you get there, it’s too late.
So that’s why it’s essential that for everything we do, every good kind thing that we try to do with technology, we have to also now think differently than when I was in school of just get it to work, to how is somebody going to misuse this? And that is a whole career in itself.
Drumm McNaughton: Oh, absolutely.
[00:16:09] Validating Ground Truth
Drumm McNaughton: And how is the data collected is incredibly important, but equally important to a researcher is how is this data validated?
Karen Panetta: That’s true. That’s true. And having it validated by experts is really essential, and I can give you an example. I was doing a health application trying to look at a cancer, and we asked six different doctors. And they were very good saying “cancer, not cancer,” but then when we asked them to grade it, it was all over the place.
So I, as somebody who’s going to use this as a ground truth, I’m like, “Well, how do I use this as a ground truth if the experts don’t agree?” So that’s an example of, you really have to have robust metrics to be able to say what is the ground truth in the beginning.
That’s why I love IEEE, because they’re trying to come up with standards. They’re, you mentioned looking at both sides. IEEE’s looking only at the public good and the, benefit to humanity, so they’re looking at the technology and the standards to say, “Okay, what are uniform standards that anybody can apply for AI, for data?”
And that’s why I think the organization’s famous for its standards, because it does that, and it’s not, that some company paid them to do it. It’s like, no, we look across, we take all perspectives and we’re going to make sure that whatever we give out to the public is in your best interest and is fair and can actually be utilized so that we have a common language.
Drumm McNaughton: Mm-hmm.
[00:17:31] Bias and Human Decisions
Drumm McNaughton: So of the three things for newcomers, we’ve talked on, data, how’s data collected, how’s it validated, but I would say probably the most important one is something we touched on earlier, having a human at the end that makes the decision.
Karen Panetta: Yes. So, how many times have we seen people’s evaluations, you didn’t get the promotion. Why? Well, the AI said it. I always give the same story that, while it wasn’t AI when I was in school, I was getting straight As in math and science, and yet when my friend who scored lower than me got, math and science scores lower, the job assessment said he could be an engineer or a politician, and mine said I could sell makeup or, or be a cook.
And I was just like, “Where did this come from?” And I was really fortunate that my dad was like, “Where’d this come from? because my daughter can’t cook.” And that’s something I still can’t do. But it was, it just showed there was bias, and we haven’t really talked about what bias is.
But when you have humans putting in the rules, then guess what? The AI’s going to perpetuate those biases.
Drumm McNaughton: Mm-hmm. Oh, absolutely. And even the humans have biases. We all know this. But when you’ve got a human programming AI, it’s going to build in a bias, a basic bias right off the bat, is it not?
Karen Panetta: It is. It is. Because, like I’ll give you a for instance. If we were going to say, like I’m, I’m at a, I’m at a school, I’m the graduate dean, and we’re looking at applications. Well, the way I look at applications, I don’t care where the student comes from. I don’t care where their institution that they were before.
But some faculty might say, “If I’m going to write this, I want them to come from a brand name institution that I’ve heard of. Their grades have to be this way. Their experience, they have to have published.” Where I’m looking more for the projects, I’m looking more for the volunteer engagement, I’m looking more for how they learn. I have a very different perspective.
So if you’ve got that type of evaluation where somebody’s just looking at, “I only want this subset of schools, I want this,” well, guess what? You’re only going to get applicants that to pass or get the higher, recommendations. And we’ve seen that in jobs, and that’s unfortunate because what happens is we exclude people that could be wonderful contributors.
But even worse is, like, when you go up for a promotion and it says, “Eh, Yes, the AI says you’re not ready.” You’re like, “What? Give it to me. Tell me why.” “No, it’s just the AI.” So it’s the same thing, like when I was in high school, nobody could tell me why I had to be a cook or sell makeup. They, other than I knew that it was because somewhere in there it was like, “If girl, then, exclude engineering.”
Drumm McNaughton: Oh, I understand. When I did those, I got out of the Navy and did some of those. They said I should be a forest ranger, and it’s like, “Seriously?”
Karen Panetta: You’ve gone from water– You’ve gone from ocean to, to, to forest. Okay.
Drumm McNaughton: Exactly.
[00:20:17] Teaching AI to Gen A
Drumm McNaughton: So AI is, in many respects, changing the face of education. We need to be teaching AI early. I think about it from the generational perspective.
We got millennials. We’ve got Gen Z. Gen Z has been with technology. They were born with a smartphone in their hand, weren’t they?
Karen Panetta: Unfortunately, yes.
Drumm McNaughton: But now we’ve got Gen A, which are the youngest coming out. They’re an AI generation. That’s scary in some ways.
Karen Panetta: It is because, you know, there’s the question now if educators are now faced with this, do I really need to teach them the foundational basics because now AI can like– they can just ask AI to get it. So, one of the things I’ve been promoting is it doesn’t matter what field you’re in, just like we all use computers today and word processors, doesn’t matter the discipline.
So everybody’s going to have to know how to utilize AI. And it doesn’t mean you have to know how to program up the deep, neural network to do it, but it also– but it does mean you need to be able to use it with caution, know how to use it intelligently. Even how to, how to prompt it, what we call prompt engineering.
The way you ask the question, you get– can get a different answer, or it can lead you to, things that you don’t expect. And so this is one of the things that we’re trying to utilize it, in schools to say, “Don’t try to avoid it.” And I use the parallel that when drones came out, everybody’s like, “Nope, drones, nobody should use drones.”
It’s going to be used for evil, and yes, it is, but at the same time, they’ve also had profound, good uses as well. So we’ve been trying to say, when they’re going to use it, and we want them to use it, we want them to learn how to use it responsibly. And so, and to clear it, and we also want them to understand that what is really your voice?
What is authentic you? So, and that’s why I think we’re worried is because we’re– everybody’s going to sound the same. We’re all going to be, averaged into what AI thinks is the norm. And communication will be different too.
Drumm McNaughton: Sure. Oh, totally.
[00:22:21] AI in Higher Ed Teaching
Drumm McNaughton: So with this, let’s dig further into the education piece. You’re the dean at the College of Engineering there at the School of Engineering.
Karen Panetta: The Graduate Dean. Graduates.
Drumm McNaughton: Graduate Dean.
How is AI changing how Tufts School of Engineering? How is it changing the teaching?
Karen Panetta: Yes. So we’re using it again for experimentation, helping us to explore and learn new concepts. I think being able to do, most of, I will say research-oriented institutions, the first thing we do is, “Oh, go do a literature search, find out what the latest and greatest is out there.” And, the days are long gone where you go sit in a library or, you search up something and then now you, and you have to read all the papers.
Now it summarizes it for you. So the question is, is the takeaway what you’re getting in the summaries of all this research enough for you to understand the fundamentals for you to advance and make innovate? So that leap is really what we’re focusing on is okay, you can use it to help get you information quickly and maybe do some preliminary analysis, but to get to the next level, you need a unique contribution.
And you can’t just say, “Hey, AI, give me a unique contribution or research area,” and I’ve seen people do it. It’ll come back and give you some suggestions, but I’m also wondering like how many other billions of people have you given that same suggestion to? And again, we know for entrepreneurship it’s not just about the good technology.
It’s got to have a great business case, right? It’s got to have a market space. So being really good, everybody just being really good at technology doesn’t mean that something’s going to lead to an impactful innovation. It just means you might have another widget, a toy that, doesn’t … Is a hot thing now, but doesn’t have any longevity and doesn’t leave lasting impact.
So we’re looking at that as well. But we’re really engaging our arts and sciences people because somebody who teaches English or history is, could be panicking like, “Oh my gosh, they’re going to replace me.” And it’s not about that. It’s more about, okay, well, what are the questions we’re not asking?
Because everybody’s just jumping on the AI bandwagon. What are we not asking that we should be? We need that population, we need those non-technical disciplines to chime in and say, “This is where AI fails. This is where you’re losing creativity,” so that we can be aware of it and be able to build in what I would say more smart hooks to protect us and also give us more explainability.
Drumm McNaughton: You brought up a point earlier about the ethical uses of AI. We’re still trying to figure all of that out. But at the same time, we’ve got to be able to make choices with how we use it. And you also said something that really struck me: there’s got to be a foundational knowledge. When I think about Bloom’s taxonomy, and of course, educators, we all know about Bloom’s if we’ve done any course development, et cetera, et cetera.
You can find out knowledge and comprehension. Those are– I don’t want to call them irrelevant nowadays, but in many respects with AI, it’s not the knowledge that’s critical. You’ve got to have foundational to be able to do critical thinking, to be able to do that analysis, that next level, to say, “This makes sense. No, that one’s out to lunch. I’m not even going to consider it.” Does that make sense?
Karen Panetta: It does, it does. You know, and I like to say we’re in the age what I call “Back to the Future technologies.” And, and I call it that because if you think of some of the innovations like we, we weren’t able to implement years ago because we didn’t have the computational power or the experimentation power to do it.
Now, stuff that was theoretical, we have new materials and computing and, that are nanomaterials and all these other things that allow us to realize theories that before were just well too, unreachable, unrealizable. So that’s a real leap, but you can’t make that leap unless you understand that fundamental theory, right?
And that’s why you have to know why it fails too. I think that one of the things, Drumm, that you’ve been talking about besides the ethical considerations is understanding failure. And that’s something that we want and stress in all our engineering programs, but we stress it across the whole learning environment is we want you to fail now early and often because when you get out there, we want to let you know how to deal with it so that you can get back up on your feet.
And I do a lot of outreach. I told you I start with them when they’re really young. I don’t, it doesn’t matter, from birth. And one of the first things that I see in all these fields is people are intimidated by it, and they say, I say they just want to quit. Engineering is not for me or math, STEM is not for me.
And I say, if you join a sports team and you lose the game, do you quit the sport?” No, right? You train harder, you, you try differently, you think strategically differently. And engineering is our sport, and that’s what we need to do, is we got to get our teammates all different levels of teammates to come together to understand that yes, we’ve got to learn this.
But if you’re afraid of it and you’re afraid it’s going to fail and, then you’re going to be left behind, and that’s not a good thing. And it’s not good for the economy, it’s not good for communities, and it’s certainly not good for children who are going to essentially need and live and breathe AI in the future.
Drumm McNaughton: Yes, I can remember my mother tutoring the next-door neighbor. She had a block on math, and my mother, my grandmother on my mother’s side was a math teacher, so obviously my mother understood how to do math and trying to train her to get her past the block. We’ve got folks in technology the same way.
Uh, my wife asks me frequently, “Can you help me with this? I know you showed me before,” but it’s like, “Sure. Why?” It, it’s a matter of teaching people, not giving them a fish, but teaching them how to fish.
Karen Panetta: Yes. It is. It is, and you have to… You can’t throw it at them all at once. You’ve got to, you know, incrementally—
Drumm McNaughton: That’s my problem.
Karen Panetta: —give them the wins. Yes. No. And I always start out, even with my outreach, when parents are saying, “How do I get my kids involved in math and sci- or not math and science, but in engineering?”
And I’ll say, “Well, what do they like?” And they say, “Well, they do like math and science.” I say, “Oh, really? So they’re doing math and science in their free time?” And the answer’s, “No, they like dancing. They like, this. They like sports.” And that’s the connection you need to make, is how can you use this in a field you love?
Like, think about dancing and human dynamics. Think of the choreographic, masterpieces that AI could actually help someone who understands how a body moves, to do that. And so I, that’s why I love it.
[00:28:56] Data Ownership and Private Models
Karen Panetta: Music, I think, is struggling because you see so many people, and AI actors, voiceover actors, there’s, huge controversy over, you’re replacing them with AI.
But you got to remember that AI was trained with somebody’s real voice. So you might be in there, and they might have your voice in there, but you’re not getting credit for it because they don’t have to prove that the AI used you during training.
Drumm McNaughton: And the challenge becomes who owns this data? From an intellectual property perspective, that’s a major issue right now.
Karen Panetta: It is, and it’s actually an asset for some of the things I’m doing on right now. I have datasets that I’ve created the first in the world. Some of them I share. Some of the databases I’ve shared are for facial recognition to prevent human trafficking. I’ve put out in the general public because I want everybody to be able to utilize that.
But I also have a dental database for training new dentists. But there’s other things that I don’t want out there because they could be used for harmful purposes. So understanding, what to share, what not to share, and giving access is really important. And I think that that’s the other problem too, is a lot of corporations don’t want to share, and that’s why IEEE comes into play because it’s like, if we all shared And we all conquered this problem together with AI, then we could make it more explainable, we could make it safer, and we could put in these hooks that I’m talking about to be able to, be able to see what’s going on or actually open its brain and see, the a snapshot of where it got its ideas from.
Drumm McNaughton: And I think that’s incredibly important. What I’ve heard a lot of institutions doing is they’re having their own AI database. They contract with an institution or a company and say, “Okay, this is fine. We want to keep it there.” These type of restrictive models, this is a way to be able to protect people’s data, is it not?
Karen Panetta: It is, and that’s why, for instance, at certain institutions you’re not allowed to use a certain large language model because they know that if I’m giving you my research data and I’m putting it out there on that large language model, that they’re collecting my data, and then they can go feed that to whatever research or whatever organization they want.
So there’s also, protections in there that say with a lot of organizations, sometimes we build it in-house because we want to know that the models we’ve built aren’t going out sharing it with other companies or other countries that, that don’t have the same sharing policies or that won’t protect the data.
And we also thinking about if especially, again, if we’ve got people in there and we’ve got human data in there, we definitely don’t want that going out. That’s going to be the, the most protected class.
Drumm McNaughton: Hmm.
[00:31:26] Advice for Higher Ed Institutions About Using AI and AI Data
Drumm McNaughton: So with that, if you had advice to give to an institution about using data, using AI, what would that be?
Karen Panetta: Don’t believe anything that says it has 100% accuracy. When somebody tells you, especially when somebody’s trying to give you a new product that’s going to expedite all your processes, ask about:
- How many test cases was this tested on?
- What, and what population?
- What of cases?
- What kind of people?
- What kind of scenarios?
And they should be able to provide you those numbers because the people get hung up on saying, “Accuracy. Oh, look at the accuracy of this and precision.” And that’s not telling the whole story, and I think that that’s sort of like, you get the sticker price or something at the top, and then you find out, you read the small print.
So that, that’s a big one. The second one is, don’t believe everything you see. Just because AI says it and just because it becomes in a headline, you’ve got to start thinking… And it’s unfortunate. We’ve got to start thinking of taking everything with a grain of salt of, okay, this is a headline, but where’s the data substantiated?
And you might say, “Well, they showed a video with it. They showed a real video.” But, and now we have to even question was it real? So, Yes, so that, that’s become a big problem. So just be cautious, and especially When you get anything that is asking you for private information or something, it’s just like the phone call.
I still hang up and I delete and I figure, you know what, if worse comes to worse and I deleted something that was real, they’ll figure out a way to get to me anyway legitimately. So, protect yourself and protect your children. I think that online now it’s even more, it’s more critical now than ever that we protect our youth because they are in electronics and we can’t possibly monitor everything that they’re being exposed to.
Drumm McNaughton: Yes. Those are great, great advice for people. This has been just fascinating for me, Karen. I’m, I am a geek at heart. What can I say?
Karen Panetta: I’m going to recruit you then. You should come back to come back and see some of the cool stuff I’m working on.
Drumm McNaughton: I would love to.
Karen Panetta: I’ll let you play with my technology toys.
Drumm McNaughton: Yes, but unfortunately I wouldn’t fit in Nerd Girls. I’d fit in, in just nerds, period.
Karen Panetta: Oh, I have Nerd Boys. We also have Nerd Boys. So
Drumm McNaughton: Oh, okay.
Karen Panetta: So you got to remember Nerd Girls was a program really talking about empowering students to make impact in humanity using their skills collectively. And the outcomes of that have changed curriculums for both boys and girls. So, Yes, thank you.
[00:34:00] Three Takeaways for University Presidents and Boards
Drumm McNaughton: So as we wrap up, as we always do, our two takeaway questions. Three takeaways for university presidents and boards. What do they need to know about AI and to make it usable and safe?
Karen Panetta: Don’t be afraid of it. Teach your students responsibly and dive right in, making sure that everybody, not just your engineers, are using it and learning it and understanding how to use it ethically and have those discussions. And we haven’t figured it out. They might actually figure it out. The other thing is making sure that your processes internally, even your selection processes, don’t utilize AI outputs to determine the future of your faculty’s careers or your students that you admit.
Drumm McNaughton: Thank you, those are great things. The human in the loop, you got to do it. What’s next for you?
Karen Panetta: I’m going to be working on some overhauling strategies for education. I work with governments to help them look at strategies for education to make sure that they can engage all their populations. And right now at IEEE, I’m working on an initiative to make sure that we can provide accessibility and education, the best educational products to anyone who wants to learn technology so that we can serve all communities, whether you’re a beginner, advanced, or whether you’re a super geek like me.
Drumm McNaughton: Well, that’s great. And when you get down that path, if you need some help, feel free to give me a shout.
Karen Panetta: I will. Thank you for having me, Drumm.
Drumm McNaughton: Karen, thank you. I’ve enjoyed our conversation. I look forward to hearing more about it in, a few months.
Karen Panetta: You will. You will. I promise. It’s not going away.
Drumm McNaughton: Absolutely. Take care.
Karen Panetta: Bye now.
Drumm McNaughton: Thanks for listening today, and a special thank you to my guest, Dr. Karen Panetta. Karen, thanks for a wonderful conversation. I thoroughly enjoyed hearing more about what you’re doing, especially in how you’re applying AI to multiple programs and areas, and I look forward to the next time our paths cross.
To my listeners, thanks again for tuning in. See you next week.
