Interested in this topic? Visit the links below for more resources.
Visit The IIA’s website or YouTube channel for related topics and more.
00:00:02 Speaker 1
The Institute of Internal Auditors presents all things internal audit tech in this episode, Charles King talks with Debbie Liu about the transformative role of artificial intelligence in internal auditing. They discussed the integration of AI tools like copilots, the importance of prompt writing.
00:00:22 Speaker 1
And how AI is enhancing audit processes? The conversation also covers training strategies, real world applications and the impact of AI on stakeholder engagements.
00:00:38 Speaker 2
Well, Debbie Lou, welcome to the All Things Internal Audit podcast. We're so happy to have you here today.
00:00:44 Speaker 3
Thank you, Charles, for inviting me.
00:00:46 Speaker 2
Let's tell the audience a little bit about your team at Kaiser Permanente. Talk to us a little bit about how big is the team, what are some of the big focus areas that your team is working on right now?
00:00:57 Speaker 3
So.
00:00:58 Speaker 3
I have a team of 82.
00:01:00 Speaker 3
People and we're divided by portfolios of risk. As you know, Kaiser is an integrated care model in the in healthcare. So we have one portfolio does dedicated towards our National Health plan, the payer side, finance compliance and then we have another team that is dedicated to the.
00:01:20 Speaker 3
Provider side the care delivery side and pharmacy. We also have a team that's dedicated to IT and enterprise shared services and then we have a team that's dedicated to things like HR community benefit because we are a non.
00:01:34 Speaker 3
Profit. We do have a tech enablement team. It's a small team, you know, I started that team and it's doing really well. We're just in that maturity development right now, but it's done a great job for being a small.
00:01:47 Speaker 2
Team I love that well, you know, as a as an AI leader and AI evangelist, I probably want to hear more about that tech enablement.
00:01:55 Speaker 2
Team than anything else we we met through an AI conversation.
00:02:00 Speaker 2
So let's talk about that a little bit at Kaiser. How do you think about using AI and really all technology analytics in your internal audit team? What does that look like for you and what are some of the Big North Star goals that you're trying to achieve?
00:02:15 Speaker 3
So we are a continuous improvement organization. So technology is very key to continuous improvement. We were reviewing the use of Gen. AI for the past year.
00:02:27 Speaker 3
And we joined an early adopters program last summer for 365 copilot. There was about four of us and I was one of the members of that early adopter team. And then when we rolled it out, we successfully integrated copilot into all phases of our audit, plus our quality assurance and.
00:02:47 Speaker 3
Quality improvement program. So we went through the entire eco system. The one thing that we've learned is prompt writing is an essential skill in being successful at Gen. AI, right? So what's a prompt?
00:03:02 Speaker 3
That prompt is instructions queues for directing Gen. AI, like copilot, to produce relevant information for you that you need. So what I did first in the journey of Gen. AI was trained the entire department.
00:03:22 Speaker 3
On prompt writing.
00:03:24 Speaker 3
Right, so prompt writing is not a unique skill to internal audit. It's not a unique skill to any organization or any industry. It's an essential skill that you'll need for the future. I would say for the future generation of auditors, this is a standard skill that you need to have as a matter of fact, I've thought about.
00:03:44 Speaker 3
Putting it into my job postings as a preferred requirement, effective prompt writing skills.
00:03:51 Speaker 3
Additionally, prop writing is crucial for guiding Gen. AI to produce high quality, relevant, and accurate responses to meet your needs. So that's why I went through that journey of training the department. We also have a prompt and copilot guidance.
00:04:10 Speaker 2
Manual. Oh wow, that's interesting.
00:04:11 Speaker 2
Tell me about that.
00:04:12 Speaker 3
So it's just I guidance manual on how to write prompts like I'm an experienced auditor. You know it has all the pieces and who's your audience you're writing to?
00:04:24 Speaker 3
We also have a library of prompts. We have a crowdsourcing platform. It's called prompt buddy, right? So we have prompts for each phase of the audit cycle, because why reinvent the prompt? If it's a standard prompt for risk assessment or to develop your risk and control matrix so people don't have to work.
00:04:44 Speaker 3
Extra heart uh. We actually have an AI landing page in internal audit, which has all things AI. So if you want training your PowerPoint decks there, if you want guidance, there's the manual. We are also in the process of developing agents.
00:05:02 Speaker 3
So we started to develop. We've completed developing an agent for writing an effective audit finding, you know with the components of issue criteria, cost impact, recommendation. So you just run this agent which is full of prompts, right, and makes it easier and each each prompt.
00:05:22 Speaker 3
For each category has several questions or prompts to make that issue much clear.
00:05:30 Speaker 2
Excellent. I love that you know, one of the things that you learned really early on with prompt engineering is that it's very difficult to write a perfect prompt that's going to get a perfect output. And some of the best prompts that I've seen in my clients are prompts that start a dialogue with the user and and exactly what you're saying. They ask questions, the user responds, and then when the AI has everything it needs from the user.
00:05:52 Speaker 2
Then it can produce its output, and it's a very different approach to prompt engineering than trying to put everything in one initial prompt. It sounds like you've done a lot for communication and training.
00:06:05 Speaker 2
How have you seen adoption? Because it's it can be hard to get people to change the way they work and use these tools and what have you. What have you found to be effective and and maybe what's not effective. If you tried some things that?
00:06:17 Speaker 3
Didn't work well. I really believe as a leader you walk to talk. So as I mentioned earlier, I was in that.
00:06:25 Speaker 3
Early adoption program so I can learn about it and see the capabilities and the possibilities and how I can dream using it. And then when we did the training, I was the person that was actually demoing the scenarios so they can say, oh, Debbie can.
00:06:40 Speaker 3
Learn it. I can learn it. And also I use it daily. So when I review auto reports, everyone knows I'm using copilot and sometimes some of my review knows is use copilot for this or did you use copilot for that? So everyone is adopting because they see their leaders using it and my leaders are.
00:07:00 Speaker 3
Using it, we also have every Friday office hours.
00:07:04 Speaker 3
So we have 1/2 hour every Friday in which you can come to the office hours, bring your questions about say I tried to write this prompt, I'm having problems or you wrote a fantastic prompt and you want to share the rest of the department. So the office hours is an opportunity to not only solve problems, but to upskill you. Right? That's where.
00:07:24 Speaker 3
We teach about, like the last office session was about writing a.
00:07:29 Speaker 3
That's right. And uh, when I saw you in your presentation talk about applets, it gave me an idea of, you know, what I'd love to write an applet where, you know, in the report writing cycle, I can send that applet to our business user, our business executive, to guide them in writing a management.
00:07:50 Speaker 3
Response or proactive action plan.
00:07:52 Speaker 3
Like step by step, what does a good corrective action plan look like, and does it address all the risks in the finding and that would make it fun and easier for them, right? I I'm thinking of the user experience. So when you think of use cases for your prompts, don't only think about your ecosystem within internal audit. You know what makes your audit process effective.
00:08:13 Speaker 3
Where are your touch points like when you have new people joining internal audit? I'm thinking about using it for onboarding.
00:08:19 Speaker 3
For education, for wherever we touch our clients. So looking at that and analyzing it, what can we use Gen. I?
00:08:29 Speaker 2
Yeah, I think that's such a great idea. I mean, not just not just walking the talk. I mean, that is that is so important, I think to drive adoption group, whether it's AI or anything else. But I think this idea of thinking about, hey, AI, how you can not just make your teams experience better but make the other stakeholders your auditees your audit clients more engaged in using AI and that's going to make ultimately that's gonna make your management.
00:08:52 Speaker 2
Action plans better that's going to reduce the amount of iteration and back and forth.
00:08:57 Speaker 2
With.
00:08:58 Speaker 2
And it makes you look so tech forward, right, you you use internal audit as this kind of beacon of innovation and you can show others what some of the possibilities are. I absolutely love it.
00:09:10 Speaker 3
I've also shared with my audit committee chair to use how we're using AI in internal audit. He had me demo.
00:09:18 Speaker 3
Now we're using.
00:09:19 Speaker 3
AI and turn audit to the audit committee, who share that with the board and it cascaded through the executives at Kaiser. And I've been doing a roadshow demoing how we use Gen. AI in internal audit with various executives at Kaiser and encouraging them that is not that difficult. Prop writings, like learning to write a language and the language is.
00:09:40 Speaker 3
English.
00:09:41 Speaker 3
Right, there are some nuances.
00:09:42 Speaker 2
Yeah.
00:09:43 Speaker 3
To write effective prompts, but it's not difficult. It's just a matter of doing it every day practicing and you'll get better and better at it, and it learns your voice too. So when I write emails, it doesn't sound as artificial, it's sounding more and more like me, yes.
00:09:59 Speaker 2
Like you, that's so great. And you know, we talk in internal audits so often about how to get a seat at the table.
00:10:00 Speaker 3
Yeah.
00:10:06 Speaker 2
And we think about how to become relevant and this is such a great example of you leading the organization from a technology standpoint and really showing everyone else at the executive level and beyond how AI can change the way you operate. And I have to think that's an inspiration for for others in the organization. Yeah, yeah.
00:10:25 Speaker 2
Well, Debbie, this has been a great conversation. I'm so glad we got a chance to talk today. And thank you again for coming on all things.
00:10:31 Speaker 3
Internal audit. Yeah. And thank you for letting me share and hopefully I'm encouraging others to just saying.
00:10:39 Speaker 1
Hey, honored Pros ready to supercharger skills and connect with the best in the field? You absolutely need to check out the CIA's 2025 international conference happening July 14th through 16th in Toronto. This is your chance to learn about emerging risks cutting edge tech.
00:10:58 Speaker 1
And global best practices that will elevate your internal audit game don't get left behind. Register now at iaaic.org.
00:11:10 Speaker 1
If you like this podcast, please subscribe and rate us. You can subscribe wherever you get your podcasts. You can also catch other episodes on YouTube or at the iaa.org that's theia.org.
