Interested in this topic? Visit the links below for more resources:
Visit The IIA’s website or YouTube channel for related topics and more.
00:00:02 The IIA
The Institute of Internal Auditors presents All Things Internal Audit Tech.
00:00:07 The IIA
In this companion episode to Internal Auditor Magazine's The Next Gen.
00:00:11 The IIA
Audit Mentor article, Marco Horvat sits down with Sarah Kuhn to explore how she built an AI-powered audit mentor to support auditor onboarding, training, and professional development.
00:00:23 The IIA
Sarah shares how a simple idea to scale knowledge sharing evolved into an agentic AI tool that
00:00:30 The IIA
helps auditors learn, ask better questions, and navigate their careers.
00:00:34 The IIA
The conversation covers practical lessons for implementing AI, managing guardrails, fostering adoption, and ensuring AI enhances rather than replaces auditor judgment.
00:00:48 Marko Horvat
With me, I have Sarah Kuhn.
00:00:50 Marko Horvat
Sarah, you want to talk a little bit about yourself?
00:00:53 Sarah Kuhn
Thanks for having me on today.
00:00:54 Sarah Kuhn
I have been an internal auditor since 2003.
00:00:58 Sarah Kuhn
So this is, I guess, I lucked into the profession early while I was still in school at the University of Tulsa and been my passion ever since and been doing internal audit a couple of different companies and learning along technology, teaching myself technology as I go.
00:01:15 Sarah Kuhn
But I would consider myself to be more of a business auditor, not necessarily an IT or OT auditor.
00:01:22 Sarah Kuhn
So hopefully
00:01:23 Sarah Kuhn
can take from this, whether you think you have technical skills or not, know that you can do whatever you put your mind to here.
00:01:31 Marko Horvat
There you go.
00:01:31 Marko Horvat
And the thing we're going to put our mind to with regards to this discussion is building an audit mentor with agentic AI, which is a pretty spicy topic, especially the agentic AI piece.
00:01:43 Marko Horvat
And the reason why we're speaking to Sarah about this is she put together a pretty interesting and I would say novel
00:01:52 Marko Horvat
approach to using AI in a way that really matters to the organization.
00:01:57 Marko Horvat
One of the things that I found is that a lot of people struggle with AI.
00:02:01 Marko Horvat
It seems to be almost like a solution looking for a problem in a lot of instances.
00:02:06 Marko Horvat
And a lot of organizations are deploying it without sort of clear direction with that background.
00:02:11 Marko Horvat
I don't know if that specifically outlines the situation you were in, Sarah, but
00:02:16 Marko Horvat
With all of that, you merged with something actually useful that you were able to scale your teams.
00:02:21 Marko Horvat
You want to talk a little bit about what is AuditMentor and how did it start and what gave you the idea to sort of explore that direction?
00:02:30 Sarah Kuhn
So you're absolutely right.
00:02:31 Sarah Kuhn
I think that at the time when all this was taking place,
00:02:35 Sarah Kuhn
AI was everywhere and was making its way in the company and everyone was saying, we need use cases and how are we gonna use this?
00:02:42 Sarah Kuhn
And we were already starting to use AI from an LLM perspective, what I would call like the Google of AI, right?
00:02:49 Sarah Kuhn
You're just asking questions of things.
00:02:51 Sarah Kuhn
And then we see this new and cool thing come on the AI agents.
00:02:56 Sarah Kuhn
And I'm like, I don't really understand what an agent is.
00:02:59 Sarah Kuhn
How is that different than what I was already doing?
00:03:01 Sarah Kuhn
And so started kind of understanding, okay, an agent is just
00:03:05 Sarah Kuhn
just something that it's not doing exactly what you tell it to do.
00:03:09 Sarah Kuhn
You're giving it a goal or an outcome.
00:03:12 Sarah Kuhn
And then it's kind of deciding how to respond to you.
00:03:16 Sarah Kuhn
It's not execute steps A through Z.
00:03:18 Sarah Kuhn
It's I want you to end at Z, but you decide how to get there.
00:03:23 Sarah Kuhn
I think as many great innovations happen is it came from a little bit of boredom, if you will.
00:03:29 Sarah Kuhn
So
00:03:30 Sarah Kuhn
We work every other Friday in our office, and so we don't usually have meetings on Fridays.
00:03:34 Sarah Kuhn
And so usually on Fridays, I dedicate time to do what I call big thinking.
00:03:40 Sarah Kuhn
if I'm not getting interruptions, sometimes I just let myself sit and get bored so that I can have an idea.
00:03:45 Sarah Kuhn
And I was trying to think of what are some ways that we could use an agent, but also I just wanted to play around with it and see what it could do and how it was different.
00:03:55 Sarah Kuhn
And at the same time, part of my job responsibility was in
00:04:00 Sarah Kuhn
training and onboarding and making sure that as a department, we're following the professional standards.
00:04:06 Sarah Kuhn
And I thought, I feel like I am the department Google, so maybe I can build an agent to replace myself.
00:04:12 Sarah Kuhn
And I know that not everyone likes to replace themselves in their jobs, but that was kind of the spark for it.
00:04:17 Sarah Kuhn
I'm very busy and I don't always have time to be Google.
00:04:20 Sarah Kuhn
So how can I help people help themselves?
00:04:22 Sarah Kuhn
So that was kind of where it all started, if you will.
00:04:28 Marko Horvat
That's kind of the interesting thing and the scary thing about agentic AI, right?
00:04:32 Marko Horvat
It's that whole, it's agentic because it has agency.
00:04:35 Marko Horvat
But the cool thing about it though, is that thinking with the goal in mind kind of forces you to put yourself in that mindset, right?
00:04:43 Marko Horvat
In terms of what is the problem that you're trying to solve?
00:04:48 Marko Horvat
And in your case, it was sort of a institutional problem and a personal problem around how do I scale Sarah?
00:04:55 Marko Horvat
Can you kind of walk us through how did that evolve from just like a pet project into something that kind of expanded throughout your organization?
00:05:06 Marko Horvat
What challenges or opportunities you had around that?
00:05:09 Marko Horvat
Because I can't imagine it was something as simple as like,
00:05:12 Marko Horvat
Monday morning, you're like, hey guys, look what I built.
00:05:13 Marko Horvat
Like, let's all just get to it immediately.
00:05:16 Sarah Kuhn
Honestly, it was way easier than I thought it was going to be.
00:05:18 Sarah Kuhn
But let me caveat that.
00:05:20 Sarah Kuhn
So I started out with a very simple set of instructions for it, because like I said, I was looking at other kind of pre-built agents that existed, template agents that I was kind of reading through like how they were built, just like clicking around in the settings and reading some of the information.
00:05:35 Sarah Kuhn
So I had a general idea for what type of things it would need.
00:05:40 Sarah Kuhn
And because I already had a
00:05:41 Sarah Kuhn
training library of a lot of examples and a methodology and tools and other documents that I could feed it.
00:05:51 Sarah Kuhn
That made it actually very simple.
00:05:52 Sarah Kuhn
So it really only took me a few hours to write the basic things in and then link it to the sources that I wanted it to kind of the guardrails to lock it in on.
00:06:03 Sarah Kuhn
These are the things that I want you to do.
00:06:05 Sarah Kuhn
So
00:06:06 Sarah Kuhn
It really didn't take long in the initial, like, does it work?
00:06:10 Sarah Kuhn
Took like a couple of hours.
00:06:11 Sarah Kuhn
But I wanted to go tell everyone about it.
00:06:14 Sarah Kuhn
But I also knew I needed it to actually work before I got people excited.
00:06:18 Sarah Kuhn
And so I didn't tell anyone for a couple of weeks that I had done it.
00:06:23 Sarah Kuhn
Every time someone came to me and asked me the question, I asked the mentor the question.
00:06:28 Sarah Kuhn
And I wanted to see what kind of responses it was giving me and making sure that
00:06:34 Sarah Kuhn
it would answer the way I wanted it to answer.
00:06:36 Sarah Kuhn
And it was never wrong, but when it first started, it was giving me the answer.
00:06:42 Sarah Kuhn
And I was like, I want you to be a mentor.
00:06:45 Sarah Kuhn
I don't want you to just give them the answer.
00:06:47 Sarah Kuhn
I don't want you to feed them the fish.
00:06:48 Sarah Kuhn
I want you to teach them to fish, right?
00:06:50 Sarah Kuhn
So don't just tell me the answer and where you found it, but why that's the answer.
00:06:54 Sarah Kuhn
And the other question they didn't ask that they should have asked, because that's a lot of, you know, when you're onboarding or have new auditors or new leads, they know a question
00:07:04 Sarah Kuhn
to ask, but they may not know the next three questions to ask.
00:07:07 Sarah Kuhn
And so it took a bit of time for me to get comfortable that it was getting more information.
00:07:12 Sarah Kuhn
So there was a bit of iteration in the design and the prompting in the background of really telling the agent how to respond.
00:07:20 Sarah Kuhn
And once I started to feel more comfortable, then I had a few of our other core professional auditors who had experience.
00:07:28 Sarah Kuhn
I gave them the link to the agent so that they could start using it, give me some feedback.
00:07:34 Sarah Kuhn
and make sure that it was giving them the same type of information.
00:07:38 Sarah Kuhn
And the interesting thing about this agent is that it can see your profile so it knows what your job title is and knows about you.
00:07:47 Sarah Kuhn
and answers based on that context with the specified information that was provided.
00:07:53 Sarah Kuhn
So it's taking all of that information and saying, okay, these are the things I have to choose from to answer the question, but I'm also putting myself in your shoes.
00:08:01 Sarah Kuhn
So if you are acting as a lead auditor versus a staff auditor, you might have to specifically tell it that in that circumstance.
00:08:10 Sarah Kuhn
Otherwise, it goes based on your job title and your conversation history and what it already knows about you.
00:08:15 Sarah Kuhn
But yeah, it took some time after we trialed
00:08:17 Sarah Kuhn
out with them.
00:08:18 Sarah Kuhn
It took a few more weeks before I rolled it out to more people.
00:08:22 Sarah Kuhn
And then it was kind of, growing from there.
00:08:25 Sarah Kuhn
But it's kind of one of those that the longer it took, I was getting excited, but also like, is it going to get the old news by the time everyone gets to use it and see it?
00:08:35 Sarah Kuhn
That was not the case.
00:08:36 Sarah Kuhn
But I will say that I think one of the biggest wins out of all of this was just getting people to start using it and stop coming to me or to others.
00:08:44 Sarah Kuhn
Like they use it now.
00:08:45 Sarah Kuhn
They go there first.
00:08:46 Sarah Kuhn
So that to me is a win.
00:08:47 Sarah Kuhn
win that says they like it and they trust it.
00:08:51 The IIA
And just like that, one auditor built it.
00:08:53 The IIA
The whole team got better because of it.
00:08:56 The IIA
That's what developing people looks like now.
00:08:59 The IIA
That's the idea behind IIA membership.
00:09:02 The IIA
We've expanded what's included so your team has access to trusted standards, practical expertise, continuous learning, and a global community helping every auditor stay ahead as AI and the profession evolve.
00:09:16 The IIA
Learn about everything included at the iia.org and lead what's next.
00:09:21 The IIA
Now let's get back to Marco and Sarah on where to start in creating an AI mentor.
00:09:28 Marko Horvat
And then being able to scale that to the organization.
00:09:30 Marko Horvat
And I think this is really, what's really interesting about this use case is that you kind of did both.
00:09:36 Marko Horvat
There's that organizational aspect to it, and then there's also that liberating aspect in terms of how it affects your own personal workflow.
00:09:45 Marko Horvat
And I think that's really key takeaways in terms of if you wanted to build something like this, that's a good framework to start.
00:09:52 Marko Horvat
And the other piece of it too is, and I want to get your perspective on this, is like as you rolled this out, were people super impressed or were they like, oh, I can just type that into ChatGPT and it'll tell me the same thing.
00:10:05 Marko Horvat
Or, you know, like how is this any different than just me using Copilot, right?
00:10:09 Marko Horvat
So
00:10:10 Marko Horvat
Was there any of that sort of convincing that you had to do around, no, this is something special that you guys really should use instead of just typing into Copilot.
00:10:21 Sarah Kuhn
It was absolutely both sides of that coin.
00:10:23 Sarah Kuhn
So there was the group of people who are, I'm going to call them the early adopters, right?
00:10:28 Sarah Kuhn
They have been on copilot since day one and they were already running up against the hallucinations.
00:10:34 Sarah Kuhn
And so it was a hurdle for them to trust the agent.
00:10:38 Sarah Kuhn
And I had to explain to them how the agent was boxed in.
00:10:41 Sarah Kuhn
Like it can't just go out to the web and make things up.
00:10:44 Sarah Kuhn
Like it has to cite its source.
00:10:46 Sarah Kuhn
It can only pull from these materials.
00:10:48 Sarah Kuhn
There's A hierarchy to the way that it answers because
00:10:51 Sarah Kuhn
for me, that was a big thing is it needs to be consistent if we're having all of our staff use this.
00:10:56 Sarah Kuhn
And by the way, most of our staff are rotational.
00:10:59 Sarah Kuhn
They're not career auditors.
00:11:00 Sarah Kuhn
They're coming into the business, into the department for a couple of years and then rotating back out into the business.
00:11:06 Sarah Kuhn
So I needed us to be confident that they were getting good information, but also them to be confident that what it was telling them was correct.
00:11:13 Sarah Kuhn
So I had that side of it, of the early adopters saying, are you sure it's not going to hallucinate and gaining that trust for them.
00:11:21 Sarah Kuhn
And then
00:11:22 Sarah Kuhn
I had the others who had not really had dipped their toe in the water yet with LLMs or ChatGPT, Copilot, whatever it might be.
00:11:31 Sarah Kuhn
They hadn't really figured out, they didn't really know how they were using it yet.
00:11:34 Sarah Kuhn
And so from that side of things, that was more just getting them used to going here and asking the question.
00:11:42 Sarah Kuhn
But yes, once they asked the question and realized, because I had pre-built prompts that were specific to their work, right, they were part of the audit process.
00:11:52 Sarah Kuhn
And so it made it very easy for them that made that barrier of entry.
00:11:56 Sarah Kuhn
They didn't have to create a prompt.
00:11:57 Sarah Kuhn
They didn't have to come up with what to ask.
00:12:00 Sarah Kuhn
I told them, just act like this is your mentor.
00:12:02 Sarah Kuhn
Like if your mentor is busy right now and you can't call them to ask this quick five minute thing, just ask the mentor the same way you would ask the person and see what it says.
00:12:12 Sarah Kuhn
And yes, they were absolutely floored and impressed.
00:12:16 Sarah Kuhn
But I will say the novelty wore off fairly quickly.
00:12:19 Sarah Kuhn
One of the biggest feedback that they gave me was
00:12:21 Sarah Kuhn
The answers are too long.
00:12:23 Sarah Kuhn
But of course, I had told it to give long information because I wanted it to teach them.
00:12:28 Sarah Kuhn
So I, again, took that feedback and I had it chunk it out in a specified format so at least they could consistently know where to go within the response to find what they needed to find.
00:12:39 Sarah Kuhn
So you can lead a horse to water, but you can't make them drink, right?
00:12:42 Sarah Kuhn
So I can tell them how to learn how to do it and not just the answer, but they might only be taking the answer.
00:12:48 Sarah Kuhn
I don't know.
00:12:48 Sarah Kuhn
But yeah, so I think it was a bit of both.
00:12:51 Sarah Kuhn
both like people very impressed, but also people very skeptical.
00:12:54 Sarah Kuhn
And what?
00:12:55 Sarah Kuhn
We need skepticism as auditors.
00:12:57 Sarah Kuhn
So bring it on, try and break it.
00:12:58 Sarah Kuhn
And that's exactly what I told him is try and make it find the faults.
00:13:02 Sarah Kuhn
Where is it not working?
00:13:03 Marko Horvat
I think like you calling this the audit mentor, like that's a very intentional choice of words, right?
00:13:09 Marko Horvat
You didn't just pick it because it sounded neat or it looked cool on the screen.
00:13:13 Marko Horvat
And that rises
00:13:15 Marko Horvat
kind of maybe a touchy question, because I think we as a profession have been struggling with mentorship probably before then, but definitely since COVID, where this whole notion, like as people start working remotely and as workflows become more digital, what does mentorship actually mean?
00:13:36 Marko Horvat
And then, how does that mean in terms of the traditional relationship of a human to human sort of resource?
00:13:44 Marko Horvat
So a couple of things I want to ask is like, why were you intentional about that mentorship and that difference between getting the answer and understanding the answer, which is so critically important for our roles within organizations?
00:14:01 Marko Horvat
And if you could explain just how you stumbled upon this mentorship, why mentorship is the right thing versus just like the audit answer bot?
00:14:10 Sarah Kuhn
A couple of things.
00:14:11 Sarah Kuhn
Now, the first was
00:14:13 Sarah Kuhn
I had been telling people to use Copilot as like their intern or this is your teammate, but you're still the lead auditor and you have to review their work or whatever, So I had already been kind of using this analogy that this AI before even agentic AI was, yes, it can do all these great things for you, but it's still the, you know, learning.
00:14:34 Sarah Kuhn
And so you have to review the work of your team member.
00:14:37 Sarah Kuhn
And so I already had that kind of like, I want them to treat it like a person.
00:14:41 Sarah Kuhn
And so to me, mentor was
00:14:43 Sarah Kuhn
at least in the category of person.
00:14:45 Sarah Kuhn
But I think the other thing for us was because we were doing more remote work, because our department is actually spread out geographically among different states.
00:14:54 Sarah Kuhn
And you don't always feel comfortable when you're new to the department reaching out to someone over Teams or Slack or chat to you don't know if you're interrupting them.
00:15:04 Sarah Kuhn
You may not have ever met this person in person before.
00:15:07 Sarah Kuhn
You've only met them over a video conference or an e-mail.
00:15:10 Sarah Kuhn
And so
00:15:12 Sarah Kuhn
I wanted that to be something that if your assigned person is not free or you feel like this is probably a stupid question, you can ask the stupid question to someone that's not going to judge you.
00:15:23 Sarah Kuhn
And then you can follow that up with a conversation to your mentor or to your lead or to whoever it is and just practice the conversation before you have the conversation if you have to.
00:15:35 Sarah Kuhn
And I just want to caveat to say that yes, calling it a mentor, but also this in no way, shape or form
00:15:42 Sarah Kuhn
I was very clear with our senior auditors and our audit leaders to say, this does not replace supervisor responsibilities.
00:15:49 Sarah Kuhn
This does not replace like you are so responsible to train people, but let this fill in the gaps, right?
00:15:55 Sarah Kuhn
And let this make it easier because it can break down the jargon because especially for someone like myself who is a career auditor,
00:16:04 Sarah Kuhn
Sometimes I don't even realize I'm using jargon anymore, and so it's helpful to have the mentor who can do that, who can break it down and say, This is what they actually are saying when they say that.
00:16:14 Marko Horvat
I've been in those meetings where somebody throws out an...
00:16:17 Marko Horvat
and you just write it down and you're like your little notebook and you're like, find out what these letters mean, right?
00:16:24 Sarah Kuhn
Yeah.
00:16:25 Sarah Kuhn
Follow up later.
00:16:26 Sarah Kuhn
Yeah.
00:16:27 Marko Horvat
I think that's really interesting too, because I think that is what a good human mentor does.
00:16:32 Marko Horvat
It's that like that openness and approachability.
00:16:35 Marko Horvat
And one of the things we found
00:16:38 Marko Horvat
we work with a bunch of clients around AI strategy and implementation, especially on the learning side.
00:16:43 Marko Horvat
But one of the things we find a lot is that when you have a digital agent that people talk to, they just open up like crazy and they end up having conversations and sometimes going deeper
00:16:58 Marko Horvat
than they otherwise would with a human being, right?
00:17:01 Marko Horvat
That is a frame of reference.
00:17:02 Marko Horvat
Did anything happen with your audit mentor where maybe it was like some question that you didn't expect or some facet of the job that maybe you'd taken for granted that kind of surprised you and how people were using it?
00:17:15 Sarah Kuhn
Yes, I mean, that's a great question.
00:17:17 Sarah Kuhn
And it was one of the things that I had not originally planned for, but became clear to me that as feedback, I kept asking for feedback from everyone the first while that we were using it to make sure it was working the way we intended.
00:17:30 Sarah Kuhn
And one of the things that kept coming up was, I wish that it could tell me more about my career progression or what skills I should have in this place.
00:17:39 Sarah Kuhn
And so we had, back when the IIA had published the kind of
00:17:44 Sarah Kuhn
professional skills and tied to your advanced or intermediate or expert and what that skill looks like in each of those things.
00:17:52 Sarah Kuhn
We had taken that and kind of put it into the language of our company.
00:17:56 Sarah Kuhn
So using the words of our department and our company, so this is what that shows up here in our department, this is how that looks.
00:18:03 Sarah Kuhn
And so I ended up loading that as another source to the mentor along with some of the other just corporate leadership skill kind of information.
00:18:14 Sarah Kuhn
And so that became a whole nother Ave.
00:18:16 Sarah Kuhn
because people were seeking that.
00:18:18 Sarah Kuhn
They got so comfortable, like you said, with it, that they were asking more questions and trying to figure out like, how do I know when I'm ready to be the lead auditor?
00:18:26 Sarah Kuhn
How do I know that I'm doing a good job as the lead auditor?
00:18:29 Sarah Kuhn
And it didn't really have a way to tell them that answer.
00:18:33 Sarah Kuhn
And so it would usually just say, well, here's the standard that applies, but it couldn't really answer the question.
00:18:39 Sarah Kuhn
And so then after giving that information, now it had a new set of reference tools that it could then
00:18:45 Sarah Kuhn
suggest additional learning plans, because it was suggesting learning plans for how to audit.
00:18:50 Sarah Kuhn
It wasn't necessarily suggesting learning plans for growing in your career as an auditor.
00:18:54 Sarah Kuhn
And now it has another tool set that it can actually provide and set out actual development plans.
00:19:01 Marko Horvat
And that's interesting too, because that can be such a specific use case, right?
00:19:07 Marko Horvat
Because I think, and I don't want to get too stereotypical, right?
00:19:10 Marko Horvat
But I think part of the makeup of auditors, there's this weird sort of tension
00:19:14 Marko Horvat
where on the one hand, you need to have a lot of confidence, right?
00:19:17 Marko Horvat
Like you need to, if like you're gonna come in and you're gonna look at something and you're gonna give your opinion that, hey, this is right, this is wrong, this is gray, and here's where it's leaning on gray.
00:19:25 Marko Horvat
Like there has to be like a certain amount of confidence that you have to have with the job.
00:19:30 Marko Horvat
But on the flip side of that, like the best auditors are like the biggest skeptics, right?
00:19:35 Marko Horvat
And sometimes that skepticism translates back into your own ability.
00:19:40 Marko Horvat
So you get this weird mix of like,
00:19:42 Marko Horvat
people that are hyper competent, but really just like feeling like this other like feeling of inadequacy.
00:19:50 Marko Horvat
So you get, so that I love that use case where it's like, how do I level up?
00:19:54 Marko Horvat
And it's helpful for maybe the auditor that's a little overconfident in their ability.
00:20:00 Marko Horvat
And also on the flip side of it, the auditors maybe a little too skeptical of their ability, because I've seen both, right?
00:20:06 Marko Horvat
Like you're ready.
00:20:06 Marko Horvat
No, I don't think I'm ready.
00:20:08 Marko Horvat
Like, no, you're ready.
00:20:09 Marko Horvat
Like leave the nest, go spread your wings.
00:20:11 Sarah Kuhn
Yeah.
00:20:12 Sarah Kuhn
And even having first time leads or even experienced leads to be able to, you know, they know they need to give feedback to their teammate, right?
00:20:20 Sarah Kuhn
Like that's always the hardest part, right?
00:20:21 Sarah Kuhn
Is that you're giving coaching notes or feedback to someone who's
00:20:25 Sarah Kuhn
Probably technically your peer, they don't actually report to you.
00:20:28 Sarah Kuhn
And so that can be really difficult for some people to be able to do that.
00:20:32 Sarah Kuhn
They're not a supervisor for a reason, right?
00:20:35 Sarah Kuhn
And so that was another.
00:20:37 Sarah Kuhn
way that people started using the mentor was coaching, how do I have this conversation?
00:20:43 Sarah Kuhn
Or how do I say this in a way that is still clear enough for them to make the change that needs to be made, but not in a way that is going to be hurting their feelings or things like that.
00:20:53 Sarah Kuhn
So that has been another use case that I had never even dreamed of, but people now will take their work papers, put it in, ask it to do a quality assurance review based on our methodology, based on the standards, whatever it might be.
00:21:07 Sarah Kuhn
give it the audit objective and then it will give you some feedback on how you did with your work paper before you even submit it for the first review.
00:21:15 Sarah Kuhn
And so that also is giving people confidence, right, to say,
00:21:19 Sarah Kuhn
Yeah, I do know what I'm doing.
00:21:20 Sarah Kuhn
And I will also caveat to say that our agent has specific instructions not to be overly flattery or provide a lot of flattery to audit, because we all know that these agents, they love to compliment you.
00:21:34 Sarah Kuhn
So we did give it specific language to keep a neutral tone and fact-based and not do all of those things, but it can still critique.
00:21:43 Sarah Kuhn
It can still give you positive and negative, but in a way that is easy to hear.
00:21:48 Marko Horvat
It's just amazing because it's like you built this thing as a resource to provide practical career-based advice, for lack of a better term.
00:21:59 Marko Horvat
And now it's sort of, there's all these use cases for it, right?
00:22:02 Marko Horvat
Like, so you use it for onboarding because it helps people get up to speed really, really quickly to ask those dumb questions in private to get that one-on-one training.
00:22:10 Marko Horvat
People are using it to like review documents.
00:22:13 Marko Horvat
So an interesting thing that I've sort of been tracking
00:22:17 Marko Horvat
And I'm sure you've probably seen some of this stuff too.
00:22:20 Marko Horvat
There was a report that was issued by MIT this year where they talked about that people that use AI a lot tend to sort of offload a lot of their cognitive skills to the AI.
00:22:32 Marko Horvat
And you get issues like in that issues, but like instances where someone has like this amazing work product, but they like never write a word of it, right?
00:22:41 Marko Horvat
So like you're asking questions about it, they never did that.
00:22:44 Marko Horvat
So how do you ensure that
00:22:47 Marko Horvat
people don't do that.
00:22:48 Marko Horvat
And is there any sort of design tips or anything like that you can give to someone to make sure that this is really like an additive tool as opposed to a replacement tool?
00:22:59 Sarah Kuhn
I agree.
00:23:00 Sarah Kuhn
I think that also worries me.
00:23:01 Sarah Kuhn
I also, I feel like that one of the dangers is that because people are using it, how do you get the skills if you never have the opportunity to get the skills and all of those things.
00:23:12 Sarah Kuhn
So I was intentional in the design in that, again, I didn't want it to just give the answer.
00:23:17 Sarah Kuhn
I wanted it to tell you why that's the answer and how, and not just, for example, like if you were saying, what type of control is this?
00:23:25 Sarah Kuhn
It's a detective control.
00:23:26 Sarah Kuhn
Okay, well, don't just tell me detective control.
00:23:28 Sarah Kuhn
Tell me all the types of
00:23:29 Sarah Kuhn
have been and why you chose detective versus one of the others, right?
00:23:33 Sarah Kuhn
So example like that, so that it's not just, I hope that now in the future you recognize there are multiple types of controls and why, what this one would have been and why that is the answer.
00:23:44 Sarah Kuhn
So from that perspective, I hope that I've built that in to try to teach people and make them curious about more questions to ask more questions.
00:23:54 Sarah Kuhn
But the other piece as far as, you know, how do you keep people from just
00:24:00 Sarah Kuhn
passing off Copilot work product or ChatGPT work product as their own.
00:24:05 Sarah Kuhn
And that really is just reminding everyone that your name is the one that signs off on this.
00:24:11 Sarah Kuhn
And whatever it does, you have to
00:24:15 Sarah Kuhn
be confident that, and that's also why I wanted it.
00:24:20 Sarah Kuhn
cannot give an opinion.
00:24:21 Sarah Kuhn
It cannot like tell you that this passes or this fails or anything like that.
00:24:26 Sarah Kuhn
It has to remain neutral.
00:24:28 Sarah Kuhn
One of the things that we, as we used the agent and learned how people were using it, we started to narrow it back in a little bit to take away some of what it could do and then ask that people, we actually then created a prompt library for outside of the agent
00:24:45 Sarah Kuhn
for people to use instead of using the audit mentor for some of those things that they were doing to help perform their work instead of just learning and said, okay, this is more for learning.
00:24:56 Sarah Kuhn
This over here, this prompt library is something that you can use as a way to like help you create your work.
00:25:03 Sarah Kuhn
And then there's a whole nother set of guardrails and things around using a prompt library to help scale and skill and check and make sure that we're using the best prompt and the prompt is going to be giving similar responses to everyone, et cetera.
00:25:18 Sarah Kuhn
But again, trying to get over that barrier of I don't know how to write a prompt and also
00:25:24 Sarah Kuhn
putting in some of the guardrails around the prompts to help make sure that the decision still rests with the auditor.
00:25:31 Sarah Kuhn
And it's not giving you enough that you can just copy and paste, like you still have to insert yourself into the process to get the answer, but is definitely making your work still easier and accelerating the pace.
00:25:43 Marko Horvat
So I love, I think there's so many just good lessons learned.
00:25:47 Marko Horvat
I highly recommend that people
00:25:50 Marko Horvat
take this as part of their learning from this experience and then the article that you wrote, in terms of, you started with a clear problem that was not only personal to you, but also the organization, right?
00:26:04 Marko Horvat
So there's like a clear reason to do it.
00:26:08 Marko Horvat
And there's also that what's in it for me piece that's really, really important when you're asking people to address their workflows.
00:26:14 Marko Horvat
And then just the intentionality behind it, where you could have made this thing this big virtual all-purpose audit robot, but you didn't.
00:26:26 Marko Horvat
And I think having that intentionality around it, tackling the problem, having the what's in it for me and incorporating that into a tool that enhances the user is amazing.
00:26:39 Marko Horvat
But let's do 2 questions.
00:26:40 Marko Horvat
First one, knowing what now, if you had to start completely from scratch, like so you show up tomorrow and there's been some kind of catastrophic event, you have to build this thing from scratch, what would you do different?
00:26:53 Sarah Kuhn
I would still do the iterations in starting and adding and things, but I do think where I've landed now is
00:27:00 Sarah Kuhn
letting it be that narrower focus.
00:27:02 Sarah Kuhn
I think that we started down that path and then I think that it was gonna get confused and muddy.
00:27:08 Sarah Kuhn
I feel like if you're asking it to do too much, like right, you can't be an expert at everything.
00:27:12 Sarah Kuhn
You can only be an expert in the topics that we're providing for you.
00:27:17 Sarah Kuhn
So I do think having that understanding from the beginning
00:27:22 Sarah Kuhn
or earlier maybe in the process would have been helpful starting over today to keep that clear focus.
00:27:27 Sarah Kuhn
Like this is the intention of what this is being built for.
00:27:31 Sarah Kuhn
We started from kind of that general place and then we started to try and add everything in and then we kind of backed off of that.
00:27:39 Sarah Kuhn
And I think that was a smart move.
00:27:40 Sarah Kuhn
I wish we would have done it maybe a little bit sooner to back off of some of that, but it was good learning and took us to a new place.
00:27:47 Sarah Kuhn
But yeah, definitely looking back, that was the right call to do that.
00:27:51 Sarah Kuhn
And
00:27:51 Sarah Kuhn
I think it also gives more opportunity because that was another thing, and I don't want to take us down a totally different path here, but once people saw this and they were like, oh, that seems so easy and I want to build one, right?
00:28:03 Sarah Kuhn
Like then, and you don't want to discourage people from building more tools, right?
00:28:08 Sarah Kuhn
And so if you can make them a little more focused, that also then opens the opportunity, the development opportunity for others to try to develop something.
00:28:16 Sarah Kuhn
And if you can be a little bit more coordinated across, you know, okay, you build something
00:28:21 Sarah Kuhn
you do this.
00:28:23 Sarah Kuhn
Now, does that get difficult if you have 500 things to manage versus one?
00:28:27 Sarah Kuhn
Of course, so there's always a balance with everything.
00:28:29 Sarah Kuhn
But circling back to the original question, I think starting, if I had to start over again today, I do think I would
00:28:36 Sarah Kuhn
lock it into, is this a mentor?
00:28:40 Sarah Kuhn
Is this a coach?
00:28:41 Sarah Kuhn
Is this training?
00:28:42 Sarah Kuhn
Is this specifically for training?
00:28:44 Sarah Kuhn
Is this really more a mentor?
00:28:45 Sarah Kuhn
And maybe split those a little further apart.
00:28:48 Marko Horvat
I love that, specificity intentionality, I think is a really good advice.
00:28:52 Marko Horvat
And looking forward, right?
00:28:55 Marko Horvat
So now that we look back, if you were doing things differently, looking forward, where do you see just the greatest opportunity, right?
00:29:01 Marko Horvat
And I'm not gonna let you like pick the easy,
00:29:04 Marko Horvat
things will happen quicker, right?
00:29:07 Marko Horvat
But what's your hot take in terms of where the greatest opportunity for a genetic AI is going to be an internal audit over the next few years?
00:29:17 Sarah Kuhn
Yeah, I do think what it does really well still is the research.
00:29:23 Sarah Kuhn
So it can find things that we wouldn't even know to look for.
00:29:27 Sarah Kuhn
So I think that
00:29:29 Sarah Kuhn
that ability to do deep research, a lot of the models that are being put out, that you can choose your agentic to use this model or use this model, use Quad or Anthropic, and they all have their benefits.
00:29:40 Sarah Kuhn
But the research capabilities, I think, are where we're going to get a lot of benefit.
00:29:45 Sarah Kuhn
Because as auditors, right, thinking back to the beginning of my career, we knew that we had this span of invoices from #1 to #1,000.
00:29:55 Sarah Kuhn
And somewhere in that thousand, we were going to select a sample to test.
00:29:58 Sarah Kuhn
And then
00:29:59 Sarah Kuhn
came out of that, then we might have found something.
00:30:02 Sarah Kuhn
And then fast forward a decade or so, and all of a sudden now we have data analytics and we can actually look at that full population of invoices.
00:30:09 Sarah Kuhn
And we know, okay, well, we don't even need to look at this because we can already see everything we need to see.
00:30:14 Sarah Kuhn
We can test 100%.
00:30:16 Sarah Kuhn
And now even with agentic AI or even AI in general,
00:30:22 Sarah Kuhn
Now we can ask a question in our planning beyond just the data to see patterns, to find information that we may not have the research piece of it.
00:30:33 Sarah Kuhn
The question we didn't even know to ask is now coming to us that we can ask a better question when we're talking to our clients.
00:30:40 Sarah Kuhn
So I still feel like that hasn't fully been fleshed out and developed yet.
00:30:45 Sarah Kuhn
And I think that that's where we're headed next is
00:30:49 Sarah Kuhn
Being able to sift through more information, not just the data, not just the easy to find data, but how do you just ask the question and it is telling you more than you would have even known to ask about?
00:31:03 Marko Horvat
Right, with the goal not just being a faster audit or like doing more, it's about like having a higher quality
00:31:12 Marko Horvat
sort of outcome to the whole thing so we can think deeper, go a little bit deeper.
00:31:17 Marko Horvat
I love that.
00:31:18 Marko Horvat
Well, that's great.
00:31:19 Marko Horvat
So start with a problem, be specific, focus in on the solution, and just try to end up better than where you started.
00:31:29 Marko Horvat
I think those are pretty basic goals that everyone should have.
00:31:32 Marko Horvat
And I think you executed it beautifully.
00:31:34 Marko Horvat
And
00:31:35 Marko Horvat
I just wanted to thank you for your time.
00:31:38 Marko Horvat
This is amazing and had a great conversation with you.
00:31:41 Marko Horvat
I look forward to the next thing you're going to build.
00:31:43 Sarah Kuhn
Thank you.
00:31:44 Sarah Kuhn
I appreciate you taking the time to talk.
00:31:47 Sarah Kuhn
I get very excited to talk about these kinds of things.
00:31:49 Sarah Kuhn
And when something works well for me, I don't want to gatekeep that.
00:31:52 Sarah Kuhn
I want to share it with everyone.
00:31:53 Sarah Kuhn
So I hope that others can find that problem statement, like you said, and maybe not feel as scared to take that on.
00:31:59 Sarah Kuhn
Like at the beginning, I said, I'm not a technical person and I'll just throw out there, AI can help you build the AI.
00:32:04 Sarah Kuhn
So
00:32:06 Sarah Kuhn
Don't be afraid to ask it how to do it.
00:32:08 Marko Horvat
Well, thank it's again, Sarah.
00:32:10 The IIA
If you like this podcast, please subscribe and rate us.
00:32:13 The IIA
You can subscribe wherever you get your podcasts.
00:32:16 The IIA
You can also catch other episodes on YouTube or at the IIA.org.
00:32:20 The IIA
That's T-H-E-I-A.org.
