ATIA - AI Upskilling - Nancy and Bryant V3 MP3.mp3

Transcript

00:00:01 Speaker 1

The Institute of Internal Auditors presents All Things Internal Audit Tech.

00:00:06 Speaker 1

In this companion episode to the global best practices, Internal Audit Upskilling for Critical AI Capabilities, Bryant Richards talks with Nancy Humm about how internal audit teams can build AI skills with intention, confidence, and discipline.

00:00:21 Speaker 1

Humm shares how her department has moved from broad AI awareness

00:00:25 Speaker 1

to practical, immersive learning that helps auditors both use AI in their work and audit it effectively.

00:00:31 Speaker 1

The conversation covers mindset, skill set, and tool set, along with real-world use cases, adoption challenges, and human oversight.

00:00:43 Speaker 2

Nancy, can you tell me how has your team's approach to learning and developing AI skills evolved over the past few years?

00:00:50 Speaker 3

Our company has been using AI and generator AI for the last few years.

00:00:54 Speaker 3

And so the question we had was, how do we get people prepared so that when AI is in production, we are confident in knowing how we audit it.

00:01:05 Speaker 3

But also equally, we are part of our company's strategy to also be committed to using AI as well.

00:01:12 Speaker 3

So I would say our journey has been where we wanted to make sure we create a baseline awareness

00:01:19 Speaker 3

across our audit department to really understand the benefits of AI, but equally important the risk of AI.

00:01:26 Speaker 3

So we have provided very broad general awareness training to set the foundation the last two years.

00:01:34 Speaker 3

And we are pretty much in the process where that has shifted from broad awareness learning to more practical immersive learning so that it is relevant learning in terms of what we need to know to use AI in our audit work, but equally

00:01:49 Speaker 3

important, we also need to know enough to audit it with confidence.

00:01:53 Speaker 3

So that's been our shift in evolution in terms of department learning and making sure that we are teaching people the right things to use in, but also be able to audit it as well.

00:02:05 Speaker 2

I love that answer.

00:02:06 Speaker 2

You've given me a lot to unpack.

00:02:07 Speaker 2

I'm going to explore a few things if you don't mind.

00:02:11 Speaker 2

When we talk about this, I hear you very importantly saying, we're using it,

00:02:17 Speaker 2

but we're also learning how to use it because we need to audit it.

00:02:20 Speaker 2

So we need to understand the risk side.

00:02:21 Speaker 2

And in order to understand the risk side, we have to kind of have a hands-on approach.

00:02:25 Speaker 2

Can you talk a little specifically about that and maybe some examples, if you can?

00:02:30 Speaker 3

Sure.

00:02:30 Speaker 3

Let me also take a step back to share with you even more specific on how we approach this.

00:02:35 Speaker 3

So I like to say that the recipe for success to

00:02:39 Speaker 3

Upskill in this space for us has been where it's important to think about this in the three-legged stool, like a three-pronged approach.

00:02:48 Speaker 3

So auditors do need the right mindset, the right skill set, and then the right tool set.

00:02:54 Speaker 3

So we like to think that those three parts are equally important to be successful and strong adopters of AI, but also equally important to be, you know, aware with experience and usage to audit it with confidence.

00:03:09 Speaker 3

So I would say those are the three ways we focus to help the individual auditor get the right learning and mindset to embrace and lean in to some of the things that I think, you know, when people hear about tools, it is intimidating, especially for business auditors in some cases.

00:03:26 Speaker 3

But I think we are promoting the right level of thinking, right level of curiosity through healthy exchange and learning so that when we give people the tools,

00:03:37 Speaker 3

They also have the skills through the formal training I kind of described earlier, but also making sure that people are really openly embracing, you know, what's provided to them.

00:03:48 Speaker 3

And I think at the department level, we provide a lot of ways to create and foster sharing of actual ways people are using AI in the audit workflow.

00:04:01 Speaker 3

And I think people are able to do that because we've actually intentionally offered AI bootcamps,

00:04:07 Speaker 3

Where, for example, we obviously as an organization, and like many organizations, offered Copilot licenses, M365 Copilot licenses.

00:04:17 Speaker 3

So people now have the tools, and when they use it, we're identifying case studies where people have used these tools.

00:04:24 Speaker 3

in a practical way to solve some of the audit tasks across the workflow.

00:04:29 Speaker 3

And it's through that sharing that I think is really fostering a very healthy level of confidence that if they see so-and-so do it, why can't I do it?

00:04:38 Speaker 3

And so it's really inspiring and motivated people as well to lean in through that type of natural healthiness sharing.

00:04:44 Speaker 2

That's terrific, Nancy.

00:04:45 Speaker 2

I'm interested in how your organization is leading this as well.

00:04:49 Speaker 2

Am I right in saying that your team, sort of the IT

00:04:53 Speaker 2

data analytics type of team is the leader within the group and you're the ones who are sort of rolling this out in some fashion.

00:04:59 Speaker 2

And if not, could you help me understand kind of how you're going about it?

00:05:02 Speaker 2

Because it sounds like there's a heavy leadership role here.

00:05:05 Speaker 3

Yeah, and I think that's a great question.

00:05:07 Speaker 3

And what's unique about me and my team is that we are uniquely positioned to really be laser focused to drive the enablement efforts, to your point.

00:05:17 Speaker 3

Because if there's not enough focus around the learning, the showing,

00:05:21 Speaker 3

I think if you leave it to chance and organic adoption, I don't think departments will really essentially see the speed and the learning curve and the accelerated adoption that's to be expected.

00:05:33 Speaker 3

So myself and our team, and through some working group formats that we've established where we engage other auditors, we are kind of co-creating what that learning roadmap will look like for our department.

00:05:46 Speaker 3

We are also offering learning series throughout the whole year where we have a very set clear schedule on topics that we're gonna learn about new tools that are emerging across enterprise, but also certain sessions, we are also spotlighting certain auditors that can share their case studies too.

00:06:03 Speaker 3

And so it's a really heavily curated, I would say dynamic immersive learning throughout the year combined with structured formal learning that is required.

00:06:12 Speaker 3

And we look at that curriculum, you know, quite frequently to make sure it stays relevant.

00:06:17 Speaker 3

And then, you know, the last part I would say is we also measure the usage too.

00:06:21 Speaker 3

So there's a level of discipline around, great, people should experiment, people now have the tools to use it, and we are sharing, you know, the success stories.

00:06:30 Speaker 3

The question is, is our audit outcome and the quality of work better?

00:06:35 Speaker 3

So we are formulating some clear measures, not just on usage, but on outcomes too, which again, really kind of, to me, closes the loop on making sure that we're seeing the true expected value and benefits, right?

00:06:47 Speaker 3

And to me, that's the part that should not be, you know, forgotten in terms of the why, why are we doing this?

00:06:54 Speaker 2

Yeah, critical.

00:06:55 Speaker 2

And I'm glad you jumped into the outcomes because why are we doing this?

00:06:58 Speaker 2

And ideally there's...

00:07:00 Speaker 2

Every organization has its reasons, but audit quality has to be part of it, right?

00:07:03 Speaker 2

There definitely has to be a quality component to it.

00:07:06 Speaker 2

It's interesting, as I've spoken with a lot of folks in the field, when they start talking about AI, there's a conversation that sneaks up a little bit that starts talking about quality and how AI is doing things, particularly for younger staff that may not have the judgment and experience yet to really know how good AI is or isn't.

00:07:28 Speaker 2

How is your organization, if you don't mind, sort of perceiving the risk of thinking you know what's going on because you're using AI, but not necessarily adopting the expertise or skills that are needed for the role, particularly in those early positions?

00:07:42 Speaker 3

I would say that it's important for many of us in audit departments to really have clarity on what is the spectrum of audit task that we believe AI can play a role in that's really going to deliver the quality and value, to your point, Brian.

00:07:56 Speaker 3

So what we've done here is that for our team, we've established many of those tasks.

00:08:03 Speaker 3

Uh, so that it sets a bit of that clarity on what type of tasks that we believe all auditors, irrespective of their domain expertise, should be using Copilot or, you know, other AI tools for.

00:08:15 Speaker 3

And I think that really creates consistency so that people don't end up using AI for certain edge cases that we don't know.

00:08:24 Speaker 3

And so I think that allows us to also measure it, you

00:08:27 Speaker 3

you know, with clarity against that menu of audit tasks.

00:08:30 Speaker 3

But to your point about junior auditors, I think there is still a very much a human in the loop aspect to how auditors are using AI in the audit workflow.

00:08:40 Speaker 3

I don't think that we're in the stage where we have delegated the whole workflow to AI at this point.

00:08:47 Speaker 3

And so I'm sure at some point in the future that could be the case in terms of how much it can be augmented with AI, but right now,

00:08:56 Speaker 3

Even though, let's say a junior auditor is performing an audit task like drafting an issue, right, which is a very common one, I don't think junior auditors are blindly trusting the output.

00:09:07 Speaker 3

I think they would review the draft issue from Copilot and still make sure that it's the best draft issue that their manager also needs to kind of review.

00:09:17 Speaker 3

So to me, I think that judgment and human in the loop and oversight is still necessary.

00:09:23 Speaker 3

However, I think we do need to unlock where, at some point, where are we comfortable to really delegate some of the decision-making to AI to really see the full value of how the auto workflow is delivered, right, with an AI-first type of ability?

00:09:39 Speaker 2

Yeah, so as we're talking about AI and we're talking maybe more tool-focused, what types of tools have you deployed

00:09:48 Speaker 2

What types of have been successful and what impacts are you seeing with any of them?

00:09:54 Speaker 3

I think the common one, as I've already mentioned, is definitely M365 Copilot.

00:09:58 Speaker 3

I think many departments already have this.

00:10:01 Speaker 3

I think there's some specialized tools that we use that allows us to do more analysis with unstructured data.

00:10:08 Speaker 3

And those are very specific niche vendor types of solution that we've deployed, which I think definitely is another level beyond what I think Copilot is able to

00:10:17 Speaker 3

deliver.

00:10:18 Speaker 3

So we've seen some really great success on a vendor tool that does that.

00:10:23 Speaker 3

And I think, you know, in terms of another common one, I think that enterprises would have is the Power Platform, suite of Power Apps too.

00:10:32 Speaker 3

So we also are able to leverage a lot of those capabilities to do some automation and workflow automation as well to streamline how some of these tasks can be done, you know, more end to end.

00:10:46 Speaker 3

And I don't think having a lot of tools is really the recipe for success, Brian, in my mind.

00:10:51 Speaker 3

I think it's about having the right tools that can really help us do the right task is really where I think people will see truly the benefit of where AI can be, you know, incorporated.

00:11:04 Speaker 3

And I think a strong adopter really understands that workflow.

00:11:07 Speaker 3

Like what are the key steps to really get an audit done from end to end with speed, with better quality,

00:11:13 Speaker 3

And obviously achieving better efficiency too, right, from productivity.

00:11:16 Speaker 3

So I think that's kind of our viewpoint.

00:11:19 Speaker 3

It's less about the tool, which is why I say tool set is really the last.

00:11:22 Speaker 3

It's always going to be the mindset and then the skills that you need to talk to AI to ultimately unlock, you know, what benefits it provides.

00:11:32 Speaker 2

So Nancy, this sets up an interesting question here in that you've had some success.

00:11:36 Speaker 2

It sounds like you've had some success and you're on your journey for sure, which is better than many organizations already.

00:11:42 Speaker 2

When we talk about AI, we think of the LLMs, we think about typing things into Copilot.

00:11:46 Speaker 2

There are workflows and stuff.

00:11:48 Speaker 2

Where are your biggest success?

00:11:50 Speaker 2

Are they at the LLM level where you're shaping

00:11:53 Speaker 2

sort of task behavior or is that at the automation level where you may have actually automated some compliance audits or really heavy workflows like a user access testing or something to that effect?

00:12:05 Speaker 3

I would say that the biggest value has been just a range across all the different audit tasks.

00:12:10 Speaker 3

So I think it's just really looking at the combination of tasks and where we maximize the auditor's ability to see, again, leveraging this menu of tasks and say, this is where all these tasks should be done and performed using AI as my copilot, right, or as my assistants.

00:12:27 Speaker 3

Because that's the level of augmentation we want to see mature over time, Brian, for us to then start measuring, is it a better outcome?

00:12:35 Speaker 3

Are the quality of the work better?

00:12:36 Speaker 3

Are we finding better insights?

00:12:38 Speaker 3

Are we doing the work at a greater speed?

00:12:40 Speaker 3

Is our stakeholders saying that we're actually providing better audit assurance and insights too?

00:12:46 Speaker 3

So I think there's pockets of automation where we've seen, but it's very pointed, right?

00:12:51 Speaker 3

More domain specific.

00:12:52 Speaker 3

So you're not really going to see the scale, I would say.

00:12:55 Speaker 3

The scale to us is really the whole end-to-end audit workflow, where every auditor across the department can truly use AI and a system across a chain of those tasks.

00:13:05 Speaker 3

And hopefully every auditor can do that consistently, right, to really maximize what the benefits are going to be.

00:13:11 Speaker 3

That's where we're really seeing the scale and invalid delivery.

00:13:15 Speaker 2

Okay.

00:13:16 Speaker 2

So that takes us really to skill set as you've been focusing on here.

00:13:19 Speaker 2

So from a skill set standpoint,

00:13:21 Speaker 2

I mean, you've shared with us a nice plan of things that have gone well in how you've built skills.

00:13:26 Speaker 2

What particular skills, how would you describe what you're building amongst this team of internal auditors?

00:13:33 Speaker 2

And what's been the most successful or impactful?

00:13:37 Speaker 3

I would say that the skills, and this is a common question that guests get asked frequently is, to me, it's, yes, you do need some level of technical understanding of some of the model, like in different models that exist.

00:13:51 Speaker 3

But I think a lot of that is not as visible in order that's necessary, I would say.

00:13:56 Speaker 3

I think the skills will be more of the things about how do you talk to AI?

00:14:01 Speaker 3

We often say communication is really paramount in everything we do, which I still believe in that.

00:14:07 Speaker 3

But it really elevates, even more importantly, it's not just talking to people.

00:14:12 Speaker 3

How do you talk to AI?

00:14:14 Speaker 3

There's this prompt engineering, and I think even that alone is evolving because apparently, two years ago, you have to structure your prompts, as we know, very clearly to give it context and clear instructions.

00:14:26 Speaker 3

But for those that have used more of the more recent AI tools, you don't need that formal structure as much as you did before.

00:14:34 Speaker 3

But you still need to give it the right goals and objective.

00:14:37 Speaker 3

So to me, communication and how do you talk to AI is going to be really, really crucial so that

00:14:43 Speaker 3

you're going to get the best results from these tools.

00:14:47 Speaker 3

And secondly is delegation.

00:14:49 Speaker 3

I don't think this is something that we often talk about, but how do we comfortably delegate the work to AI?

00:14:55 Speaker 3

To me is so important.

00:14:58 Speaker 3

And that's not something that most people embrace because we value the expertise we have.

00:15:03 Speaker 3

We like doing the work.

00:15:05 Speaker 3

But part of the real value is where are the right things to delegate to AI?

00:15:09 Speaker 3

That to me is a skill that I would like to see auditors strengthen more.

00:15:14 Speaker 3

And then lastly, I think it's validation.

00:15:16 Speaker 3

It is something that I think is innate to auditors anyway, where we have this level of skepticism and questioning, like, can I trust this?

00:15:24 Speaker 3

But to me, that also then gets even more important now with the AI output.

00:15:29 Speaker 3

I think you kind of mentioned some junior auditors, like, no one should be blindly trusting the AI output now.

00:15:35 Speaker 3

So what is the level of validation that you

00:15:39 Speaker 3

need to perform.

00:15:40 Speaker 3

And there's gonna be a spectrum of the two depending on the task and the consequence of the output that's being used for certain purposes.

00:15:47 Speaker 3

So, you know, in a nutshell, I think to me, the three key critical skills to be successful in using AI to drive better outcomes is communication, delegation, and validation.

00:15:58 Speaker 3

So really the task, or the talk, the task, and then the trust, like the three Ts, you know, a good another way I would say it.

00:16:06 Speaker 2

Love it.

00:16:07 Speaker 2

So can you give us an example then?

00:16:09 Speaker 2

You've been training some folks in your organization where you've seen that go very well and maybe a specific case where some folks have picked up on this and what have they done with it and what outcomes came from that?

00:16:21 Speaker 3

Yeah, I can spotlight a really...

00:16:24 Speaker 3

use case where it's one of those use case that cuts across the organization.

00:16:29 Speaker 3

And many auditors actually contributed to this use case too as we partner with even the second line in the risk and compliance where when you think about organization, how they manage risk and mitigate risk, obviously as they're evaluating the risks and the controls that exist to mitigate those risks, sometimes those controls don't always function and therefore what do we have?

00:16:50 Speaker 3

We have issues.

00:16:52 Speaker 3

And there's this idea of issues that are self-identified as well as issues identified by audit.

00:16:57 Speaker 3

And so where we saw the opportunity with AI is obviously AI can now draft content, create content for you.

00:17:05 Speaker 3

And so issue and drafting and writing issues is actually a real pain point.

00:17:10 Speaker 3

point in many organizations, including ours.

00:17:13 Speaker 3

The consistency in how you write an issue with clarity that meets what was the cause of this issue?

00:17:19 Speaker 3

What's the impact in terms of consequence, right?

00:17:21 Speaker 3

So there's a specific framework that every organization have to make sure there's a clear issue statement written to drive the right remediation steps.

00:17:30 Speaker 3

So what we did is we actually partnered even with IT to come up with a enterprise solution

00:17:36 Speaker 3

that leverages generative AI capability in a chatbot format that actually converses with the auditor or converses with compliance and risk, or even people in the business.

00:17:50 Speaker 3

And everyone goes to the same chatbot in order to explore, help me write a clear issue.

00:17:58 Speaker 3

And so we've deployed this for over a year and a half now since some really positive results.

00:18:03 Speaker 3

And the question is, why does this matter, right?

00:18:05 Speaker 3

It matters because if you don't have a clear issue written, you may not be fixing the right problem.

00:18:12 Speaker 3

And that actually costs, you know, you add from a compliance, non-compliance, or even just putting the resources to work on the wrong things, right, from operational rest and cost.

00:18:24 Speaker 3

So I think the outcome that we've been trying to measure is what's the quality of these issues?

00:18:30 Speaker 3

And we've been able to actually measure a positive trend in terms of issues across the company, but issues that audit also creates.

00:18:38 Speaker 3

What's the quality score overall to kind of justify, you know, the existence of a solution like this?

00:18:44 Speaker 3

So happy to assure that, you know, this AI chatbot is doing what it's

00:18:49 Speaker 3

needs to do to drive the, awareness of writing the right issues, but also making sure that it's driving the right remediation and that we can have an active pulse over the quality of these issues to then ultimately have better insights, right, when these issues are written well across the companies.

00:19:07 Speaker 3

So

00:19:08 Speaker 3

I feel like we're just in the start of it, Brian, and we see endless possibilities in what this foundation can actually take us into the near future as a company.

00:19:18 Speaker 3

So really exciting use case that Audit was able to drive in partnership and collaboration with other teams across the organization.

00:19:27 Speaker 2

So it's great to hear your successes.

00:19:29 Speaker 2

I'm glad you've started.

00:19:30 Speaker 2

You've been on this.

00:19:31 Speaker 2

You've got wins.

00:19:32 Speaker 2

Good for you.

00:19:33 Speaker 2

Lots of folks in industry are having challenges as well.

00:19:36 Speaker 2

Would you care to share what I'm sure are a fair number of challenges, maybe the most impactful one about upskilling your internal audit team?

00:19:45 Speaker 3

Sure.

00:19:46 Speaker 3

I think the biggest challenge is really keeping up with the pace, quite frankly.

00:19:52 Speaker 3

As I talked about mindset, skill set, and tool set, I think the variable here in terms of change is really the tool set.

00:19:59 Speaker 3

So as much as everyone's going to be eager to embrace this and ready to use it, I would say that some of the tools continue to change so rapidly that the tools we used last year, and even the issue chatbot that I described to you and shared as an example,

00:20:13 Speaker 3

Like we already know that there's better tools out there today that can do what that chatbot can do a year and a half ago.

00:20:19 Speaker 3

And now you're kind of in this position is like, do we pivot to this better tool and rebuild what we had deployed a year and a half ago?

00:20:28 Speaker 3

And so this constant reevaluation and knowing that the options are endless is a real challenge because the traditional technology solutions we deployed in the past, like they can have lasting where they can be in production for many years.

00:20:42 Speaker 3

But now, some of these AI tools, you could be changing them in months.

00:20:47 Speaker 3

And I think that gets us quite uncomfortable, I would say, in terms of keeping up with the change of the tool set.

00:20:53 Speaker 3

Because in the end, you need a bit of vestability, but you also know that better features are going to come out that will be cheaper, faster, and all of the above, too.

00:21:03 Speaker 3

So that's something that's been hard to keep up.

00:21:07 Speaker 3

But I think the other challenge, too, is the learning part is,

00:21:11 Speaker 3

Maintaining a relevant learning curriculum and curing relevant timely topics dynamically throughout the year for Otter is something that I think requires dedication and focus.

00:21:23 Speaker 3

So, you know, I don't think that's a capacity that most teams have the luxury to offer, but I would strongly advise that if, you know, auto shops do want to see a stronger AI adoption, really invest the time to develop

00:21:39 Speaker 3

the necessary learning and make it required for your people.

00:21:43 Speaker 3

But do it in a way that's encouraging and positive.

00:21:46 Speaker 3

So this is not like a mandate that you shall, or if not, then something happens to you.

00:21:52 Speaker 3

But do it in a very positive, encouraging way that the learning that people are going to take is an investment to their career to remain relevant for the future of how audit work needs to be delivered in the next few years.

00:22:04 Speaker 3

So that would be the second thing I would say in terms of making the time to invest and

00:22:09 Speaker 3

having a strong learning plan, that's a real challenge if you don't make that investment up front, because you leave it up to chance that your people are going to make the time on the weekends to learn these things.

00:22:19 Speaker 3

And most people will not have that time.

00:22:21 Speaker 2

So you're going to have folks in your organization who have adopted this more quickly than others.

00:22:26 Speaker 2

You're going to have some folks who might be resisting or not resisting.

00:22:29 Speaker 2

Can you tell me a little bit about what the differences in your organization are between those who are adopting more quickly and more successfully?

00:22:38 Speaker 2

than the others.

00:22:39 Speaker 3

There are definitely super users or strong adopters to your point.

00:22:43 Speaker 3

And that's to be expected because you always have a bit of that curve where, you know, people are going to jump right in without being told.

00:22:51 Speaker 3

And you're going to have the strong middle that, you know, through healthy encouragement, they come along.

00:22:56 Speaker 3

But then you're going to have a bit of the laggers right behind.

00:23:00 Speaker 3

And I like to say that

00:23:02 Speaker 3

I think it's, it's a healthy curve.

00:23:04 Speaker 3

It's what we expect.

00:23:05 Speaker 3

And some of the forums I share with you where we foster, you know, showcasing case studies and where people have used it successfully, but I think we also temper it with the fact that, hey, there's some real use cases where we say it was challenging.

00:23:20 Speaker 3

Uh, and, and so when we bring some of those laggers to those forums, I think.

00:23:25 Speaker 3

it allows them to kind of hear both sides that this is not all rainbows and sunshine.

00:23:31 Speaker 3

We like to see more of that, of course, but it's healthy that it could be a bit intimidating and sometimes it is not always easy to use, but we shouldn't give up.

00:23:40 Speaker 3

And so I think it's through those forums that hopefully we'll convert some of the laggers.

00:23:46 Speaker 3

But I also want to say that we have also a very

00:23:49 Speaker 3

strong commitment and tone at the top and our leadership in our department that's very committed to foster the learning, support the learning, but also accountable to demonstrate through their team that there is going to be a strong adoption in the right places and the right task and be able to kind of tell the story on the outcomes.

00:24:10 Speaker 3

Has this

00:24:11 Speaker 3

made the work easier to deliver, and has the quality improved when we use AI?

00:24:16 Speaker 3

So every leader is accountable to kind of really showcase that story over time on that progress.

00:24:22 Speaker 3

So I would say there's a bit of a top-down, bottoms-up approach that we're doing that I think is helping really move the needle so that we can really move a lot of the laggards to the middle to be adopters in that sense.

00:24:35 Speaker 2

So it sounds like a sort of a classic implementation playbook, right?

00:24:39 Speaker 2

An organizational change,

00:24:41 Speaker 2

make sure to educate, make sure to bring people along at the right pace, keep it positive in a lot of ways, tone at the top.

00:24:46 Speaker 2

All this sounds very, very sort of textbook right things to do.

00:24:50 Speaker 2

At the same time, AI is a different, it's different.

00:24:54 Speaker 2

It's just so different, Nancy.

00:24:56 Speaker 2

And your questions, your answers to these questions have been terrific.

00:25:00 Speaker 2

I'm really curious as we explore this, and if I can push you down, you know, the years a little bit where AI is so good at what it does,

00:25:09 Speaker 2

that it's very easy to assume it's right.

00:25:13 Speaker 2

How are we or how are you planning on keeping your team to maintain its skepticism, to maintain accountability that not just the human in the loop, but the true judgment of what we do?

00:25:24 Speaker 2

What is being built in from a skill set standpoint in your training or just what do you think should happen to make sure that we don't fundamentally just turn over this profession to the AI at some point, even if.

00:25:36 Speaker 3

It's a great point.

00:25:37 Speaker 3

And I feel like at least for us, we are in now this phase where we do need to invest more on thinking through more thoughtfully on exactly what you just said is how do we not end up in a place where people just kind of trust, you know, it is what it is and we need to get the work done.

00:25:58 Speaker 3

And we do think that AI is

00:26:01 Speaker 3

doing a pretty decent job, right?

00:26:02 Speaker 3

And so how do we avoid potential errors in the output we consume that will have a bit of a consequence?

00:26:09 Speaker 3

I do think that's why I kind of keep going back to that, like, we have to be clear on those tasks that we believe AI is doing right well.

00:26:20 Speaker 3

And I think through, you know, feedback loop and also storytelling, I think it gives us some good insight in terms of,

00:26:28 Speaker 3

where our auditors are feeling less comfortable on some of the output for certain tasks that they are applied or experimenting.

00:26:37 Speaker 3

And to me, I don't think there's a civil bullet ultimately in terms of like, hey, where can we trust somewhere we can't trust?

00:26:44 Speaker 3

The only way we're gonna have to find out is we're gonna have to dip our toes into the water and really actively get feedback and stories from people on terms of like, okay,

00:26:55 Speaker 3

Where it has not worked, what's the lesson there?

00:26:59 Speaker 3

And could we make it work, right?

00:27:01 Speaker 3

Because in the end, this is kind of the tension between we want to be users.

00:27:06 Speaker 3

And I think there's commitment to do that because there's really no big downside, I would say, because there is a lot of potential.

00:27:14 Speaker 3

There's probably more risk to not using AI in the auto work because in the end, I think the enterprise expects everyone to kind of get aligned.

00:27:22 Speaker 3

And I think we just have to be diligent on knowing where are we comfortable to play at and where do we comfortable get over time to stretch and push the boundaries.

00:27:34 Speaker 3

And I feel like as leaders of the audit profession, that's a comfort level that I think discomfort level that we need to get a little bit more comfortable to be frank.

00:27:44 Speaker 3

And I think for us, while it does sound like a bit of a textbook playbook in terms of adopting, you know, new technology,

00:27:52 Speaker 3

I think the difference for us is we're so laser focused.

00:27:56 Speaker 3

We have people dedicated to drive enablement, and we have people dedicated to make sure that we have a way to measure this.

00:28:03 Speaker 3

And to me, just like any strategy, Brian, is that it comes down to execution.

00:28:09 Speaker 3

And how do you sustain the execution and keep up with it, to me, is where I feel like we're just

00:28:14 Speaker 3

so focused on the execution and not missing a B is why we, I believe that we are seeing some really good, healthy momentum all around.

00:28:24 Speaker 2

That's terrific.

00:28:25 Speaker 2

So Nancy, as we talk about the future, and you've mentioned communication, delegation, and validation, which great, great framework, but I've also heard you say exploratory or flexibility.

00:28:40 Speaker 2

And I can see the first three going pretty well.

00:28:43 Speaker 2

I can see flexibility and exploratory being a little out of character for the profession in a lot of ways, and yet having such new skills and powerful tools to enable greater flexibility and exploration.

00:28:57 Speaker 2

Do you see anything from a skill set standpoint that you think will need to change or you'd want to encourage more to make sure that you get the most out of these tools in the future?

00:29:08 Speaker 3

I think that the profession overall

00:29:11 Speaker 3

is evolving that, yes, we do a lot of the core assurance work, which to me is absolutely valuable and will have its place.

00:29:19 Speaker 3

Where I think AI is now enabling us to now unlock is really the advisory.

00:29:24 Speaker 3

So, you know, auditors obviously like to think like control by control.

00:29:28 Speaker 3

We've come in and test and it's very black and white.

00:29:30 Speaker 3

It failed or it passed, right?

00:29:32 Speaker 3

But when we're going to be more playing an advisory role, which is, again, how do we

00:29:38 Speaker 3

get engaged early on strategic initiatives or implementation work where the business is using AI and where do we get the right insights to give them the right observations or recommendations early so they don't, you know, miss certain controls, right, when it goes live.

00:29:53 Speaker 3

And it sounds like very traditional pre-implementation work, Brian, probably, as you would agree.

00:29:58 Speaker 3

But I think

00:29:59 Speaker 3

historically, we're not able to do advisory because it's very, very limited to the conversations we had with people, right?

00:30:06 Speaker 3

The artifacts that we're able to have access to.

00:30:09 Speaker 3

We didn't always have access to, you know, external research.

00:30:12 Speaker 3

Like we would have to get that through a consulting firm.

00:30:15 Speaker 3

We have to pay for the expertise in terms of getting that outside in view.

00:30:19 Speaker 3

But we all know AI can do some great research for you.

00:30:23 Speaker 3

Obviously cautioning that it has to come from the right source, right?

00:30:26 Speaker 3

So I think the new skill or the uniqueness of the skills and how a profession is changing to deliver more value in addition to advisory and doing more advisory is the insights.

00:30:37 Speaker 3

And so how do we not just become, you know, this core auditor, but internal consultants, I would say.

00:30:46 Speaker 3

So to me, we're gonna be wearing the auditor hat, but also internal consultants to advise.

00:30:52 Speaker 3

And we can only do that effectively, in my mind, if we have better tools that allow us to do better research, more timely, right?

00:31:00 Speaker 3

Really understand industry signals to bring that in, to also understand how does the company, right, navigate through some of those external factors.

00:31:10 Speaker 3

and incorporating that to fine tuning our strategy, give me things that we can't control, but it impacts us in how we lead our business.

00:31:18 Speaker 3

That to me is kind of a new realm of where we need to operate as auditors.

00:31:22 Speaker 3

And some people might not be comfortable with that, quite frankly.

00:31:26 Speaker 3

But it's a skill that we need to really broaden, again, insights both inside that we know, but also insights that we have to gather from outside in.

00:31:34 Speaker 2

I think you've made a case, from what I'm hearing from you, is that the profession is going to change in some fundamental ways.

00:31:41 Speaker 2

I'd like to follow up with a question to try to get more at this more deeply.

00:31:45 Speaker 2

Will you change who you'll hire in the future based off their skill sets because of AI?

00:31:53 Speaker 2

Will you be looking for a different type of person?

00:31:56 Speaker 2

And if so, what will be different about them?

00:31:59 Speaker 3

I think the profile of an auditor in terms of, again, responsibilities we expect and how they get the work done, I think we will all agree that will change naturally, you know, to be expected.

00:32:10 Speaker 3

In terms of like how people demonstrate some of those skills to meet those new responsibilities, I think that there will be a new profile.

00:32:19 Speaker 3

And this is why I kind of tell everyone, even as I recruit for new hires, is

00:32:25 Speaker 3

lean in, right?

00:32:26 Speaker 3

I mean, just like if you think about many years ago when data analytics was very, very much the thing where we want people to be data literate, we want people to be very analytical.

00:32:36 Speaker 3

To me, it's the same thing where like, I think everybody needs to see these skills as something that is part of the kind of the common skill stack.

00:32:45 Speaker 3

And this is why I mentioned and emphasized that how do we become, remain relevant?

00:32:51 Speaker 3

in the age of AI.

00:32:52 Speaker 3

And it's not just being literate with AI in terms of what the possibilities and risk are and the tools, but it's about AI fluency too, which to me is very different than just being literate.

00:33:05 Speaker 3

Fluency is when you know all of those things, but you also know how to apply it effectively in the work.

00:33:11 Speaker 3

That to me is very, very distinctly different.

00:33:14 Speaker 3

And to me, the future auditor should be much more AI fluent

00:33:20 Speaker 3

And that to me is going to be a differentiator in terms of the recruitment, the workforce, and remaining relevant, right?

00:33:28 Speaker 3

Being part of an auto shop is, to me, is really the fluency of some of the skills I shared.

00:33:34 Speaker 3

Very, very important.

00:33:35 Speaker 2

And what a way to cap things off here.

00:33:37 Speaker 2

Nancy, thank you so much for sharing not only thoughts of today and the successes and challenges you're having, but also a peek into the future, which

00:33:47 Speaker 2

not only sounds interesting, but sort of compelling for the profession if we can keep up.

00:33:52 Speaker 2

Thank you so much.

00:33:53 Speaker 3

Thank you so much, Brian, for having me.

00:33:55 Speaker 3

Appreciate it.

00:33:57 Speaker 1

If you like this podcast, please subscribe and rate us.

00:34:00 Speaker 1

You can subscribe wherever you get your podcasts.

00:34:02 Speaker 1

You can also catch other episodes on YouTube or at the IIA.org.

00:34:07 Speaker 1

That's T-H-E-I-A dot O-R-G.