Transforming Audits Through Automation and AI

00:00:01 The IIA

The Institute of Internal Auditors presents All Things Internal Audit Tech.

00:00:06 The IIA

In this episode, Andrea Chirillo talks with Antonio Cachaputi about how automation and artificial intelligence are transforming internal audit.

00:00:15 The IIA

They discuss moving beyond traditional sampling and manual processes, practical applications of data analytics and generative AI, and how automation can improve audit efficiency, coverage, and quality.

00:00:28 The IIA

The conversation also explores the risks of over-reliance on technology, the importance of maintaining professional judgment, and the skills auditors need to navigate increasingly automated environments.

00:00:40 Andrea Cirillo

Hello, Antonio.

00:00:42 Antonio Cacciapuoti

Hello, Andrea.

00:00:42 Antonio Cacciapuoti

How are you?

00:00:43 Andrea Cirillo

Fine, thanks.

00:00:44 Andrea Cirillo

Hope the same for you, for the listener as well.

00:00:47 Andrea Cirillo

So should we start?

00:00:48 Antonio Cacciapuoti

Let's do it.

00:00:49 Andrea Cirillo

Perfect.

00:00:50 Andrea Cirillo

Which are the main limitations of the traditional internal audit that the automation

00:00:56 Andrea Cirillo

can realistically address today.

00:01:00 Antonio Cacciapuoti

In my opinion, the traditional internal audit, I would say, is not broken, but it's just tired.

00:01:07 Antonio Cacciapuoti

It's tired of manual work, it's tired of slow cycles, and it's tired of always looking in the rear view mirror.

00:01:18 Antonio Cacciapuoti

And that's exactly where automation can realistically help us today.

00:01:24 Antonio Cacciapuoti

About the limitation, you say that for me, the first major limitation is sampling.

00:01:31 Antonio Cacciapuoti

Because of course, traditional audit is still largely based on samples.

00:01:36 Antonio Cacciapuoti

Let's say maybe 20, 30, 50 transaction out of population that contain maybe millions.

00:01:44 Antonio Cacciapuoti

And without thanks to automation, we can move from sampling to full population analysis in most cases.

00:01:52 Antonio Cacciapuoti

from opinion to evidence, from probably compliant to demonstrably risky.

00:02:00 Antonio Cacciapuoti

Another limitation for me is the time waste on low value activities, because auditors spend an incredible amount of time extracting data or cleaning files, so fixing formats and so on.

00:02:15 Antonio Cacciapuoti

Well, for me, this is not assurance, this is friction.

00:02:19 Antonio Cacciapuoti

And

00:02:19 Antonio Cacciapuoti

A good craftsman is only as good as his tools.

00:02:23 Antonio Cacciapuoti

No matter the skill, poor tools limit the result.

00:02:28 Antonio Cacciapuoti

And in internal audit, it goes further because even the best auditor becomes constrained by manual tools that don't match the speed of the business.

00:02:39 Antonio Cacciapuoti

I really believe that automatation gives auditors back one of the most important things, time.

00:02:46 Antonio Cacciapuoti

time to ask better question.

00:02:49 Antonio Cacciapuoti

For example, why is this happening?

00:02:51 Antonio Cacciapuoti

What does it mean for the business?

00:02:54 Antonio Cacciapuoti

What is the risk trajectory?

00:02:56 Antonio Cacciapuoti

And so the time is one of the most important things because we are limited in terms of time.

00:03:02 Antonio Cacciapuoti

And the third limitation could be the reactive auditing, because automation allows real-time monitoring.

00:03:10 Antonio Cacciapuoti

It allows internal audit to detect unusual behavior or to detect control breakdown and also emerging risk early, sometimes before management even notice.

00:03:22 Antonio Cacciapuoti

And that's a fundamental shift from insight to foresight.

00:03:28 Antonio Cacciapuoti

another limitation is for sure inconsistency.

00:03:32 Antonio Cacciapuoti

Let's be honest.

00:03:33 Antonio Cacciapuoti

Two auditors can review the same process and reach different conclusion.

00:03:38 Antonio Cacciapuoti

It's human.

00:03:40 Antonio Cacciapuoti

On the other hand, automatation introduce consistency.

00:03:44 Antonio Cacciapuoti

The same rule, the same test every time.

00:03:47 Antonio Cacciapuoti

So judgment is still here, but now it's supported, it's traceable.

00:03:52 Antonio Cacciapuoti

And finally, maybe for me, the most strategic point is the credibility with the business.

00:03:59 Antonio Cacciapuoti

When an auditor speaks through comprehensive data, continues inside, the conversation of course changes.

00:04:06 Antonio Cacciapuoti

Internal audits stop being perceived as a periodic inspector, as a periodic policeman, and start being seen as a trusted advisor.

00:04:16 Antonio Cacciapuoti

And so, to conclude and to be clear,

00:04:20 Antonio Cacciapuoti

For me, automation is not about doing more audits.

00:04:24 Antonio Cacciapuoti

It's just about doing better audits, less manual, less retrospective, more intelligent, and more relevant, in my opinion.

00:04:35 Antonio Cacciapuoti

So, Andrea, we talked about limitation of traditional system, but in terms of technologies, which technologies are actually

00:04:46 Antonio Cacciapuoti

making a difference right now and in which concrete use cases.

00:04:51 Andrea Cirillo

Well, of course, I'm starting from my experience as the head of this sub-department in which we broadly apply AI and analytics to all the domain in our bank.

00:05:07 Andrea Cirillo

So what I'm saying is that actually the technologies that are making a real difference

00:05:15 Andrea Cirillo

are often the less exotic, I would say, than the conference circuit would suggest.

00:05:22 Andrea Cirillo

Let me group them by maturity, because that distinction matters for how we deploy them responsibly.

00:05:30 Andrea Cirillo

The first group, and maybe is usually the most underrated, is the plain analytics and traditional machine learning.

00:05:39 Andrea Cirillo

Process mining, for instance, for control testing to see the process for how it is and not just for how it's narrated in the operational guidance.

00:05:49 Andrea Cirillo

As you were mentioning before, full population, especially full population anomaly detection, for instance, on transactional data.

00:05:58 Andrea Cirillo

It works.

00:05:58 Andrea Cirillo

All of these are maybe less glamorous than other things that we are hearing about nowadays, but all of this works.

00:06:06 Andrea Cirillo

And the precision is high enough to drive real findings, to really express sound findings, as you were mentioning, when you start talking about data and evidences and findings based on data, the conversation immediately changes.

00:06:21 Andrea Cirillo

But a lot of internal function I see that are still not completely exploiting this kind of techniques.

00:06:27 Andrea Cirillo

And it often is because this requires a path.

00:06:32 Andrea Cirillo

You have to

00:06:33 Andrea Cirillo

to move along a path, we have to make a journey to do this, to get there.

00:06:38 Andrea Cirillo

But let me maybe introduce some less used technique that is called causal inference.

00:06:46 Andrea Cirillo

This is a quite common technique in some domains outside audit, but it's not that applied in internal audit field.

00:06:54 Andrea Cirillo

It is able to trace and measure the

00:06:57 Andrea Cirillo

statistically the level of association between a cause and effect.

00:07:03 Andrea Cirillo

And this is especially useful when you have to measure the effectiveness of some action, maybe by the management or whatever.

00:07:11 Andrea Cirillo

For instance, we used it to measure the effectiveness, the actual effectiveness of a remedial action.

00:07:17 Andrea Cirillo

So the oddity was, let's say, boasting himself, saying that this action that they have introduced, they introduced was really useful, but then

00:07:27 Andrea Cirillo

We actually measured the effect, and the actual effect was really lower because there were other confounding variables, so other.

00:07:38 Andrea Cirillo

other things that were influencing the final result.

00:07:41 Andrea Cirillo

So this is quite powerful to measure the real effect of the action you are inquiring, so to say.

00:07:49 Andrea Cirillo

Then of course, there is the most glamorous one that is generative AI.

00:07:53 Andrea Cirillo

We have to mention generative AI if we talk about automation and efficiency nowadays.

00:07:58 Andrea Cirillo

And here there is real value.

00:08:01 Andrea Cirillo

We have to see it.

00:08:02 Andrea Cirillo

Of course, there are also real limits.

00:08:04 Andrea Cirillo

And the use cases where I see

00:08:07 Andrea Cirillo

The most value and the most, of course, also the most sound one are the maybe structuring some messy and structured data.

00:08:16 Andrea Cirillo

For instance, especially working on structured data for gap analysis, policies review, and massive tests of population.

00:08:26 Andrea Cirillo

For instance, we are using it together with RPA, so robotic process automation.

00:08:30 Andrea Cirillo

We are using it to test the full population.

00:08:34 Andrea Cirillo

You were mentioning before the inconsistency.

00:08:36 Andrea Cirillo

And so with this kind of technique, you overcome this kind of problem because you can apply the same kind of evaluation criteria to all the population.

00:08:47 Andrea Cirillo

So you define with the business auditor the prompts that are used to test each and every record, each and every group of documents, each and every observable unit, so to say.

00:08:58 Andrea Cirillo

And then you apply it to all the population in a very consistent manner.

00:09:04 Andrea Cirillo

So this is, for instance, a good use case, a good technology that is really changing the way and how we work.

00:09:13 Andrea Cirillo

But for this, one thing that I want to add is that I'm seeing a lot of time the generative AI being used, let's say, just for writing the report, the audit report.

00:09:25 Andrea Cirillo

I think this is quite a temptation, so to say, because I think that this is not where the most value

00:09:34 Andrea Cirillo

when using this kind of technology because this is just doing the same things a bit quicker.

00:09:42 Andrea Cirillo

But what the generative AI, especially if you move to the, let's say, agentic way of using it, is the ability to completely reshape your audit cycle and not just to make things a bit more quickly.

00:09:58 Andrea Cirillo

But

00:09:59 Andrea Cirillo

We will have to talk about it maybe later.

00:10:01 Antonio Cacciapuoti

Using it just for writing report is just like having a Ferrari and doing just 10 kilometers per day.

00:10:09 Antonio Cacciapuoti

So it's a real limitation.

00:10:12 Antonio Cacciapuoti

And I really like going to concrete examples.

00:10:16 Antonio Cacciapuoti

So if you have it, of course, can you share a concrete example of an audit process that has been automated and what benefits it delivered in terms of time, quality or coverage?

00:10:29 Andrea Cirillo

We are working on the two sides of this kind of automation and this kind of, let's say, reshaping the audit process.

00:10:39 Andrea Cirillo

Because of course we are working on specific, let's say, use cases within the activities performed by the audit function.

00:10:51 Andrea Cirillo

But we are also looking at the

00:10:54 Andrea Cirillo

complete audit process, so to say.

00:10:57 Andrea Cirillo

So the way in which we arrange our activities, so the process that we perform as an audit function.

00:11:06 Andrea Cirillo

And we are developing this project called GAIA, so Generative Artificial Intelligence Audit Assistant, that's the name.

00:11:17 Andrea Cirillo

But when I say assistant, I mean something broader than a chatbot.

00:11:22 Andrea Cirillo

It includes a conversational layer for sure, but also a solution that is called Gaia tagging and another layer that is called preliminary risk analysis.

00:11:33 Andrea Cirillo

And all of this is embedded in our new audit operating model.

00:11:37 Andrea Cirillo

And so it's not something sitting, let's say, on the side as an experiment, but we are developing it within the renovation.

00:11:46 Andrea Cirillo

I was saying before, reshaping of the audit process.

00:11:51 Andrea Cirillo

You were mentioning before about the benefits, about the advantages.

00:11:55 Andrea Cirillo

And so the pain point, I would say that the pain point we were trying to solve was the amount of auditor time absorbed, but making sense of large volumes of documentation.

00:12:04 Andrea Cirillo

You can imagine we are, as you know, since we work in the same banking group, we are quite big banking group, a lot of people, a lot of processes, a lot of activities, a lot of documentation, a lot of information.

00:12:16 Andrea Cirillo

And a lot of time that you have to spend just making sense of these documents, so supervisory evidence, materials from other control function, internal audit knowledge, these just arrive at the structured preliminary risk view.

00:12:30 Andrea Cirillo

This is necessary work.

00:12:32 Andrea Cirillo

I mean, we have to look through this document, but highly repetitive and highly time consuming.

00:12:38 Andrea Cirillo

Auditors were effectively redoing engagement after engagement, the same activity.

00:12:43 Andrea Cirillo

And as a result, the time spent on actual judgment challenged interpretation was being quite compressed, so to say.

00:12:50 Andrea Cirillo

This is just to describe you the starting point.

00:12:54 Andrea Cirillo

So how Gaia solves this, I'm not going to get into technical details, but we can say that this is a set of AI services grounded in our audit knowledge base and integrated into the audit workflow.

00:13:10 Andrea Cirillo

I will mention three main pieces of this solution.

00:13:14 Andrea Cirillo

First of all, there is the conversational assistant, which allows auditors to navigate the document base and extract relevant information from natural language interaction.

00:13:24 Andrea Cirillo

generate access, but does not replace analysis.

00:13:27 Andrea Cirillo

So this is, let's say, even if it's a difficult piece of technical stuff that is not easy to get in place, but this is the most obvious one, I would say, that a lot of people is working to have something like that and already has something like that as we already have it.

00:13:45 Andrea Cirillo

Then there is this other solution that we call Gaia tagging.

00:13:50 Andrea Cirillo

This is a foundational layer, let's say, because it classifies documents, every kind, every piece of document or every piece of unstructured data that is able to get through the ingestion pipeline.

00:14:04 Andrea Cirillo

It classifies it in a structured way, aligned with the audit framework, risk model, control objectives, and additional attributes that may be relevant in further steps of the journey.

00:14:16 Andrea Cirillo

The fact is that the entire information base becomes readable through the eyes of the auditor.

00:14:23 Andrea Cirillo

I always say that we add the auditor glasses and so we see everything with our point of view.

00:14:30 Andrea Cirillo

This is done consistently and at scale.

00:14:33 Andrea Cirillo

Then there is the third layer that is we call Gaia preliminary risk analysis, which builds directly on the tagging layer.

00:14:41 Andrea Cirillo

So we can imagine as a layer upon the other one.

00:14:44 Andrea Cirillo

It does not simply summarize documents.

00:14:47 Andrea Cirillo

It analyzes the target content to understand what information these documents provide with respect to the identify control objective.

00:14:55 Andrea Cirillo

So it starts to perform some, as we say, preliminary risk analysis.

00:15:01 Andrea Cirillo

In other words, it helps answer a much more relevant question for audit, not what does this document say that is simply

00:15:08 Andrea Cirillo

a summary, but what does this evidence tell me about the level of coverage and risk of this control objective?

00:15:15 Andrea Cirillo

I mean, it helps to answer specific questions about the control objectives that was highlighted by the tagging solution.

00:15:25 Andrea Cirillo

So as you see, it's a pipeline.

00:15:27 Andrea Cirillo

So we can say that this solution, the preliminary risk analysis, helps to answer a relevant question about the document, not just a question like a summary could do, just say what is the main content of this document, but helps to answer questions related to the control objective.

00:15:48 Andrea Cirillo

So helps to start actually analyzing the risks that are described, that are in some way included or that are in some way

00:15:56 Andrea Cirillo

related to that document.

00:15:58 Andrea Cirillo

And because stacking and preliminary risk analysis are integrated into the new operating model, these outputs are not standalone.

00:16:07 Andrea Cirillo

They become part of how risk assessment is constructed and consumed.

00:16:12 Andrea Cirillo

So we think there is value in integrating AI within the process and making the AI part of the process, not just

00:16:20 Andrea Cirillo

adding as a final touch, so to say, to the process.

00:16:23 Andrea Cirillo

But there is an interesting book called Reshuffled.

00:16:27 Andrea Cirillo

The idea is to reshuffle the audit process using these techniques, doing things in a way that was impossible before the introduction, the possibility to have these technologies.

00:16:38 Andrea Cirillo

The idea here is that we will save time, as we were saying in the first question.

00:16:44 Andrea Cirillo

We will increment coverage in an efficient way, of course, because we will have, let's say, more productivity in this way.

00:16:53 Andrea Cirillo

And of course, we will have quality, greater quality because we will

00:16:58 Andrea Cirillo

our auditors, we will devote more time to evaluating things and not just tracing things and just doing low value activities.

00:17:08 Andrea Cirillo

This is the idea.

00:17:09 Andrea Cirillo

But we are trying to do a lot of things with Gaia, but there are things that we are not trying to do.

00:17:16 Andrea Cirillo

And this is also important because the temptation when you have a high is trying to do too many things, also things that in my opinion have to remain

00:17:27 Andrea Cirillo

human, performed by human.

00:17:29 Andrea Cirillo

So a Gaia does not formulate audit findings.

00:17:34 Andrea Cirillo

It does not assign severity ratings.

00:17:39 Andrea Cirillo

It does not write the final report as a judgment.

00:17:43 Andrea Cirillo

Even preliminary risk analysis, as I was mentioning, for all it is sophisticated and advanced, remains preliminary.

00:17:53 Andrea Cirillo

It prepares evidence

00:17:55 Andrea Cirillo

It structures signals, it highlights potential areas of attention, but the interpretation stays with the auditor.

00:18:04 Andrea Cirillo

The principle, I would say, is simple.

00:18:08 Andrea Cirillo

The models prepare the material and the auditor express the final judgment.

00:18:14 Andrea Cirillo

The signature in the document, but also in a figurative way, the signature is human and has to remain human.

00:18:21 Andrea Cirillo

That is our idea at least.

00:18:23 Antonio Cacciapuoti

This is Andrea, it's a very important signal that you are giving because this is part of a professional judgment and that in my opinion cannot be replaced just by a machine.

00:18:35 Antonio Cacciapuoti

So it's a very good signal you are giving to the audience.

00:18:38 Andrea Cirillo

Yeah, we think that has to be right because the responsibilities is our own and we have to be able to defend what we write on the audit report.

00:18:48 Andrea Cirillo

I mean, there is the temptation.

00:18:50 Andrea Cirillo

I know that

00:18:51 Andrea Cirillo

quite a lot of audit function are working on this, those to have an automatic report.

00:18:56 Andrea Cirillo

But the automatic report is not your report.

00:18:59 Andrea Cirillo

We think that you have to take the time in your head to gather all the information.

00:19:05 Andrea Cirillo

to derive from the findings, the messages, the findings, the whole story.

00:19:12 Andrea Cirillo

So you have to do this in your head.

00:19:15 Andrea Cirillo

Of course, you can be helped in drafting things or in reviewing also the text, in challenge your text to be sure that it is aligned with the methodology, whatever you want.

00:19:25 Andrea Cirillo

But the beginning of everything, when you start gathering information, you make a sense of that information.

00:19:31 Andrea Cirillo

So making a sense, sense making is what you have to do.

00:19:35 Andrea Cirillo

You cannot delegate, automate or whatever.

00:19:37 Andrea Cirillo

You have to get the possibility to do this job because it's what makes us auditor actually.

00:19:45 Andrea Cirillo

Since we were mentioning about limitations of the traditional way of doing audit, of course we have to be not to be naive.

00:19:55 Andrea Cirillo

And so we have to ask ourselves which are the risks or maybe just the limitation of both of them.

00:20:01 Andrea Cirillo

about automation.

00:20:02 Andrea Cirillo

So you introduce automation, these in internal audit, it comes as always in life with some risks and limitation.

00:20:09 Andrea Cirillo

What are, in your opinion, these risk and limitation and also what we can do to overcome them in a way or at least to mitigate them?

00:20:19 Antonio Cacciapuoti

I really like this question because as you were saying before, we have this temptation of let the machine doing everything as if the machine was God.

00:20:29 Antonio Cacciapuoti

Because we often talk about automation like it's the solution to everything.

00:20:35 Antonio Cacciapuoti

But there are some risks I always highlight.

00:20:38 Antonio Cacciapuoti

And the first one is, of course, over dependence.

00:20:42 Antonio Cacciapuoti

Once something is automated,

00:20:45 Antonio Cacciapuoti

people tend to trust it blindly.

00:20:47 Antonio Cacciapuoti

But if the under logic is wrong or just, let's say, outdated, you are scaling errors.

00:20:54 Antonio Cacciapuoti

You are just scaling errors.

00:20:56 Antonio Cacciapuoti

And in audit, you know, scaling errors is probably the worst thing you can do.

00:21:01 Antonio Cacciapuoti

Another risk I see is losing what I was saying before, this professional judgment, because audit has never been just about ticking boxes.

00:21:13 Antonio Cacciapuoti

It's more about understanding context, reading between the lines, challenging what doesn't feel right.

00:21:21 Antonio Cacciapuoti

Automation, of course, is great with the rules, but reality is often messy.

00:21:27 Antonio Cacciapuoti

So if we rely on automated check, we risk missing the real issue.

00:21:35 Antonio Cacciapuoti

And of course, another risk, the third risk, which is, in my opinion, very underestimated, is the data quality.

00:21:44 Antonio Cacciapuoti

automation, like you were saying before, runs on data, but in many organizations, data is still fragmented, inconsistent.

00:21:54 Antonio Cacciapuoti

So you can end up with a very precise but completely wrong conclusion.

00:21:59 Antonio Cacciapuoti

So based on this risk, the question is how we can manage all of this.

00:22:05 Antonio Cacciapuoti

For me, the key idea is

00:22:08 Antonio Cacciapuoti

Automation should just support decision, not replace them.

00:22:14 Antonio Cacciapuoti

You need the famous human in the loop.

00:22:17 Antonio Cacciapuoti

You need this human checking outputs, challenging the result, and of course, improving the logic.

00:22:25 Antonio Cacciapuoti

And also, the governance part is important.

00:22:28 Antonio Cacciapuoti

The governance becomes critical.

00:22:31 Antonio Cacciapuoti

You need to treat automated controls almost like, I would say, like living system,

00:22:38 Antonio Cacciapuoti

testing them, updating them, validating them over time.

00:22:43 Antonio Cacciapuoti

So I'd say that the real shift here is this.

00:22:47 Antonio Cacciapuoti

We move from manual risk to model risk and data risk.

00:22:51 Antonio Cacciapuoti

And of course, internal audit needs to be ready for that.

00:22:55 Antonio Cacciapuoti

We need to be ready for this change.

00:22:59 Antonio Cacciapuoti

So Andre, this was my opinion in your question, of course.

00:23:04 Antonio Cacciapuoti

We have different limitation, but I focus on my first three, for me, limitation, the most important limitation I can see on this automation.

00:23:15 Andrea Cirillo

I can read what you say, especially the first point about the risk of taking for good and taking for right to what the result of automation is actually a documented one, because it is called automation bias.

00:23:29 Andrea Cirillo

It's not something related to AI, but the more a process is

00:23:34 Andrea Cirillo

automated and gets out, let's say, complete and then polished, the seemingly perfect, the more the human is tempted to take it for right, even if it is wrong.

00:23:47 Andrea Cirillo

And this is more true nowadays that we have this kind of conversational agent and whatever that are

00:23:54 Andrea Cirillo

They seem to be reasonable.

00:23:55 Andrea Cirillo

They seem to be right, actually.

00:23:57 Andrea Cirillo

They seem to be saying the truth.

00:24:00 Andrea Cirillo

And also there is, you made me think about one specific topic related to Gen.

00:24:06 Andrea Cirillo

AI, that is when you ask to a Gen.

00:24:09 Andrea Cirillo

AI solution, you need whatever chatbot you can imagine, you can think of, why?

00:24:16 Andrea Cirillo

a given result was produced.

00:24:19 Andrea Cirillo

So you ask, why did you do this?

00:24:21 Andrea Cirillo

Have you done this?

00:24:22 Andrea Cirillo

Why?

00:24:22 Andrea Cirillo

Which is the reason?

00:24:23 Andrea Cirillo

Which is your reasoning?

00:24:25 Andrea Cirillo

It will provide you with some answer, some traceability, some audit trail, we would say, but that is not right.

00:24:33 Andrea Cirillo

Anthropic especially made some research on this, and they found out that this audit trace is completely made-up.

00:24:41 Andrea Cirillo

So it's not actually, let's say, computationally what happened within the model to produce that kind of result, that kind of answer, but it's just made-up to be, again, reasonable to convince you.

00:24:53 Andrea Cirillo

And this is a risk once again.

00:24:55 Andrea Cirillo

And so you just made me think about it.

00:24:58 Antonio Cacciapuoti

Yeah, no, of course, because as you were saying, AI is a powerful assistant, but of course we need to have the knowledge to understand if there is an hallucination.

00:25:11 Antonio Cacciapuoti

We need to have the right knowledge to read what AI is saying.

00:25:15 Antonio Cacciapuoti

We cannot just rely and trust it blindly, as I said before.

00:25:21 Antonio Cacciapuoti

So this is just the fear we have for a new generation that they are just relying on AI without having knowledge, without studying before.

00:25:31 Antonio Cacciapuoti

Because if you don't study, you don't know if AI is producing a real output or is just an hallucination.

00:25:39 Antonio Cacciapuoti

This is another big risk.

00:25:41 Andrea Cirillo

Yeah, it is.

00:25:42 Andrea Cirillo

And this brings me to wonder about which are the skills that the auditor of the future, or maybe of the really near future, maybe the auditor of tomorrow or this evening probably, will need, which kind of skills will become essential?

00:26:00 Andrea Cirillo

How will the role of the internal auditor being redefined in these automated environments?

00:26:06 Andrea Cirillo

So what remains

00:26:08 Andrea Cirillo

for the auditor if a lot of things are automated.

00:26:12 Andrea Cirillo

What's your take on this?

00:26:14 Antonio Cacciapuoti

Is actually the part I find most exciting because automatation is of course not reducing the importance of internal audit.

00:26:22 Antonio Cacciapuoti

It's just reshaping it.

00:26:25 Antonio Cacciapuoti

And if you look at the traditional audit model, as we were saying before, a lot of time was spent on manual activities.

00:26:33 Antonio Cacciapuoti

like collecting data, sampling or testing controls one by one.

00:26:38 Antonio Cacciapuoti

Automatician changed that completely.

00:26:40 Antonio Cacciapuoti

We said before we can test entire population in real time.

00:26:44 Antonio Cacciapuoti

So the question becomes, if machines do the checking, what do auditors do?

00:26:50 Antonio Cacciapuoti

We are, of course, smarter than them.

00:26:53 Antonio Cacciapuoti

We created them.

00:26:54 Antonio Cacciapuoti

So we find always a solution.

00:26:58 Antonio Cacciapuoti

We focus on what machine cannot do.

00:27:01 Antonio Cacciapuoti

So interpreting,

00:27:03 Antonio Cacciapuoti

challenging connecting the dots.

00:27:06 Antonio Cacciapuoti

So instead of asking, is this control working?

00:27:11 Antonio Cacciapuoti

We start asking, what does this mean for the business?

00:27:14 Antonio Cacciapuoti

Where is the real risk?

00:27:16 Antonio Cacciapuoti

What are we not seeing?

00:27:18 Antonio Cacciapuoti

And then for the second part of your question, this essential skill, I see, let's say, three major shifts.

00:27:27 Antonio Cacciapuoti

First,

00:27:28 Antonio Cacciapuoti

is the data literacy and data analytics.

00:27:31 Antonio Cacciapuoti

Not coding necessarily, of course, but at least understanding how data flows, how it's organized, and where it can go wrong.

00:27:42 Antonio Cacciapuoti

There is also a data analytics guidance for auditors.

00:27:45 Antonio Cacciapuoti

The IIA just released 2 global practice guides to help auditors improve the analytics knowledge.

00:27:54 Antonio Cacciapuoti

And third,

00:27:56 Antonio Cacciapuoti

the most important skill for auditor in my opinion, the communication skill, because of course insights are useless if they stay in a dashboard.

00:28:06 Antonio Cacciapuoti

Auditors need to translate complexity into something clear, into something relevant and useful for the business.

00:28:15 Antonio Cacciapuoti

As we were saying before, a machine cannot communicate with the stakeholders the same way a human does.

00:28:22 Antonio Cacciapuoti

it can just adjust its tone or style depending on the person in front of it.

00:28:29 Antonio Cacciapuoti

And it doesn't really understand someone's mood or the context of the situation because when the most important skill is the communication, we have this communication, this functional report to the board of directors.

00:28:42 Antonio Cacciapuoti

But if as we are inside the organization, we are internal auditors, we know our

00:28:48 Antonio Cacciapuoti

stakeholders so we can adjust our tone, our style depending on the situation, depending on the person.

00:28:55 Antonio Cacciapuoti

This is something that machine cannot do or at least not well as we can do it.

00:29:01 Antonio Cacciapuoti

So again, the role evolves from, let's say, a control tester to a strategic advisor.

00:29:10 Antonio Cacciapuoti

And frankly, that makes the job much more interesting and much more impactful, I would say.

00:29:19 Antonio Cacciapuoti

So Andrea, these are, in my opinion, the important skills auditors need to stay one step ahead and to keep a central role.

00:29:29 Antonio Cacciapuoti

And so continuing on that, I would like to ask one last question for you.

00:29:35 Antonio Cacciapuoti

And we often talk, we already did before, of the FEMS, no human in the loop, but in practice, for you,

00:29:43 Antonio Cacciapuoti

How can internal auditors stay at the center of this automation and at the same time ensure that they are driving value rather than just overseeing technology?

00:29:55 Andrea Cirillo

I'm starting to feel a bit of pain for this poor human in the loop.

00:30:00 Andrea Cirillo

I mean, because this image of this human within this loop running with

00:30:07 Andrea Cirillo

along this route.

00:30:08 Andrea Cirillo

Let's try to see what this poor human in the loop can do to have a bit of fun at least.

00:30:14 Andrea Cirillo

But I think to give an answer to this, we have to move a bit higher to a slightly different altitude maybe that we are doing with this.

00:30:23 Andrea Cirillo

previous questions that get also in some detail of techniques and similar things.

00:30:28 Andrea Cirillo

Because you are mentioning some kind of risk, I think, the one of just overseeing technology, I think you said.

00:30:36 Andrea Cirillo

It's a real risk, actually, because there is that internal auditors become operators of the automated process.

00:30:44 Andrea Cirillo

like Charlie Chaplin movie with these men within the mechanics of the process rather than owners of the process.

00:30:55 Andrea Cirillo

Someone who keeps the dashboard, monitors the alerts or escalate the exceptions, let's say custodian of the technology rather than a profession using the technology.

00:31:10 Andrea Cirillo

And if that happens, the function I think that loses weight

00:31:15 Andrea Cirillo

I think that what makes internal audit valuable in organization isn't just the work that it does.

00:31:23 Andrea Cirillo

I mean, the activity.

00:31:24 Andrea Cirillo

It is, as I was saying before, the judgment it brings.

00:31:28 Andrea Cirillo

That is independent, is structured with the systemic view, is a judgment on whether the organization is actually managing what the risks, if it is properly addressing these risks.

00:31:44 Andrea Cirillo

And so you mentioned, how do you stay at the center of this?

00:31:52 Andrea Cirillo

Let me try to provide some idea.

00:31:54 Andrea Cirillo

First of all, you have to understand the tools at the level of how they fail.

00:32:01 Andrea Cirillo

So not just how they work.

00:32:04 Andrea Cirillo

That is, you have to, but it's not, you have also to understand how they fail, when they fail, because nowadays anyone can run a model in some way.

00:32:13 Andrea Cirillo

But few can tell you the conditions under which the model is silently wrong.

00:32:20 Andrea Cirillo

That is when the danger is lying.

00:32:23 Andrea Cirillo

The second skill, so understanding how it fails, when it fails, is the one that keeps you central because you can ask

00:32:30 Andrea Cirillo

not only what does the system say, but if you ask just this, you have been already demoted to the role of the operator of the machine that I was mentioning before.

00:32:41 Andrea Cirillo

You have to ask why the system says that and under what conditions would it be saying something different.

00:32:49 Andrea Cirillo

So you have to understand also when the system fail, how it fails, why it fails, so that you keep to be someone that is able to

00:32:58 Andrea Cirillo

handle properly this tool, at the beginning you were talking about people, craftsmanship and tools.

00:33:04 Andrea Cirillo

So you have to really possess the tool and not being possessed by the tool.

00:33:08 Andrea Cirillo

And then you have to own the workflow, the design of the workflow.

00:33:13 Andrea Cirillo

The auditor has to be in the room, so to say, when the automation is being designed, not just handed it afterward as a tool to operate.

00:33:24 Andrea Cirillo

Because every automation embeds choices

00:33:28 Andrea Cirillo

what gets flagged, what gets rerouted, what gets summarized.

00:33:32 Andrea Cirillo

Each and every decision of this is a decision that the audit must be involved in.

00:33:38 Andrea Cirillo

So that's why it's really important also the process with which you get to the automation.

00:33:44 Andrea Cirillo

So involving people in this process, not just because it's always said that people do not resist change

00:33:54 Andrea Cirillo

they resist being changed.

00:33:56 Andrea Cirillo

So it's important how we introduce this kind of innovation within the audit department.

00:34:01 Andrea Cirillo

And then since we always talk about the human within the loop, I think that we have to increase the possibility of this human related to the loop, because it's not always necessary to have the human in the loop.

00:34:19 Andrea Cirillo

We have to reason about

00:34:22 Andrea Cirillo

what to automate and at which level.

00:34:25 Andrea Cirillo

And for this, a tool that we are using in our own decisions and reasoning is a metrics that is, we can call the reversibility and materiality metrics.

00:34:38 Andrea Cirillo

So you can imagine this metrics with two axes, basically.

00:34:42 Andrea Cirillo

with two possibility to entrances, so to say.

00:34:46 Andrea Cirillo

Because not every decision in the audit workflow deserves the same level of human involvement and pretending otherwise of everyone, for everything there is, we need this human in the loop, we overburden this human or under-protect the function on the opposite side.

00:35:05 Andrea Cirillo

The logic of the metrics is quite simple.

00:35:07 Andrea Cirillo

In one axis, you have how reversible is the decision.

00:35:11 Andrea Cirillo

So if we get it

00:35:12 Andrea Cirillo

Can we walk it back?

00:35:14 Andrea Cirillo

Can we reverse it?

00:35:16 Andrea Cirillo

And on the other axis, we have how material?

00:35:19 Andrea Cirillo

So what's the size of the consequence?

00:35:22 Andrea Cirillo

And from these two axes, you get the different possibility.

00:35:26 Andrea Cirillo

For instance, high reversibility, low materiality decision, like the preliminary clustering, the document tagging.

00:35:34 Andrea Cirillo

This can sit comfortably with the human on the loop.

00:35:38 Andrea Cirillo

So we supervise, we sample, we move on.

00:35:41 Andrea Cirillo

We do not have to process each and every clustering, for instance.

00:35:45 Andrea Cirillo

So we say on the loop.

00:35:48 Andrea Cirillo

If we are on the other quadrant, so low reversibility, high materiality decisions, like issue classification, severity ratings, scope changes.

00:35:59 Andrea Cirillo

So this actually need the human in the loop.

00:36:02 Andrea Cirillo

So every output reviewed before it propagates further, because this is high materiality stuff.

00:36:11 Andrea Cirillo

And then the third category, which is the audit opinion itself.

00:36:16 Andrea Cirillo

So this is where you have the human only.

00:36:20 Andrea Cirillo

So there are some activities, as we were mentioning before, talking about the final judgment, that has to remain human only.

00:36:30 Andrea Cirillo

Not because the machine couldn't do it, but because we do not want the machine to do it.

00:36:38 Andrea Cirillo

And this is a strategic decision also for the function, I think for human in general, but it's for sure, at least for the function.

00:36:47 Andrea Cirillo

So this framework can help us to understand how to introduce this technology, what to automate at which level, how, and in this way, how to stay relevant as professional, as audit professional.

00:37:01 Andrea Cirillo

Is it a good answer?

00:37:02 Andrea Cirillo

I don't know.

00:37:03 Andrea Cirillo

Is it an answer for you?

00:37:04 Antonio Cacciapuoti

Yeah, it's a wonderful answer and I would

00:37:08 Antonio Cacciapuoti

I would like to conclude this part of human in the loop or as you said on the loop.

00:37:14 Antonio Cacciapuoti

I would like to quote once again because it's my favorite quote of the professor Richard Baldwin that say that that's not AI that will replace you

00:37:27 Antonio Cacciapuoti

but it's someone who knows how to use AI that may replace the human.

00:37:34 Antonio Cacciapuoti

So in this sentence there is the importance of both parts.

00:37:38 Antonio Cacciapuoti

So the human, the central role of the human and also the importance of technology because of course

00:37:47 Antonio Cacciapuoti

technology will be the future and it's becoming the present of the internal audit function.

00:37:53 Andrea Cirillo

Right.

00:37:54 Andrea Cirillo

And so starting a good way to start applying and understanding technology maybe is to listen to this podcast, I hope.

00:38:03 Antonio Cacciapuoti

Yeah, thank you, Andrea.

00:38:04 Antonio Cacciapuoti

I really enjoyed this conversation and I hope the audience did too.

00:38:09 Andrea Cirillo

Thank you.

00:38:12 The IIA

If you like this podcast, please subscribe and rate us.

00:38:15 The IIA

You can subscribe wherever you get your podcasts.

00:38:18 The IIA

You can also catch other episodes on YouTube or at theiia.org.

00:38:22 The IIA

That's T-H-E-I-I-A.org.