Interested in this topic? Visit the links below for more resources:
Visit The IIA's website or YouTube channel for related topics and more.
00:00:02 Speaker 1
The Institute of Internal Auditors presents all things internal audit tech. In this episode, George Barham talks with Mike Calino about how agile methodologies and artificial intelligence are transforming internal audit functions. Collino shares practical insights on implementing agile, auditing the challenges and benefits of this approach.
00:00:22 Speaker 1
And how AI is being used throughout the internal audit life cycle?
00:00:27 Speaker 2
Mike, thanks for being with us today.
00:00:29 Speaker 3
Thank you guys for having me very excited.
00:00:30 Speaker 3
To be here.
00:00:31 Speaker 2
So but I guess before we get started, could you maybe give us what you would consider a definition of agile auditing? What what I think of it I think of efficiency and streamlining processes, but you you tell me, how would you?
00:00:44 Speaker 3
Define it. So I would define agile as allowing you to be dynamic in how you're conducting your audit.
00:00:50 Speaker 3
Cycle being able to conduct procedures in more of a more of an iterative way that allows you to be flexible as certain risks may may come up. Whereas typically when you think of auditing, you would have very structured sort of gated things you need to do. You need to plan, then you need to go through your execute your execution phase, then you need to go through your reporting phase and you need to go through your monitoring.
00:01:12 Speaker 3
Follow when I think of Agile, I think of doing all of those sort of in time box components that focus on one piece of scope for a given audit or for a given body.
00:01:23 Speaker 2
Of work. OK, Mike, next question, how are you using the agile mindset in your internal audit function?
00:01:30 Speaker 3
So from an internal audit standpoint.
00:01:32 Speaker 3
When we think of planning and and agile, what we do is we prepare our annual audit plan and we figure out all the work that we're gonna, that we're gonna be doing in a given.
00:01:43 Speaker 3
That work is then time boxed into what we would call an epic. An epic is equivalent to an audit. So in a given year we would have the amount of audits that would be called epics. Each of those epics would be would have sprints allocated to them and within those sprints you are covering A functional scope area.
00:02:03 Speaker 3
If we think about doing a an upper cure to pay audit for example, and a procure to pay audit in one of the sprints, we may cover the business process side of procured.
00:02:15 Speaker 3
And another Sprint, we may cover the teeny side of procure to pay, but in each one of those sprints, in each one of those two week increments, we're going to plan, we're going to have our field work, we're going to have a review. We're going to have reporting, we're going to have feedback. We're going to confirm observations and then we're going to continuously iterate. And so we're going to have that.
00:02:34 Speaker 3
Continuous feedback loop with the end user until it's done, and then once we're done with one component of a Sprint, we're gonna move on to the next scoping area. So typically we're in audit would take three months to to.
00:02:46 Speaker 3
Complete by the end of three months, your report would be fully written. All of the sprints would be completed, and each Sprint you would have all components of the audit life cycle completed and then at the end you would have stakeholders and auditees who are in fully alignment with what the output is and and less friction and resistance.
00:03:04 Speaker 2
Mike, could you give us an example of the planning and scoping phase? Some of the critical elements that you need to consider?
00:03:11 Speaker 3
I do think it's important as you embark on your agile journey to define the definition of success.
00:03:18 Speaker 3
And to me, we look at this in a few different areas. Number one, we look at Sprint progress. So in a two week period, how much did we get done relative to plan? We look at Sprint velocity, how much can we get done or what what do we plan to get done in two week period. We look at the quality of work you know when you think of a two week time period.
00:03:39 Speaker 3
It feels very condensed, but we don't want to sacrifice quality. So that's another thing we assess. We look at stakeholder satisfaction, how satisfied our stakeholders to go through their all the audit life cycle components in a two week period. It may feel that we're pressed for time.
00:03:55 Speaker 3
And but what we're trying to do is get through everything in an efficient way and give them a given finding, if you will, or give them a certain observation that they could help address more real time. And then the last area of continuous success is continuous improvement, making sure you're reflecting on what worked well and what didn't and.
00:04:15 Speaker 3
Incorporating lessons learned and better ways of working into the next phase of of your your strength.
00:04:22 Speaker 2
00:04:25 Speaker 2
How would you say agile auditing has changed over the past few years and we have AI and we have, you know, different aspects from a technology standpoint. But maybe if you said looking at it maybe 2020 versus 2025, what would you highlight are are some of the major differences?
00:04:41 Speaker 3
I think from 2020 to 2025, the major differences definitely relate to technology.
00:04:46 Speaker 3
Any advancement of technology, so if you go back to when we started our journey, which is in between that 2020 and 2025, yeah, it was around 2022.
00:04:55 Speaker 3
What agile looks for us in 2022 was having a Kanban board using project management tools to figure out what the year is going to look like in terms of our audit plan and then time boxing each audit into sprints and figuring out how we're going to sort of allocate resources and that work and also having a Kanban board where we could sort of know.
00:05:15 Speaker 3
What's not sorted? What's in progress? What is complete, what is in the backlog so agile in 2022 was very project management oriented.
00:05:23 Speaker 3
We had some data and some indicators in terms of how work was going and how we could reallocate. And then I think as the years progressed, the big thing that changed was technology and the data we had. So now there's a lot more data involved in how we manage our day-to-day and and the use of agile when we are doing continuous monitoring and we see there are certain transactions being.
00:05:43 Speaker 3
Flag we may have to adjust our audit plan.
00:05:46 Speaker 3
Working in an agile environment allows us to shift priorities on demand and allows us to really focus on the highest rated risky areas. And so I'd say the biggest difference is still having the project management discipline. Plus listening to the data and then utilizing the skills and program that you've developed to help address the the highest real risk.
00:06:06 Speaker 2
00:06:26 Speaker 3
In terms of readiness, I think it starts with education really talking to the team about how you traditionally would audit right and going through your your various phases.
00:06:35 Speaker 3
To really thinking agile and talking through what the benefits are are in that sense. So when we started our journey, I explained to the team, hey, this is gonna feel a little bit unconventional, but you're gonna reap a few different benefit.
00:06:48 Speaker 3
One of the benefits is you're going to get all the phases done of an audit in one phase and the findings and everything that are gonna come out of the audit are not gonna be outdated. So you're not gonna do an audit for six months and then give all your results at the end in the first two months, you're gonna deliver results. You're gonna have those agreed upon, and you're on this going to be sort of completed and increments over time. And that's more beneficial to the business because.
00:07:10 Speaker 3
They could be agile in their remediation. They hear of a fine and sort of real time and then they could go ahead and address it. So really educating and getting the buy in from my team that this is a step in the right direction.
00:07:22 Speaker 3
Right. The regulatory landscape is very dynamic. Business risk is very dynamic. We need to be able to operate in line.
00:07:28 Speaker 3
With the business.
00:07:30 Speaker 3
Raises a software company, the company.
00:07:33 Speaker 3
Operates under Agile or Scrum methodologies, so we cannot operate in sort of an elongated audit life cycle. If we wanted to really address risk and mitigate things on a day-to-day. Yeah, it sounds like responsiveness is is a big advantage. Just being able to adapt and respond to the business, respond to management, right.
00:07:54 Speaker 2
So from a roles and responsibility standpoint, I know there are Scrum masters and different folks. Would you say that that is something to identify the folks are going to be involved in certain roles in the beginning? Is that just kind of evolve over time or could you just talk a little bit about, you know how important it is to to go ahead and identify folks who are going to be in those those key roles?
00:08:14 Speaker 3
Polls. Yeah, it's definitely important. And it's also important to explain sort of what those roles entail, right? So your scrum master is almost your youth, your Lieutenant, right. Your lead sort of project manager. They're gonna keep everyone accountable. Make sure that.
00:08:15
Right.
00:08:27 Speaker 3
The work is progressing. Make sure your canvas board is up to date, getting the relevant updates from the other team members. We do daily stand ups. The expectation when we get to the daily standups is the scrum master would have spoken to all the team members. Make sure the board is reflected in terms of what's not started, what needs to go on backlog. So when we're having the meeting it's very intentional and everyone understands what the priorities are.
00:08:47 Speaker 3
That the priority shift, it's very clear when we're communicating and we don't need to spend 45 minutes doing a rundown of what everyone did. We can move on to the next day and say, Yep, the expectation for the next day is that we get here and so we'll do.
00:09:00 Speaker 3
Daily standups. We'll do every two weeks. We'll do some sort of retrospective to see what do we plan to accomplish versus what did we accomplish and then see where some of the the blockers were or maybe where there are certain inefficiencies.
00:09:12 Speaker 3
And we look and lean on our agile framework to continuously improve.
00:09:16 Speaker 2
So Mike, from a success standpoint or maybe from a key performance indicator standpoint, what are some of the key things that you would highlight on how we're doing along that journey and measuring our success with that?
00:09:29 Speaker 3
Auditing. So when when we're conducting an agile audit, it's it's extremely important for us to work with the end user and document the user acceptance criteria. What does this mean in practical terms? It means we're documenting the requirements that must be met for a story or or work item to be completed in the audit. So understanding.
00:09:49 Speaker 3
What the business risk is making sure that that's incorporated into the audit in addition to having your standard audit procedures that you're going through and agile is meant to be collaborative and you're working with the end user to making sure that everything is taken into account as you iterate through the.
00:10:04 Speaker 2
OK, from a training standpoint, just to try to get your internal audit function up to speed and and make sure that you know maybe on an annual basis that that you're improving and working towards being more efficient. Would you say that that that you need to go to training or is it more on the job training where as you go through different projects and different audits that you kind of learn?
00:10:28 Speaker 2
And and evolve from.
00:10:28 Speaker 3
There. So I would say you need one person that's trained.
00:10:32 Speaker 3
I joined Brazen 2021 April of 2021 as the first internal audit hire to really build out the function later that year. In 2021, I attended an awesome training in Florida hosted by the CIA about Agile auto. So I learned before I had my team learn and that's something that I hold very near and dear to the heart. So I'd say you need at least one person.
00:10:53 Speaker 3
Who understands what agile auditing is? Goes through, I'd say at least three to five days of very hands on tactical train.
00:11:00 Speaker 3
To see what's going to work for your your business needs and then be able to train other people. I don't think that requires everyone on the team, but you need one person who could then retrain and get everyone into that into that methodology of updating a Kanban board, right, working on sprints, understanding what the expectations are and when you need to finish things and.
00:11:20 Speaker 3
And how you iteratively work through a larger body of work.
00:11:24 Speaker 2
OK.
00:11:25 Speaker 2
So let's shift gears a little bit. Let's talk about artificial intelligence. Could you maybe give a couple of examples of things have maybe over the past year or so that you've seen be implemented with the agile approach, maybe talk about you know some examples, maybe some challenges that folks have had implementing it?
00:11:45 Speaker 2
Maybe just share some stories.
00:11:47 Speaker 3
Yeah. So artificial intelligence has definitely expedited the way that we could carry out and conduct our agile way of auditing some real use cases and and practical ways of thinking about it. When you think of planning, right and when you're under such a small time box component of two weeks for.
00:12:04 Speaker 3
Planning.
00:12:05 Speaker 3
The best thing you could have at your discretion is technology to help you sort of expedite the planning process. So free agile days and free AI days, you spend a lot of time on scoping. Having my planning memos, getting all the relevant information in there. Now we leverage AI to help us scope, help us write the planning memo, things that took multiple hours or multiple days.
00:12:25 Speaker 3
Are now happening in seconds and now we just need to do a quality level of review on it to make sure it's tailored towards what we're doing. That's planning right then we think of we move to your field work phase. Again, you're pressed on time, you have sort of two weeks to cover a large amount of scope.
00:12:40 Speaker 3
Well, what's the simplest thing to use AI for? Record a meeting, transcribed notes, and have sort of a prerequisite for what your work paper is going to look like. And again, we're in the past. You would have a human who would have to take very detailed notes. You then have to get into a room. Did we capture all the relevant information? Does everyone agree? Well, now we have a pretty accurate and I say pretty accurate. Right, because.
00:13:01 Speaker 3
With. With that caveat with AI, we have a pretty accurate representation of the meeting output. We could then summarize from there what your attributes that you're testing or what are what are key takeaways that you need for your work.
00:13:12 Speaker 3
And then you've you've again saved a lot of time. So it's definitely helped expedite in in that sense. When you think of data, data is important in any audit that you're doing. We use AI to.
00:13:24 Speaker 3
Analyzed tabular data so very structured data. If we're doing data analytics as part of an audit, we may write the SQL queries and scripts, but we may use AI to interpret the results.
00:13:35 Speaker 3
If we're looking at documentation, PDFs or unstructured data, we may use AI to help us expedite the review and give us some of the outputs. Or if we're looking for a specific consideration in a report, so we try to look at it and correlate AI to every sort of phase of the audit life cycle and where we can get to it. So then moving on from.
00:13:55 Speaker 3
Execution to reporting.
00:13:58 Speaker 3
Reporting is great, right? You you sort of have a I help you stage your work papers and then for a report you may take the first pass. Say, hey, can you rewrite this section so almost looking at it as a sort of assistant that can help you expedite each part of the process, but you never want to sacrifice quality. So with the use of AI, we make sure we have.
00:14:18 Speaker 3
Diligent review checkpoints to make sure that what we're putting down is sound. That makes sense and it's aligned with what the stakeholders have have communicated.
00:14:24 Speaker 3
To us and then lastly monitoring and follow up, so we use AI to maybe draft a remediation plan. We may get a written up response in terms of where a wherein oddity is in their remediation journey and we may revise it to make it a little bit more cleaner, more concise and so forth. So we've used it at every part of the audit life cycle.
00:14:45 Speaker 3
And it's definitely helped allow us to save time and operate in sort of the the two week sort of time.
00:14:51
Inbox.
00:14:52 Speaker 2
OK, great. Great examples. Would you say there are any misconceptions with with agile auditing someone who's maybe not done it before or maybe just has limited exposure to it? What would you say is is maybe something that's that's out there, that's a misconception.
00:15:07 Speaker 3
Misconception about agile auditing is that it's one-size-fits-all.
00:15:13 Speaker 3
And it's absolutely not and you need to understand the principles of age.
00:15:18 Speaker 3
File and align it with what your intended output is right. Whether you do daily standups or whether you do stand ups a few times a week, you still are going with the spirit of communicating constantly understanding when things are changing internally with the business, external factors, whether you're doing your retrospectives every week versus 2 weeks.
00:15:39 Speaker 3
So the timing, I think there's a lot of flexibility and variability in what you're doing. The documentation, they're very structured documents that the Agile manifesto states right, you have artifacts and and a few different documents. My team does not.
00:15:55 Speaker 3
Align with those documents. 11 we may call them totally different things, but we follow some of the principles to make sure that we are operating in an agile manner and we're not 100% compliant with it. And I I think anyone who tells you they're 100% agile.
00:16:10 Speaker 3
Is kind of fooling themselves.
00:16:12 Speaker 2
00:16:31 Speaker 3
Sure.
00:16:32 Speaker 3
So I think you're going to see a lot more adoption of AI and have that integrated with Agile.
00:16:37 Speaker 3
I think there may be AI Scrum masters right? Who keep your team accountable where you had a human who was sort of monitoring the progress of work and making sure everything was on track. You probably have more aid from an AI scrum master to do that. I think you're going to see a lot more in the data analytics space. You're to see a lot more agentic workflows to help sort of.
00:16:59 Speaker 3
Streamline the way we audit, so I think those are just a few examples of how you can see AI be more inserted.
00:17:06 Speaker 3
Within within agile.
00:17:08 Speaker 2
Mike, you mentioned the possibility of AI replacing the scrum master role. Could you take a little a little bit deeper dive into that, maybe give some examples and how that could happen?
00:17:20 Speaker 3
So you know, I have a vision that there could be the use of AI, you know, to replace a scrum.
00:17:25 Speaker 3
Master what this AI agent would do would be to scan or sift through your Kanban board, right? Understand where there may be too much in one of the various buckets, whether it be in progress, you know, not start it and automatically figure out a way to reallocate.
00:17:46 Speaker 3
Story cards to end users.
00:17:49 Speaker 3
To free up allocation and to get more work done in efficient manner. Because I think right now when you think of Agile, it's really at the discretion of the scrum master. Communicating with each member of the team on the amount of work they can take. If things are moving along and when you think that AI and having AI sort of expedite this process, it can analyze a lot more of this information a lot.
00:18:09 Speaker 3
Faster and figure out.
00:18:11 Speaker 3
If you are time boxing, something a story card, and you're saying it's gonna take 4 hours and it's been sitting on your campaign board in progress for multiple weeks, then you as a human have looked at it and sort of miss sized it. But perhaps, yeah, I can say, hey over a few days it still wasn't done. You've time boxed this to be 4 hours. Is this getting done in this Sprint or should it get allocated to the next Sprint? So I think in terms of like.
00:18:32 Speaker 3
AI Scrum Masters there's an opportunity for them to rationalize your Kanban board, give summary outputs of what we believe we can achieve in a two week Sprint when we're going through planning.
00:18:42 Speaker 3
And then work with us as humans to help tell the story and help drive greater efficiencies as we progress.
00:18:48 Speaker 3
Through the year.
00:18:50 Speaker 2
OK, good deal.
00:18:51 Speaker 2
Well, it was great talking to you today. Thanks for sharing your.
00:18:54 Speaker 2
Insights, Mike.
00:18:55 Speaker 3
Thank you. Thank you guys for having me. Really appreciate the conversation.
00:18:58 Speaker 1
Hey, audit pros ready to supercharge your skills and connect with the best in the field? You absolutely need to check out the IRA's 2025 international conference happening July 14th through the 16th in Toronto and virtually this is your chance to dive into emerging risks, cutting edge tech and global best practices that will elevate your internal audit.
00:19:19 Speaker 1
Lane don't get left behind and register now at the dia.org.
00:19:24 Speaker 1
If you like this podcast, please subscribe and rate US. You can subscribe wherever you get your podcasts. You can also catch other episodes on YouTube or at thecia.org. That's theiia.org.
