Makes Sense to Me (East Coast vs. Left Coast)

In my previous post, I asked you to figure out how the auditor went wrong. D. Wanner nailed it in his response to the post. But, since I’ve already written this part, I’ll go through my own explanation. 

Go back to how the percentage is calculated: number of payments processed divided by number of payments received. Calculating the average of the percentages can give a close approximation, but only if the volume received each day is very similar. In the example, the mail received on Monday really represents two days (Saturday, also) and it would be expected that the volume on Monday would be much higher (probably the reason for the poorer results). Just to drive the point home, let’s use an extreme example. All Mondays, 1,000 pieces of mail are received. Every other day, 100 pieces are received. That means that, for the two week period, 2,800 pieces of mail were received and, based on the percentages from the problem, 2,427 were deposited on the same day. Actual percentage? 87 percent. 

This may have seemed obvious to many of you, but I have seen auditors fall into this trap. In one spectacular instance — the one my example is drawn from — not only had the auditor come to the erroneous conclusion that the department had achieved its objectives, but the auditor-in-chief and the manager both agreed.

This has its roots in a concept known as Simpson’s Paradox. (Is it me, or does this just beg for the following exchange, “What do you say when you realize you fell into Simpson’s Paradox? D’oh!” **Listens to the sounds of crickets chirping.** I’m sure statisticians think that is hilarious.) The following question is based on that paradox.

An auditor is reviewing the results of operations in two separate facilities — one in New York and the other in San Francisco. Each has only two operations — customer service (which receives and processes the initial orders) and customer complaints (which handles corrections of errors). He notes that the error rate for the customer service department in San Francisco is lower than that in New York. He further notes that the error rate for the customer complaints department is also lower in San Francisco than in New York. He presents this information to his boss who says, “Glad to hear that. It supports what management has told us, that the error rates in San Francisco are lower than in New York.” The auditor — one of those who has never liked the Left Coast — says, “Sorry to tell you this, but we can’t make that conclusion. In fact, it is possible the error rate is higher in San Francisco than in New York.”

Who is right?

Give it a shot, and I’ll be back in a couple of days with the answer.


Posted on Feb 10, 2010 by Mike Jacka

Share This Article:    

  1. The auditor is correct when he says, "Sorry to tell you this, but we can’t make that conclusion. In fact, it is possible the error rate is higher in San Francisco than in New York." Here's why:

    The boss makes the assumption that because the individual error rates are lower in San Francisco that the overall error rate must be lower in San Francisco too. This is where the boss falls into Simpson's Paradox (D'oh!). The basis of Simpson's paradox is that something that is true of each subset of a population is not necessarily true of the population as a whole. That means in this situation since we don't have any numbers to go with the error rates, it is possible for San Francisco to have lower individual error rates than New York, but still have a higher overall error rate than New York.

    Here is another example of Simpson's Paradox with some numbers that might help explain it:

    Suppose ten men and ten women apply to two different graduate programs (Business and Law). For men, 6 out of 8 (75%) were accepted to business school and 0 out of 2 (0%) were accepted to law school. For women, 2 out of 2 (100%) were accepted to business school and 2 out of 8 (25%) were accepted to law school. In this example women have higher individual acceptance rates than men do for both Business and Law School. However, the overall acceptance rate is higher for men than women, 60% vs 40%.

    Simpson's Paradox, like the last example shows that numbers can lie and you need to investigate to make sure the numbers that you have make sense.

Leave a Reply