Gathering Audit Evidence: Techniques for Success

By Craig Cochran

Gathering audit evidence involves a mix of techniques that are used interchangeably: visual observation, examination of records, and employee interviews. One moment you will be looking around the work area, and the very next you’ll ask an employee a question. Let’s take a deeper look at each major technique used for gathering audit evidence.

Visual observation

This is the most basic ways to gather evidence during an audit. Simply looking around is a very powerful way to understand how an organization works. Is the place organized or cluttered? Is communication formal or informal? Smart auditors immerse themselves in the organization they are auditing and look at it from every angle. Here are some especially powerful pieces of audit evidence to look for:

• Uncontrolled documents. Look around for “bandit documents” posted on walls, machines, and desks. These are often informal specs or procedures that aren’t controlled. Bandit documents often take the form of Post-It notes, marker settings written on machines, old memos, printed emails, and photocopies of external documents. If the document provides information on product requirements, process control guidelines, or decision making criteria, you need to inquire how the information is supposed to be controlled.
• Product outside the normal flow. Look for large piles of product that appear to be outside the normal flow of production. These are often nonconforming products, moved to the side so they can be addressed. If you find nonconforming products, make sure they are being handled in accordance with the company’s process for controlling nonconforming products.
• Measuring instruments. The presence of measuring instruments usually means that there are important characteristics that must be verified. If you see measuring instruments, find out what they’re used for. If they are being used to check product, verify service, or control a process, then the organization should have a process for ensuring the fitness of the instruments. These range from complex measurement devices to simple gauges (such as templates, patterns, jigs, rulers, tape measures, and limit samples), and everything in between.
• Housekeeping and organization. It doesn’t take an expert to identify a mess. Problems with housekeeping and clutter are symptoms of larger issues. Delve deeper into these conditions and try to find out what is happening. Lack of housekeeping often points to issues with product preservation, defects, identification, and traceability.
• Product identification. Look to see that all products have some sort of identification. Identification includes a variety of methods such as stickers, tags, bar codes, paint dabs, assigned location, special bins, boxes, or bags. If you’re not clear what the identification is, ask someone in the area.
• Improvised fixes and repairs. Look for evidence that employees have had to make improvised fixes and repairs. Amateur repairs often use duct tape, rope, shims, and other crude methods. If employees have improvised repairs, it could be evidence that the maintenance program is not being carried out or that adequate resources are not being provided by management.
• Informal record keeping. Look for informal record keeping in notebooks, logbooks, scratch sheets of paper, etc. If the records relate to anything that ISO 9001 or the company’s management system addresses, the records should be handled in a formal manner.

---

---

Examination of records

Records are historical artifacts. They tell what has happened in the past. Auditors generally accept records as statements of fact. If we have a credible record that indicates something happened, we can usually conclude the action happened. Of course, records are not required for everything that happens in an organization. If a company procedure or the applicable standard (such as ISO 9001) requires a record, then obviously we need a record.

I mentioned the need for “credible” records. What exactly is a credible record? It is one that we can have faith in as being an accurate representation of the activities it vouches for. These are some characteristics that help make a record credible:

• Completely filled out. If the record starts as a blank form, then we would expect all spaces to be completed. Any blanks should have clear explanations for the omission.
• Dates. Records need verifiable dates in order to have any credibility.
• Participants. If the record was a meeting, then a listing of participants would help tell the story of what happened. If the record was simply proof of something happening, then who carried out the action would need to be recorded.
• Actual results. What actions took place? If the record was proof inspection, then the inspection results would be needed. If the record was taken from a meeting, what was decided?
• Subsequent actions. Many records will include action items or follow-ups. If the activity being recorded includes these types of actions, the record should clearly indicate it.

Employee interviews 

An interview is a structured discussion. Unlike a normal discussion that can meander over a wide variety of topics, an interview has a specific objective. Your objective is to capture factual information about the process being audited. The interviewer must plan and control the discussion so the required facts are gathered in the most efficient manner possible. In general, certain cues help an auditor know if the interview they’re leading can be considered objective evidence:

• The employee makes statements relating to things they personally saw or took part in.
• The employee’s statements relate directly to their responsibilities and authorities.
• The employee’s statements can be corroborated by records or supporting statements from other personnel.
• The employee makes statements that are specific and which include credible details (as opposed to blanket statements that are vague on specifics).

In cases where there are requirements for records, a statement alone would not suffice.

Sampling of evidence

Audits are never 100 percent inspections. There is simply not enough time to examine everything that is happening within the organization. Instead, audits sample evidence. The sample does not need to be statistically based, but it does need to be representative. If a population of evidence includes thousands of records, then a representative sample would certainly be more than one. Just take a reasonable sample of evidence given the evidence available. Think about how you might subdivide the overall population of evidence into rational sub-groups. For instance, if you’re auditing training, you might sub-divide employees into top management, hourly, employees less than 90 days old. Employees with more than 10 years of experience. Then you’ll take your samples from the sub-groups, instead of just blindly selecting samples from the overall population.

Recording notes

Note taking is an important part of the audit process. Evidence gathered must be fully traceable and highly detailed. This means that auditors must develop efficient means for capturing their notes. If you’re the type of person that takes notes during an interview, make sure to tell your auditee that you’ll be writing down details. Also remember that you likely have one chance to capture the details of evidence. Slow down, take your time, and write the required details and evidence traceability while you’re in the department. The one thing you would never do is ask an auditee to speak into an audio recorder. This makes the audit seem too much like a police interrogation.

Examples of evidence 

The best way to understand good versus bad evidence is to study some examples. Here’s what appears to be a detailed statement of facts. Read it carefully and decide what you don’t like about it:

“Returned goods were missing the nonconforming materials tags, which greatly increases the chance of accidentally shipping bad material.”

This evidence has a number of problems. In summary, it’s highly opinionated and not traceable. Here are the specifics:

• Where was this found? The area or department should be indicated.
• What returned goods are we talking about? We need to identify them to enable traceability. Part numbers or descriptions should be adequate.
• How many returned goods were missing the tags? The quantity helps put the situation in perspective.
• The auditor has included his opinion at the end. This adds subjectivity to the evidence and will only inflame the auditee.

Let’s rework the evidence. Here are the same facts, expressed in much more complete and correct terms:

“Three out of 10 returned desk kits (product code 675) in the warehouse hold area were missing the nonconforming materials tags.”

The first thing that strikes you is how much more specific this evidence is. It is the epitome of “just the facts, ma’am.” Here is why it is better:

• The area where the returned goods were located is clearly indicated (i.e., warehouse hold area)
• The identity of the returned goods is provided (i.e., desk kits, product code 675)
• The sample size is shown, helping the audited organization understand the magnitude of the situation (i.e., three out of 10)
• Only facts are stated. No auditor opinions about impacts or ramifications of the nonconformity are included.

Seeking evidence of positives

Smart auditors always ask themselves, “Am I actively looking for positives during the audit?” The audit should be a balanced snapshot of the organization. Balanced means the identification of positive practices, as well as nonconformities. Any organization that is still in business in these tough economic times is doing a lot of things right. Too often, audits become an obsessive exercise in finding the organization’s flaws. As you can imagine, audits of this type are rarely welcomed or requested.

You have to continually remind yourself to be on the lookout for positives during the audit. Ask yourself these questions during the audit, to keep the topic fresh on your mind:

• What sets this organization apart?
• What do they do especially well?
• What practices create competitive advantage?
• Where are the isolated pockets of excellence?
• Who are the innovators of new methods and tools?

If you are leading an audit, remind the other auditors under your supervision to also be on the lookout for positives and best practices.

It’s important to note that in mature management systems, identification of positives is one of the most important purposes of an audit. That’s because the discipline of the management system is well established. Audits have already picked the low-hanging fruit, so auditors can turn their attention to finding the isolated pockets of excellence. These pockets of excellence are often obscured and hidden from view, so an important purpose of the audit is to root these out. Once identified, these best practices can be widely adapted and turned into the new standard. This motivates people to embrace the audit process, while driving improvement throughout the organization.

Here are some examples of positive findings:

• Clean and well-organized receiving area
• Management of the lab is fully engaged in the management system
• Effective corrective actions generated by production
• Detailed action plans for achieving objectives in the purchasing department.

Try to write positives that are highly individualized and specific to the areas they relate to. We have no use for “boiler plate positives” that are generic and generalized. Remember, we’re looking for best practices that the rest of the organization can learn from. Keep your eyes open for positives, and you’ll find that you produce better results and are always welcomed as an auditor.

About the author

Craig Cochran is the North Metro Regional Manager with Georgia Tech’s Economic Development Institute. He has assisted more than 5,000 companies since 1999 in QMS implementation, problem solving, auditing, and performance improvement. Cochran is a Certified Quality Manager, Certified Quality Engineer, and Certified Quality Auditor through the American Society for Quality. He is certified as a QMS Lead Auditor by Certus Professional Certification.

He is the author of numerous books, including ISO 9001:2015 in Plain English and Internal Auditing in Plain English, from which this article was excerpted.

Copyright 2016 by Craig Cochran. All rights reserved.