The Fundamental Flaw in Your SOC 2 Compliance Checklist

Companies seeking to obtain a SOC 2 report are often in a hurry. So, what's wrong with searching for a SOC 2 Compliance Checklist?  Maybe this whole SOC 2 examination thing can be a quick, simple matter of working through a checklist and obtaining a report?  Not quite.

Although we specialize in helping companies prepare for SOC 2 examinations -- and we've gone to great lengths to ensure that we are attuned to the most common security controls that SOC 2 auditors tend to evaluate -- auditors are required (for good reason) to maintain independence.  That means that no provider (not Havoc Shield or anyone else) can supply the perfect checklist of items that is sure to lead to a clean SOC 2 report (an "unmodified opinion").

In the AICPA's Words

Here's how the AICPA describes the leeway that auditors have when "Designing and Performing Tests of Controls":

The service auditor is responsible for determining the nature (how the controls are tested), timing (when the controls are tested and the frequency of the testing), and extent (the number of procedures performed or the size of the sample) of procedures necessary to obtain sufficient appropriate evidence about the operating effectiveness of controls throughout the period.

Put simply, no one can predict precisely how a particular auditor will choose to test any particular control in any particular engagement.  That decision involves judgement.  The judgement of an auditor, who by obligation is independent.  Let's dig deeper.

SOC 2 Compliance Checklist Flaws

Auditor's Ability to Determine the Nature of Tests

Let's take an example of a case where an auditor is evaluating the company's security controls around revoking access when an employee is terminated.  The auditor might choose to do any of the following:

1. Request and inspect a log that records the access grants and revocations, and cross-reference that with records related to employee termination dates
2. Perform an inspection a day after an employee termination and request evidence that demonstrates that the former employee's access has been revoked as required.
3. Ask the human resources manager to supply their written procedures such as employee termination checklists
4. Observe the process of personnel performing access grants and revocations, and compare it with the stated processes

How can a SOC 2 compliance checklist be sure which of these actions (or many other alternatives) a particular auditor will find most appropriate for a particular engagement? The answer is, there isn't a reasonable way to predict this while still honoring the requirement that auditors remain independent.

At Havoc Shield, we work to prepare our clients for the types of procedures that we most frequently see auditors follow -- a pattern-recognition that is earned by paying close attention to what we see and hear as companies we serve go through the SOC 2 examination process.

Auditor's Ability to Determine the Timing of Tests

Let's take an example of a case where an auditor seeks to determine if employees are following physical security processes that require a visitor log to be maintained for anyone accessing the company's physical facilities.

During a six month SOC 2 examination (the Type 2 flavor), perhaps they'll ask for the current Visitor Log on week 5.  Or, maybe they'll ask on week 10?  Maybe they'll ask every 4 weeks?  Maybe they'll just ask for the current visitor log right in the moment that they visit the office for a site visit, wanting to see that the log is being kept in realtime.

Whatever the auditor decides in terms of timing, will reflect a judgement call made by the auditor that is well within the guidance they've received from the AICPA that gives them the flexibility to determine timing and frequency.

Auditor's Ability to Determine the Extent of Tests

Suppose there is a security control where "sampling" is the most practical way to observe operational effectiveness.  For example, if the company's policies indicate that failed system logins will be reviewed for malicious activity, and that conclusions about failed login reviews will be documented in a particular workbook.

The auditor could conclude that they'll choose to look through 1000s of rows of those logs.  Or 10 rows.  Or just ask to observe the output of a particular meeting where a set of malicious activity was reviewed.  Maybe they'll want the meeting minutes.  Maybe they'll want the agenda.  It's tough to say.

SOC 2 Compliance Checklist - All Hope is NOT LOST

Although the above examples are factually correct -- auditors could chose to use any of the above approaches -- our experience is that SOC 2 auditors generally aren't in the profession because they want to see you squirm.  They want to ask for reasonable evidence, that allows them to arrive at an informed opinion.  Not to ruin your month (or year) in the process.  We love growing the network of SOC 2 auditors that we know, because with each additional audit that we're exposed to, we get more and more attuned to the norms of what auditors are most likely to ask for.  And there is nothing wrong with having a checklist of items that we've observed as being the most likely items to come up in an audit.  In fact, that's exactly what we've done in Havoc Shield.  Just know that no one -- not Havoc Shield or anyone else -- can or should be able to predict precisely what a particular auditor will end up asking for.