infosec-rusch

  Audit Assessment Tracking Matrix

I will be uploading assessment and audit management documents soon. These will manage and track scope, deliverables and project management initiatives.

  Kentucky Wildcats Top of the Mountain

Here’s a shout out to my Kentucky Wildcats for winning their 8th national title. Great game with little brother UL in final four, see you there next year. But I would have preferred to see a fully healthy UNC in the title game and had the chance to beat-up on DUKE along the way, but hey nothing is ever perfect.

Congrats to you coach Cal! lets go for 9 next year and 10 soon after!

GO BIG BLUE NATION

So Little Time

I apologize for not having posted anything new for several months, as a move last year and new employment have kept me buried. I will be posting a new methodology and process framework on how to conduct an audit/assessment. As I feel many auditors either have no defined process or have flawed ones.

At issue I think is the lack of client side experience many auditors have or lack of client audit management experience.

I myself having 7+ years working the client side of managing audits and assessments, and then before that working in the I.T. security ranks for close to 10 years,  has really helped me understand and create an audit framework and methodology that over time has evolved into a pretty good system. I will document this and post it in the next few weeks in sections.

Although the framework/mythology used is important to both parties (audited or auditor), as they both need to understand it and it needs to serve both of their needs. It is the entity/body being audited that it should always serve and be best suited for first. As the audited body is always the customer and the auditor is there to serve them first or foremost.

  What You Need To Look For in a QSA

This article applies to not only PCI assessments, but can be applied to any audit or risk assessment.

Choosing the right QSA prior to going into a PCI assessment is very critical, particularly if it is a merchant’s first formal onsite PCI assessment where the filling of a “Report On Compliance” (ROC) is the goal. Having managed 4 first time PCI onsite assessments, two at level 1 merchants and another at a very large level 2, I have experienced this first hand.

I have interviewed and experienced working with QSA’s that know and apply the subject material very well, but severely lack project and deliverable management capabilities. I have also experienced the reverse. So how does one identify the rare ones that have a solid but balanced PCI assessment game.

After my 5+ years and almost 6 PCI assessments, and multiple other audit and risk assessments, here is what I have learned;

1. Experience

I have found out that the size of a QSA firm doesn’t matter; I have worked with the big ones and little ones and had a mix of experiences with both. First start with the resume of the company itself, are they a BIG 4 that also performs PCI assessments on top of SOX 404, SAS70, HITECH CSF, or are they more specialized in the I.T. world and just perform PCI assessments, I.T audits, risk assessments.

Have they performed an onsite assessment with multiple other level 1 or large level 2 merchants where a ROC was produced? Can you get at least 5 references from these companies. How many (this may change after scoping) QSA’s are going to be formally assigned to the QSA team. In my experience if you are a level 1 or large level 2 merchant and this is your first onsite assessment, ask for at a minimum 3, but shoot for 4, this includes the team lead.

NOTE; if you are a level 1 or a large level 2 merchant and the QSA firm hasn’t been performing PCI assessments for at least 4 years, performed several with level 1 merchants where a ROC was produced and this is your first onsite assessment, best choose someone else, trust me on this one.

Also ask up front who the specific QSA’s that would be assigned to this engagement if they were selected. You will have to give them a window of when the assessment will be conducted so they know what resources they are going to have available. Then request a copy of their professional background and experience. Do your homework here, see if they have the skills and can truly perform the assessment but more importantly offer value beyond just assessing deliverables and speaking to PCI. Can they offer value in remediation strategies and other areas?

Ask what prior experience does the QSA have both in their past work experiences but also performing as a QSA. How many assessment have they performed in your industry, and working with the business processes and technology you use. These are highly critical things to look at, have they worked with the POS that you use, have they ever administered CISCO ASA’s like the ones that you have etc. Another thing to ask is what project management experience the lead QSA has. BIG ONE!

I have put together a QSA questionnaire and comparison matrix, although I have not updated it in 2 years, it still should provide a baseline for you to tweak and use as your own; PCI_QSA_Evaluation-Scoring-Matrix Template.

2. Scoping

This is absolutely without a doubt the most critical piece and what every other item after it depends on. This starts with what and how they ask questions about your business, its goals, what it does, how it does it, how is it organized and into what functional units. What type of governance framework do you use if any (i.e. Cobit, ISO), what other regulatory obligations do you have and so on.

Also what questions do they ask when you start to get into the weeds, the type of data you take in, data entry points, data types, who takes the data in, where does the data flow (data flow points), what state is the data in (i.e. image, encrypted, db, doc) and who has access to the data.

Do you have a scope management program or equivalent (change/asset management) that tracks what you have and where it is. Ask them how do they determine scope and manage it during the assessment. These are all things that they need allot from you but they should lead. Their the experts, or at least they should be. Ask them what framework or methodology they would recommend for you to track and manage your scope throughout the year (OH YEA, THATS A BIG ONE)!

3. Project Management

How do they plan on organizing and executing the assessment? What is the QSA leads experience in project management. Ask for a project plan, does it include kickoff meetings, weekly or bi-weekly status meetings, what are their plans for corporate and remote site visits, how do they recommend managing discovered gaps and remediation projects as part of the ongoing reviews.

What phases do they propose to organize the assessment in and ask for the details of each (mine are listed below);

1-Scoping-Onsite-Interviews
2-Corporate Governance-Policy Review
3-Onsite-Observations-Interviews
4-Business-Unit-Process-Application-Assessment
5-Penetration-External-Internal-Tests
6-Network-Server-DB-Assessments
7-Remediation-Projects-ReReview
8-ROC

4. Support

This is an area I see almost all of them lacking. Only a few recently have started to put together some of this type of supporting documentation. This is what led me to start authoring these types of documents years ago because of how incredibly helpful they are to you organization.

Ask (or just download mine) what type of whitepapers and PCI assessment support documents can they provide. For example, functional business unit governance documentation, documentation request guidance, deliverable request guidelines, individual PCI requirement intent summaries, onsite visit guidelines (i.e. call center, data center billing office).

This may seem like a small one, but when you’re sending out/calling/emailing 1000′s of requests to business owners and engineers, and you get flooded back with what do you want, what do you mean, I do not understand what your asking me. Then you’re going to wish you had these.

5. Deliverable Management

I don’t understand why most QSA firms I know or have gone through assessments with either do not have a deliverable management methodology/process or the ones that do have a really bad one. It’s just been recently (after 5 years of PCI-DSS) that some QSA firms are developing processes and in some cases using applications to manage this very heavy workload area.

Would a web based application be asking for too much, one that both the QSA and merchant could both use to track the overall PCI assessment and the 10,000′s of deliverables and their current status?

Ask the QSA how they plan on developing, requesting and tracking PCI deliverables. Ask what process and methodology do they employ to determine which PCI requirements are applicable to each specific business unit/group, its processes, individual applications, databases, access controls, etc., etc.

This gets back to how they plan on organizing the assessment and as such managing deliverables (Project Management). What status is the deliverable in, open, closed, requested, in remediation, ready for review, approved, declined, what? What is their normal or proposed system to manage this and do both parties have access to it. Does it have versioning, locking, etc., etc.

Thankfully with the help of an old PCI compliance manager of mine and a QSA best friend of mine, designed a tracking system and methodology to this madness (4 years ago). I have since then massively built upon it over the years and it has evolved as the needs have changed, and it has worked really well. You can find a version of this in the download section under PCI.

6. Metrics

I guess you could say this topic is more of subset of both the project management section, and to some extent the deliverable management section, hmm ok I guess it is but I wanted to break it off so that I could speak to it separately. As mentioned in the “Project Management” section we need to ask the QSA how they organize the assessment. But to get more specific what metrics within each phase do they incorporate to track the overall health of the assessment.

Additionally what metrics does the QSA incorporate to track deliverable status; totals of requested, current status (i.e. open, approved, non-compliant, ready for review) at the corporate level, individual business units, application groups and engineer/administrator levels. You should also decide what’s important for you as a merchant to track and be able to report on, both to upper management and underlying business groups and have these metrics included.

7. Reporting

This section feeds off sections 3 and 5. Section 3 for phase and timeline status and section 5 supplying the raw numbers we need to count an exact number and percentages. But ask the QSA what type of reporting do they provide, formats (executive summary, technical report), how often they can provide these reports and what your responsibilities are to them so that they can provide them in a timely fashion.

8. Remediation

This to me is where a QSA can really differentiate themselves from the rest and why section 1 “Experience” is so critical. As you move through any assessment or audit, especially if it’s your first true audit/assessment, you know you find all kinds of stuff (we had that data over there, we were doing what with our encryption keys).

Audits to me are like flashlights in a dark room, without them the place looks pretty clean and clutter free. But turn the flashlight on and start lifting up things and looking behind furniture and you can discover a different reality.

This is not the place to discuss how to manage discussions and debates with QSA’s n gaps, material risks, compensating controls, remediation options or how to make them see it your way (I’ve discussed this in other articles), but what real value can they bring beyond the assessment.

When it’s time to try to sit down multiple business units and engineers with competing interests and budgets, how do we create a forum to discuss remediation strategies and what experience can they bear to facilitate successful remediation projects. Pick their brain on thinking out of the box on developing remediation ideas that first supported the existing business process in place, met the intent of the compliance requirements but most importantly addressed true risks.

Make sure you document your expectations regarding remediation guidance and support in your RFP and final SOW so the QSA is obligated to offer assistance here. As the initial assessment winds down it tends to make them a bit more flexible when assessing deliverables.

  A Risk Based Approach

In the movie “Under Siege 2″ (not a great movie I know) their is a scene where the lead bad guy asked one of his subordinates if Steven Segal’s character is dead, the subordinate replies “Yes he’s dead”. The lead bad guy replies “Did you see the body”, the subordinate replies “No, I assumed . . . . . . “, the lead bad guy interrupts and says “Assumption is The Muther of all F*&#Up’s”. Later as the movie goes on Segal knocks off the bad guys one by one and ultimately foils their plans.

The moral of the story being simple, DON’T EVER ASSUME!

After I watched this movie years ago, I sat down and thought about the things in my life, both personally and professionally that have not gone as I had planned and/or blown up in my face, I quickly realized that the vast majority of them were caused by an assumption I made that didn’t turn out to be true. Now in my defense I can say as I have gotten older that I try to narrow my assumptions to ones that seem to be no brainers, but even those things many times have blowup.

Making assumptions is a human trait and required to move through the many decisions that we must deal with on a daily basis. However, their is a process one should apply in determining when to make an assumption and on managing the risks involved when making these assumptions.

1. Impact Analysis; The most important thing to first determine is the impact of the assumption you made not coming true. Its not a question of the likelihood of the bomb going off, its the damage created if it does. You need to define and understand what the impact of the assumption not being correct.
   …
2. Incident Response Plan; If the impact is critical, can cause irreparable damage, or will cause an unpredictable domino affect, you need to identify this and you need to have 2-3 backup plans documented and ready. One thing that causes me allot of frustration is not when someone comes to me telling me XYZ didn’t work out as planned. Its when I ask them what contingency plans or at least ideas they have to deal with the situation and they respond “I Don’t Know” or “I Don’t Have Any”.

I guess the moral of the story is not to be a glass half empty person, but to understand that when making assumptions, think things through carefully and always apply a risk based approach to everything . . . . . . . . . . . . including “Assumptions”!

  Fighting & Scratching For Security Controls

I know all the regulations we must all abide to (PCI, SOX, GLBA, HIPAA, whatever) are all flawed, and if controls are put in place badly or your paired with a bad auditor that does not know how to apply the regulations in real world environments, then yes they can be a nightmare. HOWEVER, having come from a hands on I.T. security admin background, scratching/fighting to get security controls in place for YEARS,  if it wasn’t for most of the security  regulations, 60% or more of the security controls etc. etc that have gone into place the last 5-8 years wouldn’t be in place.