Home	About Gamma	Tour our Web Site	Events	White Papers	Services	Visitors' Book	How to contact us
				IMS	Internal Control	ISMS	Smart Cards	Common Criteria

Measuring the effectiveness of an internal control system

by Dr. David Brewer and William List, CA, Hon FBCS

The objective of this paper is to propose a methodology by which management can measure the effectiveness of the organisation's Internal Control System (ICS).  In addition the paper proposes a methodology for recording Risk Treatment Plans (RTPs), which improve the communication between risk specialists and senior management.  This methodology incorporates our concepts for classifications of ICSs.

The ICS is the way in which the management deploys the organisation's resources to achieve the organisation's objectives. 

The ICS exists in two basic parts:

• Procedures to perform the work necessary to conduct the organisation's business.  These are called operational procedures.

• Procedures to ensure that the business is conducted as expected. These are called controls.

It is this second part of the ICS which this paper examines.

All organisations have an ICS. In large organisations it is formalised; in the very small organisations it is often implemented by the boss being involved everywhere.  Most organisations are somewhere in between these two extremes. 

It is axiomatic that things will go wrong - people do not always perform as expected, great new products do not sell as well as expected, criminals attack the organisation, acts of God occur, etc.  This has always been the case.  The conundrum facing management is to decide how much resource to deploy to create just sufficient controls to limit the possibility of bad events occurring and to limit the damage when they do occur.

When an organisation outgrows the ability of the boss to supervise everything management have sought to resolve the conundrum by applying (a series of) risk assessments.  In these assessments the probable events are identified and appropriate actions to limit damage are determined.

The question "Is this an optimum deployment?"  still remains whatever controls are in place and however the need for them has been identified.  The methodology we propose seeks to assist management in answering the question. It allows management to determine by direct measurement whether or not their actual ICS is achieving the objectives they want, irrespective of what else is happening in the world.  In other words, the measurement is neither conditional on the frequency or other characteristics of events nor how damaging the resulting impacts might be.  It allows management to measure improvements in the ICS and to tune it for overall cost-effectiveness.

In summary, we propose to measure the operational effectiveness of the control part of the ICS using various time metrics.   In particular we propose to determine for each event the times relative to the time at which the event occurred (which we describe in our model as Te):

• The time of detection (TD if detected by the ICS, or if detected by some other means TM, e.g. reported in a newspaper)

• The time that the damage caused by the event is fixed (TF), should it be possible and appropriate to fix it, or otherwise resolve the problem

• The time limit after which (TW), if the damage is not fixed, some impact penalty I_P (whether financial or otherwise) is incurred.

We use the time measure because it is independent of the volume of events (which are totally variable given the threat environment) and independent of the value of events (which is random).  It allows us to classify the controls as belonging to one of seven classes.  We use these to determine the operational effectiveness of the ICS, which for convenience we express as belonging to one of five categories. We also use the time measures, coupled with frequency, to measure improvement in the ICS. Finally, in order to optimise the cost-effectiveness of the ICS, we introduce a set of financial metrics (or substitute metrics if financial measurement is inappropriate):

• The costs of normal operations - performing the work to achieve the business objectives (which we describe in our model as cost of doing business - C_BA)

• The costs of the controls of whatever form - access control, buildings insurance, business continuity planning, IT recovery procedures, etc. (which we describe in our model as cost of the ICS - C_ICS )

• The financial impact of any events that do occur (which we describe in our model as the impact penalty I_P)

• The costs of fixing or otherwise resolving the damage caused by the event (which we describe in our model as the cost of fixing the event C_F).

Having optimised the operational effectiveness of the ICS, a set of inequalities using the financial metrics then allows us to tune the ICS for cost-effectiveness.

Note that those procedures which are created to facilitate recovery from an event or to minimise the impact of an event are described in this paper as a Business Continuity Plan (BCP).

In practice, an ICS addresses many different types of event, and the optimum controls for each one could fall into any one of the seven different categories.  Thus a real ICS may have controls belonging to each and every category.  We therefore propose a methodology for choosing the optimum controls for an ICS that must address a wide variety of different events and impacts.

The remainder of the paper is divided as follows:

• The next section presents the background to this paper

• We then recount some true stories and anecdotes that provide a foundation to our theory

• We next present the fundamental model

• We define the control classes and categories of ICS

• We then describe how to measure/monitor operational effectiveness and improvements, and tune the ICS for cost-effectiveness, with the aid of some worked examples

• We then present our methodology for generating RTPs

• We finally we present our conclusions.