In a project started at noventum we were set on making the performance of an IT department or company measureable. We embarked on an interesting journey along with 8 brave students from the DBIS group at the Westfählische Wilhelms-Universität in Münster, for whom this was their masters project seminar. I will post more on this project later, just one remark, they did a fabulous job.
What framework to base measurement on?
Most other participants in the project came from the business intelligence side of things, so I was the main responsible in deciding what kind of measurements framework should be used. As I am quite firm in the ITIL® arena I could immediately dismiss the initial idea of using ITIL® as a reference for measurements. I had only used COBIT as a reference in order to find proper target definitions as well as control objectives and activities for processes and it worked pretty well at that. Many in the IT service management arena prophesy a nearing war of the frameworks in which COBIT 5 “the delayed” might take on ITIL v3 2011 “the snobbish” (what is the penalty for heresy in castle ITIL®?). So why not check out the challenger and see how he performs in close combat?
Well at first the proposal of using COBIT was great. Instead of the long bullet lists with generic information COBIT has proper tables. COBIT is much more concise in its description and it has a structure for measurement in it.
In contrast to a simple “Book/Life Cycle” then “Process” arrangement (or Activity, or something, btw. how many processes?) COBIT has a goal hierarchy. This suited our ideas very well. It might make measuring more interesting and more meaningful if it would be sorted and organized in a hierarchy of goals (starting at the business goals, finally getting rid of this plaguing inside-out thinking).
All of these goals are directly and indirectly measured by metrics, some called Outcome Measures, some Business/IT or Process Metric and others Performance Indicator. Hey, the decision to use COBIT looks better and better and we just used information up to page 23! What marvels will lay ahead in this treasure trove of a framework?
Well, being on page 23 of COBIT was the issue. I knew that the really interesting parts of COBIT were to be found in the individual process chapters. I really dig the formal input and output definitions, I like the RACI chart (even if it is too high level for many practical aspects, but how should COBIT know the organisation of my current clients) and I often used the concise control activities. But now we wanted to go to a whole new level, we wanted to put COBIT in a complete measurement framework, we wanted to provide the “über-companion” to all COBIT related activities.
Which turned out to be the problem, since people putting things in systems need exact definitions. They need a clear relationships between all objects that will allways hold, they need a mathematical correct definition of your KPI or else you will pay penalty of breaking invariants.
The first issue was easily solved. We expected the goals to be in a proper hierarchy. But then the first analysis showed that from Business Goal level down the hierarchy was not a hierarchy. Each IT goal supports more than one business goal and somehow this is hidden in the COBIT documentation. No big deal, with the right definition (turning our hierarchy into a directed non-looping network) we retained the ability to calculate measures and summarize information (a network with loops will often reduce your ability to deduct anything to zero).
Exactness of COBIT
Next upon the agenda was extracting proper Business, IT, Process, Activity Goals and their Measurements (remember, Outcome, Goal Metrics and Performance Indicators. So let us have a look at a sample definition, using DS13 Manage Operations.
So you have 3 IT goals for DS13, which are measured by 2 metrics and driven by 3 metrics. Your 3 process goals are measure by 3 metrics (great!) and driven by 4 metrics (huh?) and the 2 activities are measured by 4 metrics (not driven at all). So anyone designing a system is going to ask you (the so called subject matter expert):
Which metric drives / measures which goal?
What is the (exact!) difference between a metric driving and a metric measuring a process?
OK, you can get through that. You have to rely on the CS™ ability. Common Sense™. Then you get on to defining the metrics, so that they can be measured correctly.
Number of downtime incidents and delays caused by deviating from operations procedures
Ehm, number of incidents and their delays? How do you “AND” these measures? Addition? Maximum? Weighted Average?
Percent of work schedules that are automated
OK, so the naive approach for this is storing 2 numbers. Number of “Total Number Work Schedules” and “Number of Automated Work Schedules”. By the way, what is a work schedule? Oh and another thing, which of these are outcome measures, goal/activity metrics and performance indicators? Do we need to separate them? And why is it that the description on page 149 lists the two IT goal metrics and one of the activities metrics and not the other 6 metrics? Oh and while you are at it, what indicator in your incident management database defines that an incident impacting a service level is an operational incident? I could continue here for hours.
So what do you do? You sweat over each of the measure and try define something useful. Something that is sometimes useful and can be measured. Or you go for some normally unmeasurable metrics and make them measurable, like the ratio of unauthorized changes to the infrastructure. Try find the unicorn.
Size does matter!
COBIT has 4 Perspectves, 34 Processes, 115 Activites and a mind-boggeling 340 listed KPIs (approximately, see below). Who manages by looking at 340 KPIs? We wanted to go from top to bottom at least for one business goal (17 in total listed), so we though 340/17 (naive, I know) should result in 20 KPIs beneath a single business goal. We focused on one business goal (BG4) which contained 2 IT goals (3 & 23), which was influenced by 10 processes, which resulted in quite a high number of activities. All of these contained 117 measurements. Let me repeat that: one hundred and seventeen different measurements beneath a single business goal out of 17, which for the reason of simplicity is only supported by 2 IT goals.
After you digested this a bit, let me mention that many of these measurements are not simple measurements, but calculated measurements which result from combining at least 2 different basic measurements (e.g. Percentages). Or other strange combinations seen above. A little chart here to visualize this problem:
After you somehow and unsatisfactorily solved all these issues, you get to a load of other things that make your life hard. Some are rooted in the way COBIT is working with metrics, some are just basic issues of how to combine several meaningful measures into one meaningful value for a goal metric without loosing too much information and without overwhelming the management user. More on this in later posts.
Well, in this project we have overcome a lot of these issues. We have used COBIT as a reference framework, but we have used CS™ more on the detail level.
Did COBIT help us with defining a measurement framework?
Yes it did. The overall accepted reference to COBIT makes it easier to present a solution. The predefined goal hierarchy (ehm, non-looping directed network, my graph theory classes are a long time away) was a big help in getting started.
Did COBIT make it more difficult defining a measurement framework?
Yes it did. The KPIs in the framework are quite frankly an untested mess created by people that have never gathered these measures in reality. They may have seen reports containing them, but they never had to gather the information. Many of the listed COBIT measures are not defined, use ambiguous terms, can not be computed, are not worth reporting this way, provide no or dubious value or are just plain nonsense.
So is COBIT practical enough for real world usage? In my opinion it does contain a lot of value but it is in version 4.1 far from a practical how-to. The danger is that it contains information that you can regard as practical advice like the KPIs which does not live up to the kindled expectations.
Re-reading this post leaves quite a few loose ends and some teasers, but I think that is a good thing. I will try to live up to that in the next weeks.
The project idea started 1,5 years ago, the students put so much effort into it that I can really justify a little tease. Any further ideas on measurement frameworks? Please comment, tweet, google+ message or an old fashioned email is really appreciated.