For years, there was primarily one number that mattered for law firms: profits per partner. While that remains an important measure of the financial health of the firm, law firms are discovering that other indicators can be important in assessing their health and guiding their development.
As law firm managers have become increasingly sophisticated in identifying useful performance metrics for the firm as a whole, my conversations with law firm knowledge managers suggest that there may not be a correspondingly nuanced view of performance metrics for knowledge management (KM). To be honest, some elements of KM work may seem very hard to quantify. (For example, do you know the actual impact of a model document on time saved, knowledge transferred, or risk averted?) Consequently, many KM professionals and their firms have taken the stance that not only is it impossible to identify meaningful performance metrics for KM, but it may not even be worth the effort to try. Further, for years law firms have funded KM work based on the belief that it was the right thing to do, so those firms did not look too closely into how their money was being spent. In the absence of a nuanced understanding of metrics, they settled for evidence of activity rather than impact, to justify additional KM expenditures.
However, heightened cost management by clients and firms means that using KM because it just makes sense no longer cuts the mustard. Now we need to provide evidence not only of activity, but of impact as well. We must take a closer look at the available types of metrics and think harder about how they might shed light on our KM activities. At the end of the day, the point of using metrics is to help tell a story that enables law firm partners and managers to understand better the value of KM operations, and to do so in a way that is firmly grounded in verifiable data.
To assist in this goal, this article seeks to expand the reader's knowledge about possible metrics and how they might best be used to tell the story of the efficacy of KM work in a law firm.
Before delving into the details, it is important to state some caveats:
- This article is not a recommendation for management by metrics above all else. Rather, it advocates using a variety of metrics to shed more light on KM operations.
- Metrics flow from goals. In other words, if you do not know what you want to achieve operationally, you run the risk of choosing an inappropriate metric.
- Metrics alone are insufficient. If you want metrics to be useful within your KM operations, you must clearly communicate to your team what your goals are and how particular metrics further those goals. Without this context, people tend to work blindly towards the short-term numbers and, in the process, may miss the long-term goals. For example, you may have staff trying to maximize the number of requests handled without realizing the real goal is to maximize the level of internal client satisfaction.
- Measurement changes behavior. This is known as the Hawthorne Effect, in which the fact of being observed (or measured) changed the behavior of the people being observed. In some cases, it has been found to contribute to a short-term increase in productivity. Therefore, be careful about how you collect and communicate metrics.
- Secure your baseline. If you are going to measure anything, start by establishing the baseline. Otherwise, it will be hard to prove the before and after case.
Expand Your Toolkit
There is no single metric that tells the whole story of your KM work. Different projects may require different metrics. Different outcomes will certainly require different metrics. Therefore, KM professionals should expand their toolkit by developing an understanding of a range of metrics. For each metric, the KM professional must learn what that metric does, how it is collected, and how it is most safely interpreted. With these new tools in hand, it becomes much easier to select an available metric that is suited to the task of showing whether or not you have achieved a specific outcome.
There are four types of metrics that should be in each KM professional's toolkit, namely:
- Activity metrics
- Operational metrics
- Behavior metrics, and
- Outcome metrics.
Of these, activity metrics are the easiest to identify and collect. For that reason, they are often the first metric people think of to measure KM impact. What does an activity metric do? It tells you how many users and uses there are of your KM products and services.
Many online systems now automatically provide backend statistics that give a sense of activity. For example, backend data in an intranet or portal would tell you (among other things) number of visitors, length of visits, and number of page views.
The single biggest drawback to activity metrics is that while they may tell you what is happening, they rarely can tell you why it is happening. Without knowing why, the other data are not terribly helpful — unless, of course, you have unsophisticated firm management who are willing to settle for activity rather than impact.
Nonetheless, there are times when activity metrics are meaningful and perfectly appropriate. For example, during pilot phases or launch phases, activity metrics may be indicators of early adoption.
If activity metrics show usage at a particular time, operational metrics show usage over a span of time. In other words, operational metrics show trends. The data underlying the operational trends are likely to be the same backend statistics used for activity metrics. Since the set of data required is collected over several time periods, this is not a metric that can be displayed at the very beginning of a project. However, it is important to track and graph these sets of data over time to see what patterns emerge.
Like activity metrics, operational metrics can tell you what is happening, but they cannot tell you why. Consequently, they serve as a launching pad for further inquiry when interesting trends emerge on your graph.
Behavior metrics show how and why KM efforts are changing behaviors. While activity and operational metrics may point to possible behavior change, the best way of obtaining evidence and an explanation of behavior change is to conduct interviews or surveys, or to shadow internal clients. The key question to ask is: What are you doing differently now?
Outcome metrics are the most meaningful of all these metrics because they demonstrate the actual impact a particular KM initiative is having. While some outcomes may be inferred from activity and operational metrics, the more direct way of identifying outcomes is to conduct interviews. If the question for behavior metrics was "What are you doing differently now?" the question for outcome metrics is "What difference does this new behavior make?" In other words, "What new result are you achieving?"
Once you have the interview responses, the key question for senior KM professionals and law firm management is: How do these results contribute to operational excellence? Once you have established a link between a KM initiative and operational excellence, and based that link on verifiable data, then you finally have a useful story to tell about your KM efforts.
Provided you do not fall into the trap of management by metrics (where the sole goal is to hit specific targets), you will find that a wider range of metrics thoughtfully chosen, tracked, and interpreted can give you a better understanding of the impact of your KM efforts on firm excellence. This is a prerequisite to educating firm management on the value of KM efforts. When you can affirm that KM is the right thing to do, and then show that you are doing KM right, then you will have delivered something of value to your firm.
Back to Contents