When I enter a company I look to see where it is from an analytics intelligence perspective. Analytics Intelligence is defined as:
Analytics Intelligence: The amount of understanding within an organization (at all levels) that show how deep data is being used to drive innovation and action leading to better, more effective outcomes.
- Is there a dedicated data team? (10 points)
A company that believes data is important will dedicate resources to that end. Embedded within this is a plan to expand that beyond where it currently is. The data team should be structured in a way that provides daily business value. You cannot expect the business to wait months for features. - How expansive are the metrics? Counts, Sums and Averages? (5 points)
Basic metrics are table stakes. As you grow, you start to understand the nuances of data capture and reporting. Averages are very misleading in data sets that have anomalistic characteristics. Ask yourself, “Does the business examine the data using median, mode and understand the expected variances in temporal analysis?”.
Additionally, how many objective metrics are you tracking? At most, you should have 10-15 company-defined metrics to track success. - Are they focused on exact counts at all levels? (10 points)
Companies that focus on expecting data to match exactly or “tie out” are missing the point. At a small business level, this is critical but as you get bigger and move to an enterprise space, the focus should be on patterns and trends, not small variances. Of course, if there are audits, there should be high levels of accountability. That is why audits take time. But to inject that level of scrutiny in day to day operations, slows down the business exponentially. It really is the difference between a $50M and a $500M company. Id throw in “Does the business provide Board/C-Level reporting in excel?” to this. Once excel isnt a method of consistent reporting, you take that next step. - Is there a data model? (10 points)
You should know how your data works. A business should have a published, consistent model for the data it collects and reports. If there isn’t a solid method for how data is organized, you are set up to fail. Abstracting data away from operational structure is pivotal to that next level of thinking. - Are analytics performed directly on operational databases / Is there a data warehouse? (10 points)
The business should not be running reporting or analytics on production databases. Not only does it impact customer performance but you can’t really do the types of analysis you need to do in that environment. - Is data available without an analyst or engineer? (5 points)
80-90% of all data should be easy to access with no analyst or engineer. Data should be accessible from a SaaS BI platform and updated frequently. This is a high value KPI to determine efficiency. Sometimes you might think a business needs more people, when in fact it just needs better tooling. - Is raw data consumed or does it flow through an automation tool for organizational use? (10 points)
Raw data should remain raw. It allows for reprocessing when needed. Instead, data should be ingested into the raw data lake and then transformed and loaded (as a new data set) into a centralized data warehouse. There it is consumed by other processes and made available to the business. - Does the business understand movement metrics (behavioral patterns)? (5 points)
A movement metric is 3 dimensional. A single value contains multiple pieces of information, not just a single value. You might have a value that displays a count of users on a specific day, but that count might be the number of users who moved from one category (28 days ago) to a new category today. This is behaviorally known as a movement metric/cohort. These are super powerful to understand where the business is trending. A single metric KPI that you can use for multiple things. - Does the business understand the value of cohort analysis? (7 points)
A key component to understanding user behavior is to group those users in different ways. Usually it starts by looking at time and then that leads to other characteristics. Ex: A group of users who joined the platform the week of Jan 1 – Jan 7. The major impact by viewing users as cohorts is you can track the impact of actions or features to see if there was any material change within those groups. - Is the data separated and tagged for segmentation analysis? (7 points)
A main component that drives innovation is the understanding of who your users are. If you segment your users, you can find those similarities and differences (correlations) within groups. - Is the data team involved in discussions on how data collection is instrumented? (7 points)
Usually one of the last things that companies do is to include data teams at the beginning. Good data teams can help development teams instrument products for better reporting. It is very easy to overlook this item in the day to day. - Does the business use A/B testing? (5 points)
Everyone likes to talk about it but not many actually implement it. This is the most impactful component to innovation. Try it out on a small portion of users, if successful try more, if not, adapt and rework, repeat. - Does the business use indexes instead of actual counts? (10 points)
The bigger you get, it is really hard to look at hundreds or even thousands of records to determine if something is performing well. At scale, companies usually roll numbers into index metrics and use algorithms to determine a baseline. For an example, look at the Dow Jones. The number is an index of strength in how all the stocks listed are doing. Do you really want to look at every stock in the list or can you just look at the trend of the index? - Does the business monitor and understand how the data and reports are being used? (10 points)
The ability to understand who is accessing the platform is a critical compliance component. For sanity, the business should have an easy way to look at the queries being run and who is making those queries. Monitoring performance is really about the freshness of data. At scale, being able to see when records are / are not flowing appropriately is important for reliability. - Does the business have real-time visibility (data observability) into the data pipeline? (5 points)
A new one to the list is data observability. This is a relatively new concept in the world of analytics. Sure, you could map the data infrastructure and its a crazy hard process. There are tools now, in the marketplace that have opened up this very important component. If you are a company that needs to give full view of data pipelines across the organization, you should implement a tool that gives this ability.
Scoring Model
When looking at this, there is a total of 116 points available. I rank this on a A – F scale.
A (104-116 pts) – Company has solid analytics intelligence and all the necessary components to be better than most organizations.
B (93 – 103 pts) – Company has good analytics intelligence and is better than the average organization.
C (81 – 92pts) – Company is ok but has some work to do. The average organization is in this range.
D (70 – 80pts) – Company is behind in analytics intelligence and growth is halted without adding tremendous amounts of people.
F (0 – 69pts) – Company is very far behind in analytics intelligence. Major changes and time is needed to create organizational change.
