Senior Content Analyst at Gartner
Let's face it. Nowadays, most startups and midsize businesses do not use the same traditional business intelligence (BI) and data warehousing (DW) as before. Traditional BI and DW models just can't meet the needs of resource-strapped, quickly-growing companies while decision-makers require answers from their data as soon as possible in order to make important business decisions.
Mostly found in large organizations with more than 1,000 employees, BI departments are usually dependent on separate teams to handle the DW component and infrastructure. The BI team operates on top of the data structures built by the DW team, enabling the BI team to focus on aspects of business intelligence such as defining KPIs, reporting, and dashboarding.
DW teams, on the other hand, are typically responsible for maintaining the data warehouse where collective data from the company resides, which includes all data from various departments and sources. They also handle defining the metadata (the data definitions from the source data); implementing the extract, transform, and load (ETL) processes; and working with the BI stakeholders to provide data for their reporting needs.
DW teams typically implement traditional data warehousing concepts and architecture in order to create a robust data environment. The issue with these project life cycles is that they usually last for years, not months or even days. With the complexity needed to implement some of these architectures, you can imagine the coordination required between the BI and DW teams as reporting and data increases within a company.
The problem for smaller companies and startups is that data professionals (data engineers, data analysts, and data scientists) are often in a situation where they do not have the luxury of having an entire team to implement a traditional data warehouse that is perfectly aggregated and cleaned for their analytics needs. Sometimes there is a BI manager to field requests, but often not. Typically analysts are bombarded with requests from managers across the organization from various departments (marketing, sales, product, finance) asking for specific data. Many times, these business users only have a question they want answered and do not know the complexities of what the request entails, so the data analyst must become resourceful in order to handle requests properly.
This means that analysts will need to work with data that is not, at the current state, suitable for reporting. Issues arising may include:
Traditionally, these issues would be handled by a data warehousing team. In a traditional BI department, if the data is not available, then the request will be punted over to the DW team and put into a project plan, which could takes months or even years. Business stakeholders at startups, on the other hand, require answers as soon as possible—sometimes even yesterday.
Optimize the infrastructure
Many databases are moving to the cloud, which means there is less and less reliance on a data warehousing team to maintain a database. Providers such as Amazon Web Services (AWS) and Microsoft Azure now allow you to spin up a database instance immediately at a very low cost. You no longer need a full-time database administrator for your analytical needs.
If you’re a small business or startup, consider using a cloud infrastructure to spin up your necessary hardware and allow you to analyze your data quickly, often at a fraction of the cost of on-premise solutions.
Using the right tools
In order to quickly and efficiently respond to requests, the resourceful data professional will rely on the tools available to them at the time. Modernizing for BI means using tools that are not outdated and being able to answer analytical questions as quickly and efficiently as possible. Traditional tools for analyzing data, such as Excel, will only get you so far; there are limitations when looking at data only on a spreadsheet. What, then, if you also need to examine data that is in Google Analytics or your platform data in MySQL with over a million rows?
With some modern BI tools, the reliance on data warehousing diminishes. Many new BI and data visualization tools, such as Tableau and Power BI, have connectors to various data sources that remove the ETL step. For example, you can connect directly to Salesforce where these tools automatically create the metadata from Salesforce objects into the OLAP view, all ready for drag-and-drop analysis. Typically, data integrations would require data engineering help to integrate and set up ETL processes, but now, with modern BI tools, many of these steps are no longer needed.
If you want to adopt a modern BI approach, ask yourself these three key questions when reviewing BI tools for your company:
Look at your team and see where you can optimize inefficiencies in your current data environment. For example, which processes take the most time and are the most expensive? Talk with your business users to see how modern BI tools and technologies might be able to help.
There are many considerations when it comes to budget constraints and resource capacity to take on new changes. However, in a long-term perspective, and if you’re a small and nimble data team, consider leveraging modern BI to put you in a better position to harness your data. It is a worthwhile investment in the long run.
Business Intelligence (BI) professionals share their success and failure stories with you.
Each app is scored using 5 factors, worth 20 points each, for a possible total of 100.