Implementing modern BI practices within your organization

Business Intelligence (BI)

Let's face it. Nowadays, most startups and midsize businesses do not use the same traditional business intelligence (BI) and data warehousing (DW) as before. Traditional BI and DW models just can't meet the needs of resource-strapped, quickly-growing companies while decision-makers require answers from their data as soon as possible in order to make important business decisions.


The traditional way of BI

Mostly found in large organizations with more than 1,000 employees, BI departments are usually dependent on separate teams to handle the DW component and infrastructure. The BI team operates on top of the data structures built by the DW team, enabling the BI team to focus on aspects of business intelligence such as defining KPIs, reporting, and dashboarding.

DW teams, on the other hand, are typically responsible for maintaining the data warehouse where collective data from the company resides, which includes all data from various departments and sources. They also handle defining the metadata (the data definitions from the source data); implementing the extract, transform, and load (ETL) processes; and working with the BI stakeholders to provide data for their reporting needs.

DW teams typically implement traditional data warehousing concepts and architecture in order to create a robust data environment. The issue with these project life cycles is that they usually last for years, not months or even days. With the complexity needed to implement some of these architectures, you can imagine the coordination required between the BI and DW teams as reporting and data increases within a company.

Challenges of traditional BI

The problem for smaller companies and startups is that data professionals (data engineers, data analysts, and data scientists) are often in a situation where they do not have the luxury of having an entire team to implement a traditional data warehouse that is perfectly aggregated and cleaned for their analytics needs. Sometimes there is a BI manager to field requests, but often not. Typically analysts are bombarded with requests from managers across the organization from various departments (marketing, sales, product, finance) asking for specific data. Many times, these business users only have a question they want answered and do not know the complexities of what the request entails, so the data analyst must become resourceful in order to handle requests properly.

This means that analysts will need to work with data that is not, at the current state, suitable for reporting. Issues arising may include:

  • Unclean data: Ubiquitous syntax issues and grammatical errors. Syntax and grammatical errors are prevalent throughout dirty data that often arrives “as-is” to analysts.
  • Unaggregated data: Data not aggregated for the level of reporting that is required.
  • Disparate data sources: The data needed lives in a data source which is not integrated in the current environment.

Traditionally, these issues would be handled by a data warehousing team. In a traditional BI department, if the data is not available, then the request will be punted over to the DW team and put into a project plan, which could takes months or even years. Business stakeholders at startups, on the other hand, require answers as soon as possible—sometimes even yesterday.

Implementing modern BI

Optimize the infrastructure

Many databases are moving to the cloud, which means there is less and less reliance on a data warehousing team to maintain a database. Providers such as Amazon Web Services (AWS) and Microsoft Azure now allow you to spin up a database instance immediately at a very low cost. You no longer need a full-time database administrator for your analytical needs.

If you’re a small business or startup, consider using a cloud infrastructure to spin up your necessary hardware and allow you to analyze your data quickly, often at a fraction of the cost of on-premise solutions.

Using the right tools

In order to quickly and efficiently respond to requests, the resourceful data professional will rely on the tools available to them at the time. Modernizing for BI means using tools that are not outdated and being able to answer analytical questions as quickly and efficiently as possible. Traditional tools for analyzing data, such as Excel, will only get you so far; there are limitations when looking at data only on a spreadsheet. What, then, if you also need to examine data that is in Google Analytics or your platform data in MySQL with over a million rows?

With some modern BI tools, the reliance on data warehousing diminishes. Many new BI and data visualization tools, such as Tableau and Power BI, have connectors to various data sources that remove the ETL step. For example, you can connect directly to Salesforce where these tools automatically create the metadata from Salesforce objects into the OLAP view, all ready for drag-and-drop analysis. Typically, data integrations would require data engineering help to integrate and set up ETL processes, but now, with modern BI tools, many of these steps are no longer needed.

If you want to adopt a modern BI approach, ask yourself these three key questions when reviewing BI tools for your company:

  • Can I consolidate and analyze data from my various data sources in one place? You should be able to connect directly to data sources. This includes:
    • Relational databases: MySQL, PostgreSQL, Oracle, SQL Server
    • Flat files: Excel, CSVs
    • Servers: Many modern BI providers have integrated servers and application stores directly
    • Cloud databases: For example, Amazon Redshift
    • NoSQL databases
  • Can I import large amounts of data? The limitation with older tools such as Excel is that it will begin to bog down when you reach a certain row threshold. Modern BI tools have instead been built to work with many millions of rows.
  • Can I answer ad hoc questions on the fly? With the consolidation of data sources, modern BI tool interfaces are built for data visualization, not just looking at numbers. This allows you to analyze data quickly and to visualize trends and outliers immediately.

Considerations

Look at your team and see where you can optimize inefficiencies in your current data environment. For example, which processes take the most time and are the most expensive? Talk with your business users to see how modern BI tools and technologies might be able to help.

There are many considerations when it comes to budget constraints and resource capacity to take on new changes. However, in a long-term perspective, and if you’re a small and nimble data team, consider leveraging modern BI to put you in a better position to harness your data. It is a worthwhile investment in the long run.

Buyers guide
Thomas LaMonte
Thomas LaMonte
Senior Content Analyst at Gartner
This buyers guide will help you understand the key features of business intelligence and analytics (BI) software, its benefits and challenges, information on who uses BI software, and industry trends.
Case studies
Thomas LaMonte
Thomas LaMonte
Senior Content Analyst at Gartner
Our Business Intelligence (BI) Analyst shares real-world case studies of BI implementations.