GetApp’s Category Leaders

Methodology v6: April 2020 - Present

The Basics

Category Leaders rankings highlight top-ranked North American software products based on ratings from end-users in five key areas: ease of use, value for money, functionality, customer support, and likelihood to recommend.

To create each Category Leaders ranking, our research team may evaluate hundreds of products in any one category. Only the products with the highest scores become Category Leaders.

Each Category Leader is shown with a radar chart depicting that product’s scores for each of the five areas (shown in aqua blue) relative to the average scores of all products in the category (the dark blue lines).

Data sources include approved user reviews, public data sources, and data from technology vendors. Please refer to the GetApp Community Guidelines for more information.

Our Category Leaders reports use a snapshot of review data from a defined time frame and are not updated after publication. Category Leaders reports should therefore be used along with the detailed and current information available on each product’s profile page for the most up-to-date view of the data.

Inclusion Criteria

To be eligible for inclusion in a category, products must meet the following criteria:

  • Product has at least 20 unique product reviews published on GetApp within 24 months of the start of the research process for a given report. Two years of reviews provides a sample that we have determined is large enough and recent enough to be valuable to buyers. The criteria is also set low enough to ensure emerging vendors can be represented.

  • Product achieves a minimum normalized rating in each of the 5 key areas (ease of use, value for money, functionality, customer support, and likelihood to recommend) after application of the Category Leaders method for normalizing and weighting review recency (see description under “Scoring” section).

  • Product shows evidence of offering required functionality as demonstrated by publicly-available sources, such as the vendor’s website.

  • Product serves North American users, as demonstrated by product reviews submitted from that region.

  • Product is relevant to software buyers across industries or sectors—in other words, no “niche” solutions that cater exclusively to one specific type of user, as determined by our analysis of user reviews and/or market research.

Scoring

To arrive at a final score per product:

  • Scores are calculated for each of the five dimensions (ease of use, value for money, functionality, customer support, and likelihood to recommend) based on average user ratings. 

  • User ratings are collected using various scales (usually 1 to 5 or 0 to 10); all ratings scales are translated to a standard 5-point scale before use in our calculations.

  • Newer ratings are weighted more heavily, and products with more ratings are advantaged over those with fewer, to provide a sample large enough and recent enough to be valuable to buyers.

  • Scores for each dimension are scaled to a maximum of 20 points, resulting in a total maximum score of 100.

  • Category averages (represented by the dark line for each dimension) are calculated from the ratings for all products in the category.

Although two products may sometimes receive the same total score when rounding to the nearest integer, true ties are rare when we consider decimal points; however, in the event of a true tie, products with tying scores will be listed in alphabetical order.

Products are represented, included, and/or scored solely based on user reviews and independent of any relationship that GetApp has with vendors.

Methodology v5: June 2019 - Feb 2020

The Basics

Category Leaders rankings highlight key factors that small businesses should consider when assessing which software products may be right for them. These factors include user input (“user reviews”), compatibility with other systems (“integrations”), mobile offerings ("mobile"), technical capabilities (“functionality”), and product security features (“security”).

To create each Category Leaders ranking, our research team typically evaluates hundreds of products in any one category. The 10-15 with the highest scores become Category Leaders.

Since Category Leaders is intended to cover a given category at large, the Category Leaders team use their market experience and knowledge, existing market-based research, and small business software buyer needs analysis to assess an application’s suitability for that category depending on whether it can reasonably be expected to be relevant to most software buyers across industries searching for a system in the category.

To be eligible for inclusion in a category, products must:

  • Be sold in the North American software market

  • Offer required functionality as determined by our research analysts, who provide coverage of and have familiarity with products in each category. Each product must have all core features plus a majority of common features for that market.

  • Have at least 10 unique user-submitted product reviews across the three Gartner Digital Markets web properties —softwareadvice.com, capterra.com, and getapp.com— published within 12 months of the start of the analysis period for a given report’s update.

Product scores are calculated using five unique data points, each scored out of 20, resulting in a total maximum score of 100. In the event of a tie, each data point is weighted for importance, with security taking precedent, followed by reviews, integrations, mobile apps, and functionality, respectively.

Products are scored independent of any relationship that GetApp has with vendors.

Methodology Details

User Reviews.

Review scores are calculated using reviews collected from GetApp and its sister sites, Software Advice and Capterra. This score is based on average user ratings (on a scale from 1 to 5) of how satisfied users are with a product overall. The scoring includes ratings from reviews within a 12-month analysis period. The methodology calculates the percentile ranking for each product relative to all other products in the software category that have qualified for inclusion. That percentile ranking is then translated into a 1 to 20 score.

Integrations.

The total possible score for integrations is 20 points. Five points are awarded if the product has an open API; up to 15 points are awarded based on an assessment of the product’s integrations.

As part of this integrations assessment, our research team defines a list of the most commonly cited integrations in a given software category. (E.g., in the Project Management software category, products may commonly integrate with Zapier or G Suite.) Each product is scored according to how many of those common integrations it offers, based on publicly available data. When a product lists an integration with a suite of applications (e.g. Microsoft Office), we assume it integrates with each component application comprising that suite (e.g. Microsoft Word, Microsoft Outlook, etc.). The more common a specific integration is, the more heavily it is weighted in our scoring.

Mobile Platforms.

Mobile scores are calculated based on assessing the availability of a native iOS app, a native Android app, and a mobile web app. When native iOS or Android apps are present,the score factors in user reviews of those apps. Products can earn a maximum possible score of 20 points. If a product offers no native iOS or Android app and no mobile web app, that product receives 0 points for their mobile score.

Functionality.

Functionality scores are assigned based on a combination of two assessments: user ratings for “Features” and functionality breadth data. Fifty percent of the score is based on average user ratings on a scale from 1 to 5 of how satisfied users are with the product’s features and functionality. The scoring includes ratings left within the 12-month analysis period.

The methodology calculates the percentile ranking for each product relative to all other products in the software category that have qualified for inclusion. That percentile ranking is then translated into a 1 to 10 score.

The other 50% of the score is based on a functionality breadth assessment of the product. For each software category, our research analysts define a core set of features that a product must offer, as well as additional common and optional features. Products are assessed according to how many of those pre-defined features they offer, and products with more features receive a higher score. We use this data to calculate a product's percentile ranking, which allows us to determine how products compare relative to one another rather than determine an absolute number.

All told, a vendor can receive up to 10 points based on user ratings for features and up to 10 points based on the functionality breadth assessment for a total maximum score of 20 for “functionality.”

Security.

Security scoring is calculated using answers from a vendor-completed survey. The questions in the survey are based on the Cloud Security Alliance self-assessment form as part of the Security, Trust & Assurance Registry, supplemented by the expertise of our security research experts. Each answer in the 7-question survey is assigned a point value. The scoring then calculates the percentile ranking for each product relative to all other products in the software category that have qualified for Category Leaders consideration. That percentile ranking is then translated into a 1 to 20 score. Our research analysts may audit vendor responses to the security survey for accuracy.

Vendors update their security information each year or certify that the information is still correct. In the event a vendor has not completed their security survey by time of publishing, they will automatically be assigned a baseline security score that reflects a set of security offerings that the average software vendor is reasonably expected to have, according to our security research experts.

Data

Data sources include published user reviews, public data sources and data from technology vendors. The user-generated product reviews data incorporated into Category Leaders are collected from submissions to all three Gartner Digital Markets sites (see above section for details). As a quality check, all reviews are verified and moderated prior to publication. Please refer to the GetApp Community Guidelines for more information.


Methodology v4: July 2018 - January 2019

The Basics

Category Leaders rankings highlight key factors that small businesses should consider when assessing which software products may be right for them. These factors include user perception (“user reviews”), compatibility with other systems (“integrations”), mobile offerings ("mobile"), technical capabilities (“functionality”), and product security (“security”).

To create each Category Leaders ranking, our research team typically evaluates hundreds of products in any one category. The 10-15 with the highest scores become Category Leaders.

Category Leaders is focused on the North American software markets. To be eligible for inclusion in a category, products must offer a core set of functionality determined by our research analysts, who provide coverage of and have familiarity with products in each category.

Since Category Leaders is intended to cover a given category at large, individual analysts use their market experience and knowledge, existing market-based research, and small business software buyer needs analysis to assess an application’s suitability for that category depending on whether it can reasonably be expected to be relevant to most small business buyers across industries searching for a system in the category.

In addition, a product must have at least 20 unique user-submitted product reviews across the three Gartner Digital Markets web properties: softwareadvice.com, capterra.com, and getapp.com—and those reviews must be published within 18 months of the start of the analysis period for a given quarter’s update.

An app’s score is calculated using five unique data points, each scored out of 20, resulting in a total maximum score out of 100. The ranking and scoring methodology ensure complete impartiality and independence from any relationships that GetApp has or doesn’t have with app vendors.

More Methodology Details

User Reviews.

Review scores are calculated using weighted reviews collected from GetApp and its sister sites, Software Advice and Capterra. This score is based on average user ratings (on a scale from 1 to 5) of how satisfied users are with a product overall. The scoring includes ratings left within an 18-month analysis period; ratings that are more recent are weighted more heavily. The methodology calculates the percentile ranking for each product relative to all other products in the software category that have qualified for inclusion. That percentile ranking is then translated into a 1 to 20 score.

Integrations.

Integration scores factor in a number of variables. Up to 16 points are awarded based on the number of integrations an app has. The first 10 points are awarded based on the number of integrations relative to the category average, while 4 points are awarded based on the number of integrations relative to the leaders in the category. An additional 5 points is awarded for an integration with Zapier, a widely used software integration platform, while an extra point is given for having an open API. The total possible score for integrations is 20 points.

Mobile Platforms.

Mobile scores are calculated based on the availability of an iOS and Android app. Five points are awarded for the presence of each app, with an additional 10 points awarded for the number and average rating of reviews in both the App Store and Google Play. These review scores are calculated on a scale based on the category average for a maximum of 5 points per platform and an overall maximum score of 20.

If there isn't a native iOS or Android app, the product's website is assessed using Google's PageSpeed Insights to measure mobile usability. A score out of 100 is given, then scaled to a score out of 5 for a maximum of 5 points.

Functionality.

Functionality scores are assigned based on a combination of two assessments: user ratings and functionality breadth data.

Fifty percent of the score is based on average user ratings on a scale from 1 to 5 of how satisfied users are with the product’s functionality. The scoring includes ratings left within the 18-month analysis period; ratings that are more recent are weighted more heavily.

The methodology calculates the percentile ranking for each product relative to all other products in the software category that have qualified for inclusion. That percentile ranking is then translated into a 1 to 10 score.

The other 50% of the score is based on a functionality assessment of the product. For each software category, our research analysts define a core set of features that a product must offer, as well as additional “nice-to-have” common and optional features.

Products are assessed according to how many of those pre-defined features they offer, and products with more features receive a higher score. We use this data to calculate a product's percentile ranking, which allows us to determine how products compare relative to one another rather than determine an absolute number.

All told, a vendor can receive up to 10 points based on user ratings and up to 10 points based on the features assessment for a total maximum score of 20 for “functionality.”

Security.

Security scoring is calculated using answers from a vendor-completed survey. The questions in the survey are based on the Cloud Security Alliance self-assessment form as part of the Security, Trust & Assurance Registry, supplemented by the expertise of our security research experts. Each answer in the 16-question survey is assigned a point value. The scoring then calculates the percentile ranking for each product relative to all other products in the software category that have qualified for Category Leaders consideration. That percentile ranking is then translated into a 1 to 20 score.

In the event of a tie, each data point is weighted for importance, with security taking precedent, followed by reviews, integrations, mobile apps, and functionality, respectively.


Methodology v3: 2017

GetApp's Category Leader ranking is for business owners and decision makers looking for a comprehensive list of the leading cloud-based software products in the industry.

The ranking serves as a point of reference highlighting key factors that small businesses should look at when assessing a piece of software. These factors include user perception (reviews), compatibility (integrations and mobile compatibility), market presence (media presence), and security features (security).

How are the scores calculated?

An app's score is calculated using five unique data points, each scored out of 20, giving a total score out of 100. The ranking and scoring methodology ensure complete impartiality and independence from any relationships that GetApp has with app vendors.

User Reviews.

Review scores are calculated using reviews collected from GetApp and its sister sites, Software Advice and Capterra. Scores are calculated based on the total number, average rating, and recency of reviews.

The number of reviews and average user rating are scaled against the category average. A weighting is then applied to reward recency of user reviews: reviews written in the last 6 months score up to 10 points, those written between 6 and 12 months ago score up to 6 points, and those more than 12 months old score up to 4 points. These numbers are added together to give a total maximum score of 20.

Integrations.

Integration scores factor in a number of variables. Up to 16 points are awarded based on the number of integrations an app has; the first 10 points are awarded based on the number of integrations relative to the category average, while 4 points are awarded based on the number of integrations relative to the leaders in the category. An additional 5 points is awarded for an integration with Zapier, a widely used software integration platform, while an extra point is given for having an open API. The total possible score for integrations is 20 points.

Mobile Platforms.

Mobile scores are calculated based on the availability of an iOS and Android app. 5 points are awarded for the presence of each app, with an additional 10 points being awarded for the number and average rating of reviews in both the App Store and Google Play. These review scores are calculated and scaled against the competition for a maximum of 5 points per platform and an overall maximum score of 20.

If there isn't a native iOS or Android app, the product's website is assessed using Google's PageSpeed Insights to measure mobile usability. The score is returned out of 100, then scaled to a score out of 5 for a maximum of 5 points.

Media Presence.

Media presence is calculated using the number of Twitter followers and Facebook fans that an app has. The number of fans and followers are added together, and each app is scored based on this overall number, compared with others in the category. The first 15 points are awarded for the combined number of followers and fans scaled against the category average, while the final 5 points are awarded for the combined number of followers and fans scaled against the top leaders in the category. These two scores are added together for a total out of 20 points.

Security.

Security scoring is calculated using answers from a vendor-completed survey. The questions in the survey are based on the Cloud Security Alliance self-assessment form as part of the Security, Trust & Assurance Registry. With 15 questions, each answer is assigned a point value based on vendor response, with additional points given for security certifications, for a maximum possible score of 20.

In the event of a tie, each data point is weighted for importance, with security taking precedent, followed by reviews, integrations, mobile apps, and media presence, respectively.


Methodology v2: 2016

GetApp’s quarterly ranking showcases the top 25 apps based in the cloud in a given software category. Each app is scored using five factors, worth 20 points each, for a total possible score out of 100. These factors include user reviews, integrations, mobile app availability, media presence, and security. Each app’s score is independent of commercial interests and existing relationships that GetApp has with app vendors.

How are the scores calculated?

An app’s score is calculated based on five unique data points and is completely independent of any relationships that GetApp has with app vendors. The data points are calculated as outlined below:

User Reviews.

The score for user reviews is calculated using a Bayesian estimate, which is a weighted average that includes the number of reviews and the rating of an app, benchmarked against other apps in the category. This raw number is then scaled to represent a value between one and 20.

Integrations.

Integration scores are calculated by scaling and adding two different averages for the number of listed integrations on GetApp: the first 15 points are awarded for the total number of integrations scaled against the average for the top 25 percent of apps in the category, while the final five points are awarded for the total number of integrations scaled against the leader in the category.

Mobile Platforms.

Mobile platform scores are calculated using data from Google Play and the Apple App Store. Five points are awarded for having an Android App, and another five for an iOS app. The additional 10 points are calculated based on the user ratings of the app in each store, rated out of five, for a total of 20 points.

Media Presence.

Media presence is calculated using the number of Twitter followers and Facebook fans a vendor has. Each social network contributes a maximum of 10 points to the total score. The first five points are awarded for total number of followers scaled against the average number of followers that each vendor in the category has, while the remaining five points are awarded based on the total number of followers, this time scaled against the category leaders. The scores for Facebook and Twitter are added together for a total score out of 20.

Security.

Security is calculated using answers from a vendor-completed survey. The questions in the survey are based on the Cloud Security Alliance self-assessment form as part of the Security, Trust & Assurance Registry. With 15 questions, each answer is assigned a point value based on vendor response, with additional points given for security certifications, for a maximum possible score of 20.

In the event of a tie, each data point is weighted for importance, with security taking precedent, followed by reviews, integrations, mobile apps, and media presence, respectively.


Methodology v1: 2015

GetApp’s quarterly ranking showcases the top 25 apps based in the cloud in a given software category. Each app is scored using five factors, including user reviews, integrations, mobile app availability, media presence, and security. Each app’s score is independent of commercial interests and existing relationships that GetApp has with app vendors.

User Reviews.

The GetRank score for user reviews is calculated using a Bayesian estimate, which is a weighted average that includes the number of reviews and the rating of an app, benchmarked against other apps in the category. This raw number is then scaled to represent a value between 1 and 20.

Integrations.

Integration scores are calculated by scaling and adding two different averages for the number of listed integrations on GetApp: the first 15 points are awarded for the total number of integrations scaled against the average for the top 25 percent of apps in the category, while the final 5 points are awarded for the total number of integrations scaled against the leader in the category.

Mobile Platforms.

Mobile platform scores are calculated using data pulled from Google Play and the Apple App Store. In this case, a maximum of 5 points is attributed for having an Android App, and another 5 for an iOS app. The additional 10 points are calculated based on the ratings of the app in each store, rated out of 5, for a total of 20 points.

Media Presence.

Using Brandwatch as a source of data, GetApp analyzed the number of social media mentions of each application and vendor over the course of the quarter. In order to scale the volume of mentions and filter for irrelevant content, the Twitter handle of the vendor was used as the query search in Brandwatch.

Similar to the scoring for integrations, the first 15 points are awarded for the total mentions scaled against the average of the top 25 apps in the category, while the final 5 points are awarded for total mentions scaled against the category leader.

Security.

Security is calculated with data collected from the GetRank security survey. The survey and scoring were developed in collaboration with Microsoft. The questions in the survey are based on the Cloud Security Alliance self-assessment form as part of the Security, Trust & Assurance Registry. With roughly 70 questions, each question is weighted for importance, and each answer is given a score. Security certifications are also scored and weighted based on importance. The raw score is calculated by totalling the app’s certification and questionnaire score, then benchmarking against the top scoring app in the category, turning that proportion into a score out of 20.


Information for Vendors About Category Leaders

External Usage Guidelines

Providers must abide by the Category Leaders External Usage Guidelines when referencing Category Leaders content. Except in digital media with character limitations, the following disclaimer MUST appear with any/all Category Leaders reference(s) and graphic use:

GetApp Category Leaders constitute the subjective opinions of individual end-user reviews, ratings, and data applied against a documented methodology; they neither represent the views of, nor constitute an endorsement by, GetApp or its affiliates.

Escalation Guidelines

We take the integrity of our research seriously. If you have questions or concerns about Category Leaders content, the methodology, or other issues, you may contact the Category Leaders team at DigitalMarketsResearch@gartner.com. Please provide as much detail as you can, so we can understand the issues, review your concerns and take appropriate action.

Tentative Publication Schedule

Please visit our full publication schedule here. Note: This schedule is intended to be used as a guide only and is subject to change at any time at our discretion.

FAQs

Do software providers need to notify you if they’d like their product(s) to be considered?

No, we will automatically consider all products with profiles on our site and that meet our eligibility criteria in a given category .

Do you need any additional information from providers to create the reports?

No, at this time we don’t need nor accept any information directly from our software and SaaS provider community. We obtain all the information we need via our internal databases and publicly available data.

Does it matter which of the Gartner Digital Markets sites — Capterra, GetApp or Software Advice — reviews are published on?

No, reviews are shared across all three sites and all are considered equally in our research .

When will you publish [x] category next?

Please refer to our research calendar for a tentative schedule of reports by month and category. Note: This schedule is intended to be used as a guide only and is subject to change at any time at our discretion.

Are software providers running paid campaigns with GetApp more likely to place higher on reports?

No. A vendor’s status as a client or non-client is not a factor in any way.

How can providers improve their products’ placement on reports?

While software and SaaS providers cannot directly influence their position in reports, the best way to ensure a product is considered is to collect more reviews.

Can you update a product’s placement in a recent report if that product recently received a number of new reviews?

Currently, our reports are published snapshots in time as denoted by the specific publish date on the report. We do not make changes after they have been published unless there is a significant error in its production. Any such products and new reviews will be considered when we next update the report. Please refer to our research calendar for our tentative schedule of reports by month and category.