GetApp’s Category Leaders

Methodology v5: June 2019 - Present

The Basics

Category Leaders rankings highlight key factors that small businesses should consider when assessing which software products may be right for them. These factors include user input (“user reviews”), compatibility with other systems (“integrations”), mobile offerings ("mobile"), technical capabilities (“functionality”), and product security features (“security”).

To create each Category Leaders ranking, our research team typically evaluates hundreds of products in any one category. The 10-15 with the highest scores become Category Leaders.

Since Category Leaders is intended to cover a given category at large, the Category Leaders team use their market experience and knowledge, existing market-based research, and small business software buyer needs analysis to assess an application’s suitability for that category depending on whether it can reasonably be expected to be relevant to most software buyers across industries searching for a system in the category.

To be eligible for inclusion in a category, products must:

  • Be sold in the North American software market
  • Offer required functionality as determined by our research analysts, who provide coverage of and have familiarity with products in each category. Each product must have all core features plus a majority of common features for that market.
  • Have at least 10 unique user-submitted product reviews across the three Gartner Digital Markets web properties —softwareadvice.com, capterra.com, and getapp.com— published within 12 months of the start of the analysis period for a given report’s update.

Product scores are calculated using five unique data points, each scored out of 20, resulting in a total maximum score of 100. In the event of a tie, each data point is weighted for importance, with security taking precedent, followed by reviews, integrations, mobile apps, and functionality, respectively.

Products are scored independent of any relationship that GetApp has with vendors.

Methodology Details

User Reviews. Review scores are calculated using reviews collected from GetApp and its sister sites, Software Advice and Capterra. This score is based on average user ratings (on a scale from 1 to 5) of how satisfied users are with a product overall. The scoring includes ratings from reviews within a 12-month analysis period. The methodology calculates the percentile ranking for each product relative to all other products in the software category that have qualified for inclusion. That percentile ranking is then translated into a 1 to 20 score.

Integrations. The total possible score for integrations is 20 points. Five points are awarded if the product has an open API; up to 15 points are awarded based on an assessment of the product’s integrations.

As part of this integrations assessment, our research team defines a list of the most commonly cited integrations in a given software category. (E.g., in the Project Management software category, products may commonly integrate with Zapier or G Suite.) Each product is scored according to how many of those common integrations it offers, based on publicly available data. When a product lists an integration with a suite of applications (e.g. Microsoft Office), we assume it integrates with each component application comprising that suite (e.g. Microsoft Word, Microsoft Outlook, etc.). The more common a specific integration is, the more heavily it is weighted in our scoring.

Mobile Platforms. Mobile scores are calculated based on assessing the availability of a native iOS app, a native Android app, and a mobile web app. When native iOS or Android apps are present,the score factors in user reviews of those apps. Products can earn a maximum possible score of 20 points. If a product offers no native iOS or Android app and no mobile web app, that product receives 0 points for their mobile score.

Functionality. Functionality scores are assigned based on a combination of two assessments: user ratings for “Features” and functionality breadth data. Fifty percent of the score is based on average user ratings on a scale from 1 to 5 of how satisfied users are with the product’s features and functionality. The scoring includes ratings left within the 12-month analysis period.

The methodology calculates the percentile ranking for each product relative to all other products in the software category that have qualified for inclusion. That percentile ranking is then translated into a 1 to 10 score.

The other 50% of the score is based on a functionality breadth assessment of the product. For each software category, our research analysts define a core set of features that a product must offer, as well as additional common and optional features. Products are assessed according to how many of those pre-defined features they offer, and products with more features receive a higher score. We use this data to calculate a product's percentile ranking, which allows us to determine how products compare relative to one another rather than determine an absolute number.

All told, a vendor can receive up to 10 points based on user ratings for features and up to 10 points based on the functionality breadth assessment for a total maximum score of 20 for “functionality.”

Security. Security scoring is calculated using answers from a vendor-completed survey. The questions in the survey are based on the Cloud Security Alliance self-assessment form as part of the Security, Trust & Assurance Registry, supplemented by the expertise of our security research experts. Each answer in the 7-question survey is assigned a point value. The scoring then calculates the percentile ranking for each product relative to all other products in the software category that have qualified for Category Leaders consideration. That percentile ranking is then translated into a 1 to 20 score. Our research analysts may audit vendor responses to the security survey for accuracy.

Vendors update their security information each year or certify that the information is still correct. In the event a vendor has not completed their security survey by time of publishing, they will automatically be assigned a baseline security score that reflects a set of security offerings that the average software vendor is reasonably expected to have, according to our security research experts.

Data

Data sources include published user reviews, public data sources and data from technology vendors. The user-generated product reviews data incorporated into Category Leaders are collected from submissions to all three Gartner Digital Markets sites (see above section for details). As a quality check, all reviews are verified and moderated prior to publication. Please refer to the GetApp Community Guidelines for more information.

External Guidelines

Providers must abide by the Category Leaders External Usage Guidelines when referencing Category Leaders content. Except in digital media with character limitations, the following disclaimer MUST appear with any/all Category Leaders reference(s) and graphic use:

GetApp Category Leaders constitute the subjective opinions of individual end-user reviews, ratings, and data applied against a documented methodology; they neither represent the views of, nor constitute an endorsement by, GetApp or its affiliates.

Escalation Guidelines

We take the integrity of our research seriously. If you have questions or concerns about Category Leaders content, the methodology, or other issues, you may contact the Category Leaders team at DigitalMarketsResearch@gartner.com. Please provide as much detail as you can, so we can understand the issues, review your concerns and take appropriate action.


Methodology v4: July 2018 - January 2019

The Basics

Category Leaders rankings highlight key factors that small businesses should consider when assessing which software products may be right for them. These factors include user perception (“user reviews”), compatibility with other systems (“integrations”), mobile offerings ("mobile"), technical capabilities (“functionality”), and product security (“security”).

To create each Category Leaders ranking, our research team typically evaluates hundreds of products in any one category. The 10-15 with the highest scores become Category Leaders.

Category Leaders is focused on the North American software markets. To be eligible for inclusion in a category, products must offer a core set of functionality determined by our research analysts, who provide coverage of and have familiarity with products in each category.

Since Category Leaders is intended to cover a given category at large, individual analysts use their market experience and knowledge, existing market-based research, and small business software buyer needs analysis to assess an application’s suitability for that category depending on whether it can reasonably be expected to be relevant to most small business buyers across industries searching for a system in the category.

In addition, a product must have at least 20 unique user-submitted product reviews across the three Gartner Digital Markets web properties: softwareadvice.com, capterra.com, and getapp.com—and those reviews must be published within 18 months of the start of the analysis period for a given quarter’s update.

An app’s score is calculated using five unique data points, each scored out of 20, resulting in a total maximum score out of 100. The ranking and scoring methodology ensure complete impartiality and independence from any relationships that GetApp has or doesn’t have with app vendors.

More Methodology Details

User Reviews. Review scores are calculated using weighted reviews collected from GetApp and its sister sites, Software Advice and Capterra. This score is based on average user ratings (on a scale from 1 to 5) of how satisfied users are with a product overall. The scoring includes ratings left within an 18-month analysis period; ratings that are more recent are weighted more heavily. The methodology calculates the percentile ranking for each product relative to all other products in the software category that have qualified for inclusion. That percentile ranking is then translated into a 1 to 20 score.

Integrations. Integration scores factor in a number of variables. Up to 16 points are awarded based on the number of integrations an app has. The first 10 points are awarded based on the number of integrations relative to the category average, while 4 points are awarded based on the number of integrations relative to the leaders in the category. An additional 5 points is awarded for an integration with Zapier, a widely used software integration platform, while an extra point is given for having an open API. The total possible score for integrations is 20 points.

Mobile Platforms. Mobile scores are calculated based on the availability of an iOS and Android app. Five points are awarded for the presence of each app, with an additional 10 points awarded for the number and average rating of reviews in both the App Store and Google Play. These review scores are calculated on a scale based on the category average for a maximum of 5 points per platform and an overall maximum score of 20.

If there isn't a native iOS or Android app, the product's website is assessed using Google's PageSpeed Insights to measure mobile usability. A score out of 100 is given, then scaled to a score out of 5 for a maximum of 5 points.

Functionality. Functionality scores are assigned based on a combination of two assessments: user ratings and functionality breadth data.

Fifty percent of the score is based on average user ratings on a scale from 1 to 5 of how satisfied users are with the product’s functionality. The scoring includes ratings left within the 18-month analysis period; ratings that are more recent are weighted more heavily.

The methodology calculates the percentile ranking for each product relative to all other products in the software category that have qualified for inclusion. That percentile ranking is then translated into a 1 to 10 score.

The other 50% of the score is based on a functionality assessment of the product. For each software category, our research analysts define a core set of features that a product must offer, as well as additional “nice-to-have” common and optional features.

Products are assessed according to how many of those pre-defined features they offer, and products with more features receive a higher score. We use this data to calculate a product's percentile ranking, which allows us to determine how products compare relative to one another rather than determine an absolute number.

All told, a vendor can receive up to 10 points based on user ratings and up to 10 points based on the features assessment for a total maximum score of 20 for “functionality.”

Security. Security scoring is calculated using answers from a vendor-completed survey. The questions in the survey are based on the Cloud Security Alliance self-assessment form as part of the Security, Trust & Assurance Registry, supplemented by the expertise of our security research experts. Each answer in the 16-question survey is assigned a point value. The scoring then calculates the percentile ranking for each product relative to all other products in the software category that have qualified for Category Leaders consideration. That percentile ranking is then translated into a 1 to 20 score.

In the event of a tie, each data point is weighted for importance, with security taking precedent, followed by reviews, integrations, mobile apps, and functionality, respectively.

External Usage Guidelines

Providers must abide by the Category Leaders External Usage Guidelines when referencing Category Leaders content. Except in digital media with character limitations, the following disclaimer MUST appear with any/all Category Leaders reference(s) and graphic use:

GetApp Category Leaders constitute the subjective opinions of individual end-user reviews, ratings, and data applied against a documented methodology; they neither represent the views of, nor constitute an endorsement by, GetApp or its affiliates.


Methodology v3: 2017

GetApp's Category Leader ranking is for business owners and decision makers looking for a comprehensive list of the leading cloud-based software products in the industry.

The ranking serves as a point of reference highlighting key factors that small businesses should look at when assessing a piece of software. These factors include user perception (reviews), compatibility (integrations and mobile compatibility), market presence (media presence), and security features (security).

How are the scores calculated?

An app's score is calculated using five unique data points, each scored out of 20, giving a total score out of 100. The ranking and scoring methodology ensure complete impartiality and independence from any relationships that GetApp has with app vendors.

User Reviews. Review scores are calculated using reviews collected from GetApp and its sister sites, Software Advice and Capterra. Scores are calculated based on the total number, average rating, and recency of reviews.

The number of reviews and average user rating are scaled against the category average. A weighting is then applied to reward recency of user reviews: reviews written in the last 6 months score up to 10 points, those written between 6 and 12 months ago score up to 6 points, and those more than 12 months old score up to 4 points. These numbers are added together to give a total maximum score of 20.

Integrations. Integration scores factor in a number of variables. Up to 16 points are awarded based on the number of integrations an app has; the first 10 points are awarded based on the number of integrations relative to the category average, while 4 points are awarded based on the number of integrations relative to the leaders in the category. An additional 5 points is awarded for an integration with Zapier, a widely used software integration platform, while an extra point is given for having an open API. The total possible score for integrations is 20 points.

Mobile Platforms. Mobile scores are calculated based on the availability of an iOS and Android app. 5 points are awarded for the presence of each app, with an additional 10 points being awarded for the number and average rating of reviews in both the App Store and Google Play. These review scores are calculated and scaled against the competition for a maximum of 5 points per platform and an overall maximum score of 20.

If there isn't a native iOS or Android app, the product's website is assessed using Google's PageSpeed Insights to measure mobile usability. The score is returned out of 100, then scaled to a score out of 5 for a maximum of 5 points.

Media Presence. Media presence is calculated using the number of Twitter followers and Facebook fans that an app has. The number of fans and followers are added together, and each app is scored based on this overall number, compared with others in the category. The first 15 points are awarded for the combined number of followers and fans scaled against the category average, while the final 5 points are awarded for the combined number of followers and fans scaled against the top leaders in the category. These two scores are added together for a total out of 20 points.

Security. Security scoring is calculated using answers from a vendor-completed survey. The questions in the survey are based on the Cloud Security Alliance self-assessment form as part of the Security, Trust & Assurance Registry. With 15 questions, each answer is assigned a point value based on vendor response, with additional points given for security certifications, for a maximum possible score of 20.

In the event of a tie, each data point is weighted for importance, with security taking precedent, followed by reviews, integrations, mobile apps, and media presence, respectively.


Methodology v2: 2016

GetApp’s quarterly ranking showcases the top 25 apps based in the cloud in a given software category. Each app is scored using five factors, worth 20 points each, for a total possible score out of 100. These factors include user reviews, integrations, mobile app availability, media presence, and security. Each app’s score is independent of commercial interests and existing relationships that GetApp has with app vendors.

How are the scores calculated?

An app’s score is calculated based on five unique data points and is completely independent of any relationships that GetApp has with app vendors. The data points are calculated as outlined below:

User Reviews. The score for user reviews is calculated using a Bayesian estimate, which is a weighted average that includes the number of reviews and the rating of an app, benchmarked against other apps in the category. This raw number is then scaled to represent a value between one and 20.

Integrations. Integration scores are calculated by scaling and adding two different averages for the number of listed integrations on GetApp: the first 15 points are awarded for the total number of integrations scaled against the average for the top 25 percent of apps in the category, while the final five points are awarded for the total number of integrations scaled against the leader in the category.

Mobile Platforms. Mobile platform scores are calculated using data from Google Play and the Apple App Store. Five points are awarded for having an Android App, and another five for an iOS app. The additional 10 points are calculated based on the user ratings of the app in each store, rated out of five, for a total of 20 points.

Media Presence. Media presence is calculated using the number of Twitter followers and Facebook fans a vendor has. Each social network contributes a maximum of 10 points to the total score. The first five points are awarded for total number of followers scaled against the average number of followers that each vendor in the category has, while the remaining five points are awarded based on the total number of followers, this time scaled against the category leaders. The scores for Facebook and Twitter are added together for a total score out of 20.

Security. Security is calculated using answers from a vendor-completed survey. The questions in the survey are based on the Cloud Security Alliance self-assessment form as part of the Security, Trust & Assurance Registry. With 15 questions, each answer is assigned a point value based on vendor response, with additional points given for security certifications, for a maximum possible score of 20.

In the event of a tie, each data point is weighted for importance, with security taking precedent, followed by reviews, integrations, mobile apps, and media presence, respectively.


Methodology v1: 2015

GetApp’s quarterly ranking showcases the top 25 apps based in the cloud in a given software category. Each app is scored using five factors, including user reviews, integrations, mobile app availability, media presence, and security. Each app’s score is independent of commercial interests and existing relationships that GetApp has with app vendors.

User Reviews. The GetRank score for user reviews is calculated using a Bayesian estimate, which is a weighted average that includes the number of reviews and the rating of an app, benchmarked against other apps in the category. This raw number is then scaled to represent a value between 1 and 20.

Integrations. Integration scores are calculated by scaling and adding two different averages for the number of listed integrations on GetApp: the first 15 points are awarded for the total number of integrations scaled against the average for the top 25 percent of apps in the category, while the final 5 points are awarded for the total number of integrations scaled against the leader in the category.

Mobile Platforms. Mobile platform scores are calculated using data pulled from Google Play and the Apple App Store. In this case, a maximum of 5 points is attributed for having an Android App, and another 5 for an iOS app. The additional 10 points are calculated based on the ratings of the app in each store, rated out of 5, for a total of 20 points.

Media Presence. Using Brandwatch as a source of data, GetApp analyzed the number of social media mentions of each application and vendor over the course of the quarter. In order to scale the volume of mentions and filter for irrelevant content, the Twitter handle of the vendor was used as the query search in Brandwatch.

Similar to the scoring for integrations, the first 15 points are awarded for the total mentions scaled against the average of the top 25 apps in the category, while the final 5 points are awarded for total mentions scaled against the category leader.

Security. Security is calculated with data collected from the GetRank security survey. The survey and scoring were developed in collaboration with Microsoft. The questions in the survey are based on the Cloud Security Alliance self-assessment form as part of the Security, Trust & Assurance Registry. With roughly 70 questions, each question is weighted for importance, and each answer is given a score. Security certifications are also scored and weighted based on importance. The raw score is calculated by totalling the app’s certification and questionnaire score, then benchmarking against the top scoring app in the category, turning that proportion into a score out of 20.


Published Categories and Tentative Publication Schedule

This schedule is intended to be used as a guide only and is subject to change. Six to eight weeks before an update is published, our research team sends a notification to all vendors listed in one of the categories we cover with information about the specific month an update will publish, the reviews analysis time period for the research and a product survey to inform our fact-base.

Any category that was last updated in January 2019 follows Methodology v4. Starting in June 2019, Category Leaders updates will follow Methodology v5.

CategoryLast UpdateNext Update
AccountingJune 2019June 2020
Applicant TrackingJune 2019June 2020
Business IntelligenceAugust 2019August 2020
Business ManagementMarch 2019Q1 2020
Call CenterMarch 2019Q1 2020
CRMJune 2019June 2020
Customer ServiceAugust 2019August 2020
Document ManagementMarch 2019October 2019
Email MarketingMarch 2019September 2019
Employee SchedulingMarch 2019Q1 2020
ERPAugust 2019August 2020
Event ManagementMarch 2019November 2019
Expense ManagementMarch 2019November 2019
Facilities ManagementJuly 2019July 2020
Field ServiceMarch 2019September 2019
HRAugust 2019August 2020
Instant Messaging & ChatMarch 2019Q1 2020
Inventory ManagementJuly 2019July 2020
ITSMMarch 2019September 2019
Live ChatMarch 2019October 2019
LMSMarch 2019Q1 2020
Marketing AutomationMarch 2019Q2 2020
Point of SaleMarch 2019September 2019
Project ManagementMarch 2019October 2019
Project Portfolio ManagementMarch 2019Q2 2020
Reservation & Online BookingMarch 2019November 2019
Social Media MarketingMarch 2019October 2019
SurveyMarch 2019Q2 2020
Time Tracking & ExpenseMarch 2019Q1 2020