Methodology

More than 34,000 people participated in the Developer Ecosystem Survey 2020. This report is based on the input of 19,696 developers from 18 countries, weighted by several criteria as described in the following paragraphs. Every graph in the report was built on data from no fewer than 300 responses obtained for each point.

To shorten the survey and reduce its response burden, some sections were shown to respondents randomly. There were six randomized sections, of which each respondent saw only two:

1. Continuous Integration, Issue Tracking, and VCS
2. Testing
3. DevOps and Hosting
4. Static analysis, Open-source, etc.
5. Education
6. Cross-platform and Microservices

For example, if a respondent selected Tester / QA Engineer or DevOps Engineer / Infrastructure Developer as their job role, they would inevitably be given one definite section about their job role, plus one other section selected randomly.

To invite potential survey respondents to complete the survey, we used Twitter ads, Facebook ads, Instagram, Quora, Vkontakte, Codefund, Baidu, and JetBrains’ own communication channels.

In our ad campaigns, we used two targeting waves. The first wave targeted specific countries, while the second one targeted rarely used programming languages to reduce the variance in the sections corresponding to those programming languages. We targeted Ruby, Scala, Rust, Swift, and Objective-C. We also asked our respondents to share the link to the survey with their peers.

To minimize bias, the report is based on the data weighted with regard to responses coming from Twitter ads, Facebook ads, Instagram, Quora, Vkontakte, Codefund, Baidu ads, and respondents’ referrals.

We took into account each respondent’s source individually to generate the results based on the weighting procedures. We performed three stages of weighting to get a less biased picture of the worldwide developer population.

First weighting stage: populations of professional developers in 18 countries

In the first stage, we assembled the responses collected while targeting different countries, and then we applied our estimations of the populations of professional developers in each country to these data.

First, we took the survey data on professional developers and working students that came from ads posted on various social networks in the 18 targeted countries, together with the data that came from various peer referrals. Then we weighted all these responses according to our estimated populations of professional developers in those 18 countries. This ensured that the distribution of the responses corresponded to our estimates of the numbers of professional developers in each country.

Second weighting stage: the proportions of currently employed and unemployed developers

In the second stage, we forced the proportion of students and unemployed respondents (that came to us through the same external ad campaigns) to be 17% in every country. We did this to maintain consistency with the previous year’s methodology, as that is the only estimate of their populations we have available.

As a result, we had a distribution of 10,116 responses from external sources weighted by country and employment status.

Third weighting stage: employment status, programming languages, JetBrains products usage

The third stage was rather sophisticated, as it included calculations obtained by solving systems of equations. We took those weighted 10,116 responses. For developers from each country, besides their employment status, we calculated the shares for each of the 30+ programming languages, as well as the shares for those who answered “I currently use JetBrains products” and “I have never heard of JetBrains or its products”. Those shares became constants in our equations.

The next step was to add two more groups of responses from other sources: JetBrains internal communication channels, such as JetBrains social networks accounts and our research panel, and social-network ad campaigns targeted at certain programming languages users. This brought 9,580 more responses, which we weighted to keep all those shares the same.

Solving the system of 30+ linear equations and inequalities

We composed a system of 30+ linear equations and inequalities that described:

• The weighting coefficients for the respondents (for example, Pierre from our sample represents on average 180 software developers from France).
• The specific values of their responses (Pierre uses C++, he is fully employed, and he has never heard of JetBrains).
• The necessary ratios among their responses (for example, 27% of developers have used C++ in the past 12 months, and so on).

In order to solve this system of equations with the minimum variance of the weighting coefficients (which is important!), we used the dual method of Goldfarb and Idnani (1982, 1983), which helped us collate optimal individual weighting coefficients for the respondents.

Despite these measures, some bias is likely present, as JetBrains users might have been more willing on average to complete the survey.

Also, our community ecosystem is developing, and there might be some data fluctuations despite our weighting stages and efforts. For instance, the share of Kotlin users who compile their applications for JVM has grown in our data owing to Kotlin/JVM bias in our sources, although there have been no changes to the overall share of the Kotlin language.

We will continue to update and improve our weighting methodology in the future. Stay tuned for our DevEco 2021!