The team

Medicamentalia Vaccines is a journalistic investigation by Civio into the current situation of vaccination worldwide, combining data analysis with field research in a range of countries. This is the sequel to our first investigation, focused on access to medicines, which was awarded Investigation of the Year (small newsroom) in the Data Journalism Awards and the Gabriel García Márquez Journalism Prize (Innovation category).

This second investigation, like the first, has benefited from a grant from Journalism Grants, funded by the Bill and Melinda Gates Foundation, which also funds big projects related to vaccination, such as GAVI. Neither Journalism Grants nor the Bill and Melinda Gates Foundation influenced the project or its journalistic investigation. In fact, like all the other readers, they did not have access to its contents until it was published.

Medicamentalia is the work of the members of Civio in conjunction with an array of valued collaborators, in a range of fields, from a variety of countries. We are pleased to introduce you to the team.

• Eva Belmonte. Lead journalist, research, database creation and analysis, field research in Central America, reporting and editing.
• Raúl Díaz Poblete. Developer. Web development, interface design, data analysis and visualization.
• Miguel Ángel Gavilanes. Journalist. Research, database creation and analysis, field research in South Africa and reporting.
• Javier de Vega. Journalist. Research, reporting, editing and communication.
• David Cabo. Developer. Data mining, database creation and analysis.
• Amir Campos. Project Management.
• Manuel Penados. Filming and photography in Guatemala and El Salvador.
• Denise Pimentel. Video editing in Central America.
• Serusha Govender and Anne Gonschorek. Filming in South Africa.

Collaborators:

• Philip Smart. Spanish-English translation
• Asier Andrés. Research support in Guatemala.
• Julio Hernández. Film sound in El Salvador.
• Santi Cogolludo. Photos in Barcelona.

Acknowledgements

This project would not have been possible without the initial grant from Journalism Grants. We would like to thank the European Journalism Center for sponsoring this research, as they did for the first part of the Medicamentalia project, and their infinite patience and understanding with the problems that delayed the launch; especially to Diana Lungu who was always ready to lend a hand.

Our media partners also gave the green light to this project before it became a reality. Thanks to Correctiv, El Mundo, Euronews, La Sexta, The Bureau of Investigative Journalism, Il Sole and Euro Scientist for their faith and support. Without them, we would reach a far smaller audience and much of this work would have been in vain. Hugo Garrido, Verónica Ramírez and Hristio Boytchev helped us to adapt the contents to their media.

To examine a subject as complex as immunization in greater depth, we needed to get seriously involved in the topic. We read dozens of scientific papers and hundreds of existing articles on the subject. Many of them are referenced in the study. Thanks to all the researchers and specialist journalists that led the way for us, such as Javi Salas, from El País, author of numerous articles on the subject. Thanks also to the friends that helped us along the way, such as Ximena Villagrán in Guatemala.

Thanks to everyone that gave up their time and knowledge in the interviews, both those in the preliminary research and those that appear in the reports, for their generosity. During the process, we received important awards that gave us a powerful boost. We would like to thank the juries and the teams at the Data Journalism Awards and the Gabriel García Márquez Journalism Prize for this recognition.

Methodology

Preliminary research

As with all Civio projects, the first step forwards in the investigation implied a necessary step backwards - we had to take a plunge unto the subject. Whilst in the first part of the Medicamentalia project we focused on researching access to medicines (prices, patent systems, etc.), in this second phase we wanted to investigate vaccines on an international level. This time, we wanted to go beyond the costs and focus on other elements such as immunization rates, the problems that exist with immunization, how pooled purchasing systems operate between different countries and supply issues, to name but a few.

We read dozens of academic articles and thoroughly analysed documents from international organisations, as well as recent reports from the Strategic Advisory Group of Experts (SAGE), a World Health Organization (WHO) group that analyses themes related to immunization.

We were not the first to investigate such a topic and we were inspired by the research of our peers. The analysis and visualization of the effects of the introduction of the measles vaccine in various US states, created by the Wall Street Journal is of particular note. It made us consider the possibility of creating a similar graphic on a wider scale, to cover all the countries in the world (or most of them, anyway). So, we created the graphic that you can see in the report on immunization levels. We were also inspired by the study by The Economist about the surprising differences in immunization rates between developing countries and developed countries. This is a theme that caught our attention when we first studied the data and is reflected in the article about anti-vaccine sentiment.

We also studied thoroughly existing analysis on price differences across countries and the importance of transparency in state-pharmaceutical negotiations, as well as all information on ongoing trends accompanying the WHO’s V3P project. In parallel, we analyzed the positions from industry actors and the different organizations involved, e.g. GAVI, Médecins Sans Frontières and the WHO itself.

The data

For this research, we needed to cross check various databases: immunization rates, cases by disease, deaths by disease, immunization schedules, levels of stockouts, public budgets allocated to immunization campaigns, purchase prices of vaccines for countries and organisations, as well as the basic elements required to provide context such as GDP or population by country or region. The majority of data collected, although updated in December 2016, refers to information covering the period up to 2015.

We found some of the data in easily reusable databases, although it required some cleansing. Other information was obtained from websites where it required scraping to obtain a version of the data that we could work with. In more complicated instances, we had to draw upon a range of several data sources.

Which vaccines did we analyse?

We began by considering the WHO list of essential medicines. We studied the advice and immunization schedules for each one, in an attempt to cover the most common diseases worldwide, which would allow us to make comparisons between different countries.

We created a document for each vaccine to link them to the disease and better understand how they work (doses, advice, etc.) We had to exclude from some analysis the relationship between vaccines and diseases which have more than one origin, as is the case with meningitis (which has various vaccines depending on the type of the disease), for example. Although there is a long list, according to the analysis of the results we included particular vaccines in the research. This does not mean that the others are less important, we simply found less relevant information about them or only information that was not comparable.

This data is not perfect, however it was sufficiently meaningful for us to be able to extract conclusions. In fact, one of the concerns of the group of vaccine experts at the WHO is the poor data quality. “How can you conduct an investigation without all objective information available on the subject?” Immunization data is not collected from certain countries. Moreover, some of the figures gathered by the WHO and other statistical services do not concur. An example of this problem is the statistics of cases of each disease from the WHO and those from the European Centre for Disease Prevention and Control (ECDC), which do not entirely concur.

Immunization data

This is used in a great deal of our research and shows the percentage of people that have received a vaccine out of the number of people that should receive it. Data runs from as far back as 1980 for some diseases, to 2015. The source is the WHO. We also reviewed the WHO interactive immunization atlas, but this does not relate immunization rates with GDP and other variables - just the geographical location.

Although this is fairly complete data and very useful for our research, we must take into account the fact that these are estimates.

What is the methodology of the WHO?. Over 95% of countries provide annual estimates of their immunization rates to the World Health Organization (WHO), which subsequently revises the data. The WHO anticipate that these estimates have a margin of error of ± 3% which may reach up to ± 20%.

Countries themselves have two methods to establish immunization rates of their population.

• By administration - vaccination rates are obtained by dividing the number of vaccines administered (the numerator) by the target population (the denominator). This is the most common. This method can produce errors if there are flaws in the data, for instance with the total population figure or in cases of high domestic mobility. Source: WHO , PAHO, Ministry of Health.

For example, to calculate the immunization coverage in children aged between 12 to 23 months, the number of children aged between 12 and 23 months that have received all the doses that they were due, according to the immunization schedule of that country, is divided by the total number of children aged between 12 and 23 months in that country.

• By surveys - which should be carried out every 3 to 5 years. These are useful to study the reliability of the administration method, or to substitute it. Data is obtained from immunization records and the vaccination history of medical centres or interviews with families. The calculation of the numerators and denominators is similar to that of the administration method, but in smaller groups. Source: WHO.

In both cases, we had to consider how to handle the issue of multiple dose vaccines. Sometimes it was interesting to compare the results between the first and the last dose, but for certain research we decided which dose was the most representative. In principle, for the majority of the analysis we used the final dose, on the basis that having reached this stage implies having received the full course of immunization required to be effective. In our analysis to test our stance, we found very few anomalies in which the immunization percentage of the last dose is greater than that of previous doses.

There are a few specific exceptions. For instance, for the second dose of the measles vaccine, France only submitted data for the period up to 2014. For our analysis, in the same way as the WHO does in their country factsheets, we prolong the 2014 figure into 2015, until the final data, which will be published over the course of 2017, becomes available.

For herd immunization we have used a rate of 95% for measles (the figure which is agreed in most research). For the DPT vaccine line we used 80%, the minimum coverage level recommended by the WHO SAGE group. According to calculations of the necessary rate to achieve herd immunization, polio would require between 80% and 86%. We used 80%, the lowest rate within this range. Two sources (here and here) discuss the subject, although there are many more.

Cases and deaths by disease

In order to determine the number of cases of each disease recorded every year, we used the WHO table of incidences, to which we have added certain data:

• In order to be able to include cases of polio in 2016 in the successes article video, we used the information available in Polio Eradication, an extremely up to date report on the global state of polio. The dots are ubicated by country, but inside a country they are randomly distributed.
• For the video about Guatemala, we added the information from recent months taken from the survey carried out by the Guatemalan Human Rights Council, Action Against Hunger, ALIANMISAR (National Alliance for Indigenous Women’s Organisations for Reproductive Health, Nutrition and Education) and the Alliance for Nutrition.
• We reviewed a complementary source for measles data from the WHO’s monthly reports. However, since some countries send the data late and with different timing, we only used the monthly data to be aware of the most recent trends, and didn’t rely on it for the final content.
• Data on the incidence of tuberculosis in the report on shortages are WHO estimates from its World Health Observatory (here).

As is the case with vaccination coverage, the data on numbers of cases is not perfect. Taking into account the fact that the data only includes cases which are treated under the healthcare systems and are therefore officially recorded, the conclusion is clear: more cases occur than those which appear in the data. As the WHO explain: “Many countries face challenges in accurately identifying, diagnosing and reporting infectious diseases due to the remoteness of communities, lack of transport and communication infrastructure, and a shortage of skilled health-care workers and laboratory facilities to ensure accurate diagnosis.”

An indication that data is not 100% exact is the lack of agreement with other databases. For example, to gather the information from European countries there is also the ECDC (their methodology, here). We carried out various tests to ascertain that the data did not concur (although by only a small margin of error) and this demonstrates the need for a clear and consolidated system of data collection. The ECDC data is more recent and was useful to contextualise the data, however we did not include this in our database because it only covers certain diseases and its scope is limited to Europe.

We excluded countries with very little data from the analysis and presentations.

We have also used mortality figures, from two sources:

• Child mortality by country and by disease. With data from 2000 to 2015.
• General mortality data by disease, here.

Stockouts data

Data on vaccine stockouts come from the WHO monitoring system of vaccine-preventable diseases (available here). The information extracted from this repository was last updated by the WHO in December of 2016, and only has data until 2015.

Countries notify WHO if they have experienced stockouts, either at national level or in some of the health districts where each country is divided. The global figure of 97 states that suffered a shortage in 2015 of some of the essential vaccines includes both those produced at the regional and national level. However, to make the map of stockouts of the vaccine against tuberculosis simpler, only the problems at national level have been taken into account, because of their greater relevance.

Price data

Trying to shed light on the market turned out to be much more difficult in this second phase, where we focused on prices paid by countries for vaccines, than in the first, when we investigated the price of certain essential drugs. There is very little information available, and what is there is partial. And comparisons, because of the multiple variables involved, are complex and force us to take the results with caution. The WHO price database includes about 50 countries, but their names are anonymised.

We started with information from seven countries and three organizations about ten vaccines, those for which we had the most information available -although our database collects many more data points. Our goal, if we’re able to find enough resources, is to keep growing the database, including additional countries and vaccines. Do you want to collaborate?. In all cases we have compared prices without taxes.

The sources of the prices paid by the organizations are the following:

• PAHO Revolving Fund, with data from 2016. A PDF that we converted into structured data. The main issue with it is that the information is not broken down by manufacturer and only a weighted average is published.

• Unicef published its price data in 2011 for the first time, and does break down prices per manufacturer. The information published includes data from 2001 to 2016, so we used the latest year.

• MSF sent us a database containing the prices paid when purchasing their vaccines.

For countries, the search for information is much more complicated. We reviewed public procurement sites, government transparency portals and official purchasing documents from more than 30 countries, trying to cover all types of regions and income levels. But we didn’t manage to do it. In most cases, the information is not public or can’t be found without deeper local knowledge. In others, it’s just hidden. In the end, unfortunately, we could only find a country outside the high income range, but we continue to work on improving the representativeness of the data.

• Spain. We use the national level procurement site, Contratación del Estado, and the Official Gazette, BOE. The price information comes from the framework agreement signed for 2015 and 2016, which covers 15 of the 17 regional governments (Andalusia and the Basque Country are left out), the cities of Ceuta and Melilla and purchases made by the central government through the Ministries of Defense and Finance. As of the time of this investigation being published, the Government is evaluating the award of the next framework agreement, which will regulate prices between 2017 and 2020. We’ve also reviewed purchases made during 2015 in Catalonia, Basque Country and Andalusia, since they publish -in their procurement information sites- data about prices per dose, although often hidden in the bid formalization paperwork. Although some of these regions are outside the national joint purchasing system, they’ve purchased at the same prices in most cases. Hence, if the price of a vaccine was not available from the framework agreement (because it wasn’t awarded to any bidder in the first round), such as for the pneumococcus vaccine, we’ve used the price of adjudication in the regions.

• United States. The CDC (Center for Disease Control & Prevention) publishes current prices in a easy to understand list. We collected the data in February 2, 2017 and subtracted taxes.

• Portugal. Prices extracted from the official public contracts database BASE, by IMPIC (Institute of the Public Markets of Real Estate and Construction). Prices refer to purchases made in 2016.

• Italy. The Directorate-General for Prevention publishes a list of contracts, which includes vaccine prices for 2015. At the moment there is no more recent information.

• Hungary. We use public procurement data from the Department of Health.

• Ukraine. We used Prozorro, a public procurement information platform.

• Poland. We use the webpage from the Department of Public Contracts of the Ministry of Health. We were able to extract information from only some of the adjudications shown, since in many of them there was no price per dose or it was impossible to calculate.

In all cases, currency conversions to euros for the years 2015 and 2016 were made using the US Internal Revenue Service average exchange rates. For 2017, we used exchange rates as of December 31, 2016.

We needed to cluster all the purchases into basic groups of vaccines, which we then differentiated more finely according to their target users (i.e. children or adults), whether they were conjugated or not… We decided to analyze only those vaccines for which more data points were available or were more relevant from a journalistic point of view (e.g. because they are more expensive). Each of the vaccines has its own casuistry and, as the WHO itself admits, it’s challenging to perform a 100% rigorous comparison.

We used some lists like the one from the WHO, as well as manufacturers’ web pages, some vademecum and technical specifications, to understand which subtypes existed globally for each vaccine.

We have included in our articles all the prices, as a range, paid by a country or organization in the analyzed time, in order to represent all the the different options (e.g. brand, packaging). We’ve tried to compare in the articles only those data points that, because of their composition and other characteristics, were actually comparable, but in some cases the information was not enough. If you find any errors, please let us know.

Cervical cancer data

The latest global data on cervical cancer incidence, cited in most studies on the subject, is that from Globocan 2012. Although we reviewed it for our investigation, we decided to use more recent data for the main data visualization, in this case about mortality. The WHO gives estimates of causes of death in the world in 2015 and from there we extracted, adding population and GDP, the mortality information shown in the visualization. Additionally, we used WHO statistics (updated in January 2017) to show whether or not countries had introduced the vaccine on 2015.

We discussed which variable should be used to represent mortality. According to all studies analyzed, cervical cancer affects only women and only from certain ages, so we decided to show mortality per 100,000 women over the age of 30. This accounts for almost 274,000 out of the 278,000 total estimated deaths in 2015.

For the country purchasing price data we considered the three vaccine types separately (Cervarix, Gardasil and 9-valent Gardasil). To get its price in private clinics, the journalist Asier Andrés gathered the data in five medical centers located in Guatemala City with different price ranges.

Other data sources

• Immunization schedules for all countries via WHO.
• The years of vaccine introductions. In some cases, the dates of introduction were very recent and did not match the information from the countries. Moreover, 28 countries did not have the information. We requested this information from eight countries and only one responded - Guatemala.
• GDP data is taken from the World Bank, as is population data. This enabled us to calculate the number of cases per capita and, in some cases, omit certain countries from the charts in order to simplify the visualizations: concretely, we omitted data from countries with a population of less than 300,000 from the graph that compares different immunization rates.

Reports and website

Analysis of all this information allowed us to draw some very interesting conclusions, but we needed to examine it in even more detail. We were clear from the start that we needed first-hand reports. We focused the research on two regions and prepared the interviews. We chose two destinations:

• Central America. The aim was to analyse what had happened in Guatemala, which had suffered a significant drop in immunization coverage for basic vaccines, and analyse the situation in Guatemala and El Salvador with regards to the Human Papilloma vaccine, as this was not included in the vaccination schedules but cervical cancer rates in these countries are sky-high. We also wanted to talk to the people in charge of pooled purchasing systems such as the PAHO revolving fund and the COMISCA group to understand how these work.
• South Africa. Although the initial idea was to go to India, various problems with the embassy delayed and finally prevented the trip. We needed a plan B and we chose South Africa because we were very interested to see how the vaccination system worked in a country that did not receive external aid and had suffered important problems of stockouts.

We combined the data analysis, the reporting and the stories that we were told on this trip into this website, in English and Spanish, which is developed with Jekyll, and uses Javascript and D3.js for the data visualizations. You can find the code of the web in our Github repository. Journalism Grants offered their support throughout the project, but this in no way influenced any editorial decisions.

Most of the data visualizations are customized, displaying information depending on the country from which the reader is connected. This is the case for the three visualizations on measles, DTP and polio immunization levels in the article about vaccination, and in the two charts (on measles vaccination and vaccine mistrust) in the article about the anti-vaccine movement.

Home pictures credits (from Flickr): Untitled from CDC Global, Packaging vials, Filling machine and Dengue immunization public program in Paraná state of Brazil from Sanofi Pasteur.

Collaboration and reproduction

Medicamentalia Vaccines is not a closed project. If you have relevant information or you work for an organisation that wishes to reproduce all or part of the content, write to us and we will help you to make it happen.

You can collaborate, for example, by helping us find the information about the prices that your country pays for vaccines. We will include them in our database but, if you are a journalist, we’ll give you time to write your own articles before publishing the new infomation.

The database and all materials produced by the project (i.e. not including photographies provided by third parties) are Creative Commons (CC BY). You may use anything that you need or that interests you on one condition: you must attribute and link to Medicamentalia. We also ask that you tell us about it! We would love to hear from you.