How not to run a vaccine clinical trial – The Sputnik case study

• Russia’s Sputnik V jab became the world’s first registered Covid vaccine in August 2020 — prior to the publication of any trial results.
• Even when the results were published, they were hard to assess because the researchers didn’t release the protocol or lab data for the vaccine.
• Some scientists have raised questions around the trial’s findings of almost 92% efficacy due to substandard reporting.

South Africa’s medical regulatory body has received an application for the Russian vaccine, called Sputnik V. But it hasn’t yet made a decision on whether to approve the jab.

The answer to why not, lies in the questions outlined below.

There are three fundamental questions you need to ask when assessing a clinical trial:

1. How good is the vaccine?
2. How good is the research?
3. How good is the reporting (either in a medical journal or from a regulatory body)? 

The problem: You can’t answer the first question without answers for the other two.

In the first of this four-part series, we’ll be taking a closer look at the existing data for Sputnik V — as well as the gaps in it.

Meet the Covid jab from Russia – Sputnik V

Sputnik V, or Gam-Covid-Vac, was developed by the Gamaleya Research Institute of Epidemiology and Microbiology. The Gamaleya Institute falls under the Russian government’s department of health.

The Sputnik jab became the world’s first registered Covid vaccine on 11 August 2020 — prior to the publication of any trial results.

Here’s how the vaccine works:

The vaccine uses two different types of adenoviruses, which is a type of virus that causes common colds. This adenovirus acts as a vector or carrier (like a Trojan horse), which sneaks an unharmful form of a different virus into your body (in this case SARS-CoV-2, the virus that causes Covid-19). This type of vaccine is known as a viral vector.

Viral vector jabs work in a different way than mRNA vaccines, such as the Covid vaccines of  Pfizer (our vaccine roll-out uses this jab) and Moderna. Those shots use man-made pieces of genetic material to deliver instructions to your cells. Instead, viral vector shots such as Sputnik use another virus to sneak DNA into the body.

In the case of Sputnik, people receive two shots of vaccine, 21 days apart. The first jab uses adenovirus 26 (this is the same adenovirus used in the Johnson & Johnson jab) and the second uses adenovirus 5 as the vector.

Using these types of common cold viruses in vaccines has been tested in HIV vaccines. Where Sputnik differs from other two-dose Covid jabs is that instead of using the same virus in both shots like in AstraZeneca’s jab, the two doses use different forms of adenoviruses. 

There are, however, some concerns around the use of the adenovirus 5 vector, which is used in the second dose of the Sputnik vaccine. At least one study has shown that it can increase a person’s risk of being infected with HIV. 

A vector cannot replicate inside your body, so it won’t make you sick. It enters your cells and delivers the instructions coded into the DNA contained inside.

These genes carry the code for the spike protein that sits on the surface of the SARS-CoV-2 virus. Your cells then translate that code and begin building spike proteins.

The appearance of the spike proteins, while unable to harm you, sends out an alert signal to the immune system. Your body’s defences then respond by sending in fighter cells to destroy the foreign invader (in this case the spike proteins).

These cells remember and mark the spike protein as a harmful object. This way, if it appears again, like if you get infected with the actual coronavirus, your immune system is primed and ready to kill the virus on sight.

The lifecycle of a vaccine trial

Before a vaccine is authorised for use, it goes through several stages of testing to determine if it is safe and effective (in this case at preventing disease). Along each step of the way, more information is gathered about the jab which can help fine-tune its use — such as the dosage required or correct time interval between shots.

Prior to medicines (such as vaccines) being studied in people, scientists must first conduct tests in laboratories and on animals to determine whether it’s safe to use in humans. These studies are called “preclinical” research. 

They’re usually small studies, but they’re crucial. Preclinical results are used to determine, for instance, what levels of the dose are toxic and which are safe. 

Preclinical results also help researchers refine the set of rules they’ll use for the trial going forward, called the study protocol. Moreover, this information lays out exactly how the research will be conducted and what scientists are setting out to find. 

Once this early data has been collected, studies can move onto clinical trials in people. These trials happen in stages, which progressively get bigger in size. Phase one (the starting point) tests the safety and dosage and ordinarily includes between 20 and 100 people. The next step is phase two, which includes several hundred people and begins to track efficacy and side-effects.

The final stage is phase three which is the largest scale study and includes thousands of participants. Results from this phase, if successful, will then be used by manufacturers to apply for regulatory approval and begin distributing the intervention (for instance, a vaccine). Only around a quarter of the products make it through from phase one to phase three.

Under Covid, it became common to combine phases of trials in order to speed up the research process and get results faster, without compromising on the intervention’s safety. Clinical trials normally take several years to yield results, but with Covid this was condensed into a few months under pandemic conditions.

The drive to reach the finish line faster means Covid vaccine trials were done on a much larger scale than usual, involving tens of thousands of people. This larger sample size helps shorten the time frame for trials. The more people in a trial, the easier it is to get enough people infected to be able to get a statistically significant result on the efficacy of a vaccine.

A giant game of leapfrog? What the Sputnik trials found

The Gamaleya Institute hasn’t released the preclinical data for their vaccine – nor the protocol they followed for the trial. They skipped right to the results. 

Hilda Bastian is a founding member of the Cochrane Collaboration, which specialises in reviewing and assessing research on health interventions. She also runs the Absolutely Maybe blog about medicines for Plos, an academic journal. 

Bastian says: “[The researchers] kept reporting data during the trial and then [the trial was stopped] and they said ‘we’ve got enough numbers’ but we didn’t know what the rules were for deciding when there was enough data.”

The Gamaleya Institute published a paper in The Lancet on a joint phase one and two trial of the vaccine in September 2020.

The research was conducted at two Russian hospitals and enrolled just 38 people in each phase of the trial, consisting of men and women between the ages of 18 and 60. 

“Almost everything that could be wrong with an early trial was wrong with that particular one,” argues Bastian.

The trial was unblinded, which would usually mean that the doctors, trial runners and participants knew who had received the actual vaccine and who got a placebo or dummy drug. Trials are normally blinded so that either group is not treated differently by doctors because this can skew the results of the study in a manner that is difficult to fix. 

But in the case of Sputnik V, there was no placebo group — so there was no baseline information with which to compare the study’s results in the end. Instead, the researchers compared the two different formulations of the Sputnik V jab — one of which was a normal frozen version (as used in other jabs) of the two adenovirus shots and the other was a freeze-dried formulation (which needs to be mixed at the site before it can be used).

The Sputnik study was also non randomised. Randomisation is a key element of clinical research that helps make sure the study participants are not selected in a biased way and that they represent the general population. Randomisation also helps to eliminate external factors, which scientists call confounding variables, which could potentially influence study results.

The tiny trial’s participants ended up being mostly young, white men of a similar body weight. The few women (all young and white barring one Asian woman) were not divided equally into the two phases of the study. Diversity in trial participants is needed to assess how well the vaccine works in different populations. 

The predominance of young people in this study should be a cause for concern, Bastian says. “The first people that were going to get [this vaccine] were going to be old people, and so they went into this phase three trial without ever having even really tested if it was safe in people of all ages.”

Two months after its approval and widespread use in Russia, data from the phase three clinical trials for the Sputnik V vaccine were published in The Lancet in February. 

The study reported 91.6% efficacy in a group of over 40 000 adults (around half of whom received the Sputnik V jab while the other half received a dummy shot).

This put the vaccine’s efficacy closer to those of the top mRNA candidates such as Pfizer and Moderna than its other adenovirus counterparts, AstraZeneca and Johnson & Johnson.

But were the results too good to be true? According to some scientists, yes.

Bastian cautions: “The phase three trial just had all these red flags.”

Enter the parade of red flags

Going into the phase three trials, there were already questions about the quality of the research being done by the Gamaleya team, says Bastian. This is because up until that stage, the only data shared had been from the phase one and two trials, which had been poorly designed (because it was an unblinded study, there was no control group and the participants weren’t selected in a randomised way). 

Bastian says early study results put her mind on high alert: “That trial put the question on the line of ‘Can they do a proper trial?’ and so the first thing you look at coming into the phase three trial is do they have all the basics?”

The answer, for her, was no.

Here’s why:

Red flag number 1: The study design wasn’t public

It became very hard to assess the trial properly because at this late stage they still had not released the protocol, explains Bastian.

Protocols provide a guide or outline of the rules of the study and what exactly is being measured. Not sharing this part of the research would be like saying you hit a hole in one when no one knows what golf course you’re even playing.

Red flag number 2: There wasn’t proper reporting of adverse events, or side-effects

Trials should have systems in place for monitoring people after they receive the jab. Gathering this information helps assess the vaccine’s safety and what symptoms people should look out for after they have been immunised.

This monitoring can be done in two ways: either actively or passively. In active monitoring, the researchers would check in with all participants to record possible symptoms they may be experiencing. Passive monitoring is a reporting system where people are encouraged to log the data in themselves if they think they are having side-effects from the jab.

The Sputnik V trial used an existing state-run electronic database in Moscow called the United Medical Information and Analytical System to track possible adverse events. This system is used to log doctor’s appointments and also provides information about hospital admissions.

The issue with this approach is that there wasn’t a separate data collection system for the trial and the paper didn’t explain the reliability or acknowledge any potential limitations of the database being used, says Bastian.

Red flag number 3: Lack of diversity in trial participants

The study was done entirely in Russia — and in one city only. In itself, this isn’t a problem, but if it leads to trial participants all being of the same race, for instance, generalising the study’s results becomes tricky. 

In the case of Sputnik V, a large proportion of the trial participants were white. The lack of racial diversity does matter when it comes to clinical trial results for Covid because we know that the virus impacts people differently and race is a factor in that experience.

Covid vaccine trials like Johnson & Johnson’s took place in multiple countries, while Moderna was conducted in one country (the United States).

But the Moderna trial was done with a diverse group of participants across a variety of age groups and representing multiple ethnicities. Unfortunately, Sputnik did not have the same advantage.

A back and forth between researchers

In addition to the shortcomings of the trial itself, the research community also decried the “substandard reporting” in the paper that was subsequently published by The Lancet.

The Gamaleya Research Institute’s critics wrote to The Lancet, noting unexplained changes in the data which they deemed “peculiar” including missing enrollment dates for 100 participants. They also cautioned that the number of participants in different age groups didn’t add up to the grand total of participants stated in the paper. 

What exactly the researchers were investigating is also unclear, since the main objective of the study seemed to change. At first, the efficacy of Sputnik V would be assessed after one dose, but in the end, the assessment was done after two doses. 

The finding that Sputnik V is 91% effective hinges on this change, but the reasons for it have not been made public, critics argued.

These issues and others would be easier to understand and assess critically had the study protocol and individual patient records been made available, but the Gamaleya Institute didn’t respond to the researchers’ request for access. “We invite the investigators once more to make publicly available the data on which their analyses rely,” the scientists said in their letter. 

The Institute responded saying the numerical mistakes were “simple typing errors that were formally corrected” and that the protocol and dataset were submitted to The Lancet in line with international publishing standards. 

But Barry Schoub, who heads up South Africa’s ministerial advisory committee on Covid-19 vaccines, argues there’s only so much journal editors can do with the data provided. 

He explains: “I would more or less trust the high citation journals such as The Lancet, it’s one of the oldest medical journals. I think that their scientific integrity is pretty good.”

This story was produced by the Bhekisisa Centre for Health Journalism. Sign up for the newsletter.