The generation of evidence on the basis of opinions: The Delphi panel

I recently gave a presentation to doctors of the Hospital de Los Valles, Universidad San Francisco de Quito USFQ, Ecuador about Delphi panels and I thought I’d translate it into a blog post to keep a trace, but also because I am sure some have asked themselves the question. Indeed, many have encountered the term “Delphi” without really digging into what it means. This post aims to be brief and straight to the point and answer the following questions:

– What is a Delphi panel?
– What are Delphi panels applications in health?
– What is the position of Delphi panels within health evidence?


The Delphi technique is, theoretically speaking, a research method aiming to rigorously organise convergence of opinion from participants concerning real-world issues (1). More practically, it refers to an iterative (repetitive) process whereby a questionnaire is submitted in several rounds to selected experts. Each subsequent round is supported with a non-nominative qualitative summary of the previous round (2).

A Delphi methodology is used to address questions in the absence (or lack) of scientific evidence. This means two things: first, an extensive (systematic?) review of the literature about the topic must be completed first to ensure that no better evidence exists on the topic; second, it is not appropriate to conduct a Delphi panel because existing evidence goes against the point you wish to make. Furthermore, a Delphi panel usually aims to tackle issues at the centre of controversies, by which I mean that no single answer to this question is widely scientifically accepted as true.

– Example of a question inappropriate for a Delphi panel: how far is the Moon from the centre of Earth?

– Example of a question appropriate for a Delphi panel: how long before mankind walks on Mars?

Additionally, it cannot be a polar question (yes/no) as the interest (an feasibility) of obtaining a consensus on such a question is limited. It therefore must be (usually is) a Wh- question.

Finally, it must be a neutral question, based on scientific grounds, with the possibility of an answer from experience. By that I mean that the Delphi question cannot itself rest on controversial hypotheses. For example, preparing a Delphi panel discussing quality of life after death would rest on the monumental assumption that life after death is a scientific fact.

The Delphi method rests on the scientific fact/assumption that, given the chance, circumstance and rationale, people tend to change opinion towards a consensus. By chance, I mean that people are given the opportunity to address the question again; by circumstance, I mean that people are provided a setting where they will feel free to express a change of mind (Note: we are culturally shamed for admitting we are wrong/we changed our mind; even more so when we are recognised experts in a certain field) – consequently, Delphi responses must be anonymous; finally, by rationale, I mean that our likelihood of changing our mind is strongly associated with the quality of the argumentation provided, that given a structured reason for an opinion helps understand a viewpoint, therefore contributes to our change of opinion.

A key aspect of the Delphi panel is its target population: to whom do we ask the questions? It is generally agreed that Delphi panelists should be experts in the field that the question belongs to. There can be several methods to identify such experts, but I think that one goal should be reproducibility of the panelist selection process.

Consequently, the Delphi methodology rests on core the principles of iterativity, anonymity, argumentativity and expertise.

What are Delphi panels used for in health? There can be many applications to Delphi panels, provided the earlier requirements mentioned above are respected. Nevertheless, in my experience in health research, it seems that Delphi panels are essentially used for the generation of clinical guidelines, to establish basis for another research project (e.g. in the context of clinical trials and the identification of clinical outcomes to be included as part of the assessments), or finally, to contribute to health policy development.

Earlier, I wrote that Delphi panels should be used to address questions in the absence (or lack) of scientific evidence. But what I really meant was that they should be used in the absence of evidence better than opinions. Indeed, Delphi panels are “only” structured ways of collecting clinical opinions. And opinions are not ranked very high on the pyramid of evidence. However, evidence low on the pyramid does not mean that Delphi panels stand as invalid evidence. In many situations, opinions are, at one point in time, the best evidence available. This is true until robust clinical trials (if applicable), systematic literature reviews and/or meta-analyses (again, if applicable), are produced. This being said, not all opinions are equal. The strength of the Delphi comes from its methodology and more specifically, the number of opinions collected, the recognised expertise of the invited panelists and the rigorous methodology of collection of opinions.

I view Delphi panel as belonging to the best methods to gather opinions.

Here are two recent works I contributed to displaying the use of Delphi methodology

consensus in acne scar risk factors

consensus in actinic keratosis

(1) Hsu C-C, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess Res Eval. 2007;12(10):1-8.
(2) Thangaratinam S, Redman CW. The Delphi technique. Royal College of Obstetricians and Gynaecologists. R Coll Obstet Gynaecol. 2005;7:120-125.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: