How to Run a Delphi Study

A Delphi study sounds simple: ask a panel of experts a set of questions, share what they said, ask again. Then your committee asks what counted as consensus, why you stopped after round two, and who decided your panelists were experts. Some committee members won’t have met the method at all, so your proposal has to explain it as well as justify it.

In our 10 years of helping students with their methodology, the Delphi is one of the designs where a sound idea most often meets a thin plan. In this post, I’ll explain what a Delphi study actually involves, when it’s the right design, and what “modified” means. Then I’ll cover the six decisions to make before round one, and walk you through two real dissertations that ran one.

---

What is a Delphi study?

A Delphi study is a structured way of reaching agreement among experts. You send a panel a questionnaire, summarize the group’s answers, send that summary back with the next questionnaire, and repeat. Each round, panelists can reconsider their judgment in light of what the group said, without knowing who said what.

It was developed at the RAND Corporation in the early 1950s, for the US Air Force. The best-known early experiment asked seven experts to estimate how many atomic bombs it would take to cut US munitions output by a set amount, viewed from the side of a Soviet planner. The memo describing it is dated November 1951 and stayed classified for a decade. When its authors, Norman Dalkey and Olaf Helmer, published it, they described the method’s object in one line:

Notice the two things that sentence holds together: the questionnaires and the feedback between them. Over five rounds, the gap between the panel’s highest and lowest estimates narrowed from 100 to 1 to about 3 to 1. That convergence is the whole idea.

The classic design has four defining features:

1. Anonymity. Panelists don’t know who said what, and often not who else is on the panel, so nobody’s seniority or confidence sways the result. You, the researcher, know who they are.
2. Iteration. The same questions (or the ones still unresolved) go out in more than one round.
3. Controlled feedback. Between rounds, you tell the panel how the group responded, usually as a summary statistic and a summary of the comments.
4. Statistical aggregation. The group’s answer is calculated from everyone’s ratings, not argued out in a room.

Some methodologists reserve the name for studies that keep all four. Drop one without saying so, and a committee may fairly ask whether you ran a Delphi or a survey.

“Is a Delphi study qualitative or quantitative?”

Often both. In a classic Delphi, round one asks open questions, which you analyze qualitatively to produce a list of statements. Later rounds ask the panel to rate those statements, which you analyze with descriptive statistics. Most of the graduate Delphi projects in one review followed that pattern, so plan for both kinds of analysis in your methods chapter.

---

When a Delphi study is the right choice

A Delphi earns its place when:

• The evidence is thin, and expert judgment is the best evidence available. Developing criteria, a curriculum, a set of quality indicators or a list of priorities in a new area are the most common uses in graduate research.
• Your experts are spread out, across organizations or countries, so an in-person meeting isn’t practical.
• People won’t speak freely in front of each other. Because panelists don’t see who said what, seniority and status carry less weight, which helps when your panel mixes junior and senior people. Anonymity doesn’t hide them from you, though, so if your own role is what might make people cautious, say how you’ll handle that too.

It’s the wrong choice when you want to know what a population thinks. That’s a survey, and one widely cited guide, drawing on an earlier paper, puts the difference neatly: “Common surveys try to identify ‘what is,’ whereas the Delphi technique attempts to address ‘what could/should be.’” It’s also a poor fit if you can’t keep the same experts engaged for several rounds over a few months, or if you need the correct answer to a factual question rather than the best agreed judgment.

Two methods often get confused with it:

• A focus group puts people in a room to talk, so you hear how they reason, but also how the loudest voice steers the rest.
• The nominal group technique is a structured face-to-face meeting. People note ideas privately, share and discuss them, then rank them individually, with the scores pooled.

A classic Delphi is the only one of the three that never needs the panel in the same place.

If you’re still choosing between designs, our guides to research design and justifying your methodology cover the reasoning a committee will expect to see. This video walks through how to choose a methodology that fits your research questions.

Classic, modified and e-Delphi

This is where, in our experience, students get tripped up, because “modified Delphi” doesn’t mean one thing. Different papers use it for different changes to the classic design. The table below sets out two common meanings beside the classic version:

Type	How round one works	Do panelists ever meet?
Classic	Open questions; you build the statements from the answers	No
Modified (literature-first)	A list of statements you’ve built from the literature, rated straight away	No
Modified (with a meeting)	Either of the above	Yes, usually between rounds

Starting from the literature is described as “both an acceptable and a common modification” when the basic information on the topic is already available. Adding a meeting is the other common meaning: in one review of 80 healthcare Delphi studies that selected quality indicators, 49 of the 78 that said which type they used had added one. Norms differ by field, too. One widely used guide from health research treats a rated first round as the standard design, and an open first round as a variant.

An e-Delphi is about the medium rather than the design: the questionnaires run online rather than by post. It can be classic or modified, and online delivery is now so usual that it’s arguably the norm.

So don’t just write “a modified Delphi was used.” Name what you modified and why. A meeting, for example, cuts across one of the method’s founding features, because it gives a confident or senior panelist the chance to dominate the discussion. So, if you decided to include a meeting, say how you guarded against that.

---

How a Delphi study works, round by round

Most Delphi studies run for two or three rounds. Here’s the loop, from the first questionnaire to the point where you stop:

Three rounds is the common shape for a classic design, with the third used to re-rate the items that hadn’t reached agreement. Three rounds are often sufficient, and a third round usually brings “only a slight increase in the degree of consensus.”

Some guides allow up to five, depending on how much agreement you need, and a mixed panel can take more rounds to agree. So, if you plan more than three rounds, make sure you clearly justify why the extra rounds are worth the time and the panelists they could cost you.

Each round also costs you panelists, which, in our experience, students underestimate. Expect each round to take a few weeks: a common recommendation is a minimum of 45 days for the whole process, with two weeks for panelists to respond to each round.

---

The six decisions to make before round one

These are the decisions that turn a good idea into a method you can defend. Make each one before round one, write it into your proposal, and report it in your methods chapter whether or not things went to plan.

1. Who counts as an expert

There’s no fixed definition, which is exactly why yours needs to be written down. A widely used set of requirements covers four things: knowledge and experience of the issue, the capacity and willingness to take part, enough time to see all the rounds through, and the ability to communicate clearly.

Turn those into criteria you can check, such as years in a role, a credential or publications on the topic, and say how you found the people who meet them. That makes this a form of purposive sampling, and your chapter should describe it that way.

Hahn’s study, one of the two dissertations below, shows what this looks like in practice. Notice that every criterion can be checked:

Notice the two tiers: professional standing first, then experience of the topic, through practice or publications. To find people who met them, Hahn started with ten experts and asked each to nominate others (snowball sampling), building a pool of at least 30. The thesis also reports how many panelists met each criterion (12 of the 13 were licensed, for example), which is exactly the kind of detail an examiner looks for.

2. How many experts, and how many to invite

There’s no standard number. For a homogeneous panel, 10 to 15 experts is often cited as potentially sufficient, with more needed if your panel mixes different groups. Across roughly 40 dissertations and two theses in one review, panels ranged from 8 to 345 participants.

If you see “4 to 171” quoted as the range for graduate studies, check the source: in that paper, it describes a table of 16 published studies, not the dissertations.

Whatever you settle on, invite more people than you need, because some will drop out between rounds. One of the dissertations below shows how to work that out.

3. The rating scale

Nine-point scales are often used, and three-, five- and seven-point scales all appear too. Pick one, define what each point means, and use the same scale in every rating round, so the rounds can be compared.

4. The consensus rule

If I had to pick one decision to get right, it’s this one. In our experience, it’s the one examiners ask about first, and published studies are often loose with it. One review of 100 Delphi studies found that only 72 defined consensus at all. Percent agreement was the most common definition, with 75% as the median threshold. And although 86 of the studies concluded they’d reached consensus, only 42 of those had set a threshold in advance.

The common options are:

• Percent agreement: for example, 75% or more of the panel rating an item 4 or 5 on a 5-point scale.
• Central tendency plus spread: for example, a median of 4 or more and an interquartile range (IQR) at or below a threshold you set. The IQR is the spread of the middle 50% of ratings, so a small IQR means the panel’s answers are close together.
• A combination of the two, which is what many healthcare Delphi studies use.

None of these thresholds is “correct.” What matters is that the rule is decided at the start of the study, not after you’ve seen the ratings. Choose one, cite a precedent for it, and set it before you see any data.

5. When you’ll stop

You can stop when your consensus rule is met, when answers stop changing between rounds (stability), or after a fixed number of rounds. In the same review of 100 studies, only 23 stopped because consensus was reached, while 70 stopped after a set number of rounds. A fixed number is easier to plan around, but say so honestly, and report what still hadn’t reached agreement when you stopped.

6. What feedback panelists see

Between rounds, you’ll usually show each panelist the group’s median or percentage for each item, sometimes their own previous rating, and a summary of the comments.

Decide this in advance too, because the feedback is where you can most easily steer the result without meaning to. The method’s own inventors admitted it of their experiment: “Some ‘leading’ by the experimenters inevitably resulted from the selection of the information supplied by the experts.” Summarize the comments fairly, including the ones that cut against the way the ratings are heading.

Set all six out in a short table in your proposal, with a sentence of justification and a citation for each. A committee that has never seen a Delphi study will follow it far more easily when the rules come first. Here’s what that might look like for a hypothetical study of the skills new school counselors need:

In your own table, cite each source in your university’s referencing style. The full references for the ones above are in the sources list at the end of this post.

Two real Delphi dissertations

To make all of this a little more tangible, let’s look at two real-world examples. Here’s how two completed doctoral studies made these decisions. The first ran a classic Delphi and the second a modified one, and both show their working in their methods chapters.

Example 1: a classic three-round Delphi

The first is a PhD in counselor education from the University of Texas at San Antonio, in which Crystal Hahn built a consensus on what entry-level counselors should be taught about military service-related PTSD.

The panel was planned around dropout. Hahn wanted at least 11 panelists to finish all three rounds, and looked at the dropout rates other counseling Delphi studies had reported, which ran from 10% to 39%. Then the plan worked backwards from the worst case:

In the end, 13 experts took part and 11 (85%) completed the final round. The panel came in below the recruitment target but exactly at the minimum the plan was built to protect, which is the point of planning for dropout. I’d copy this reasoning straight into your proposal. Show the same arithmetic: the panel size you need, the dropout you expect, and the number you’ll therefore invite.

Round one was open. All 13 panelists answered eight open questions, producing 104 responses, which Hahn coded into 116 curriculum statements for the panel to rate.

The consensus rule had two parts. Rounds two and three used a 5-point scale from “inappropriate” to “appropriate,” and Hahn “used a median of four and an interquartile range (IQR) of 1.0 to determine consensus.”

The median condition is the one that’s easy to skip, and the data show why it matters. In round three, one statement, that career development was the right core course for this content, met the spread condition (IQR = 1) with a median of only 3, the scale’s neutral point. The panel agreed, and what it agreed on was that it couldn’t call the statement appropriate. Because the rule also required a median of 4, Hahn didn’t retain it.

That’s the lesson in one item: consensus is agreement, not endorsement. A spread rule on its own will happily certify that your experts all agree on a shrug.

The stop is where the write-up wobbles. Hahn planned three rounds, and after round three, 25 statements still hadn’t reached consensus (a count that includes statements the panel suggested in round two). The methods chapter gives “time constraints” as one reason for not running a fourth round, the procedures say a fourth round wasn’t needed, and the limitations concede that “a fourth round could have been beneficial.”

All three may be true, but an examiner will ask which one decided it. A stopping rule written into the proposal answers that question before it’s asked.

Example 2: a modified Delphi with a meeting

The second is a PhD in community health sciences from the University of Calgary, featured in our public health dissertation collection. Rosmin Esmail wanted to know which knowledge translation theories, models and frameworks would suit health technology reassessment, the process of reviewing technologies already in use.

Esmail’s modified Delphi had a five-member committee and three rounds. In round one, members independently rated each of 36 frameworks “yes,” “potentially yes” or “no.” The consensus rule was strict: a framework was kept only if all five members rated it “yes” or “potentially yes,” and dropped only if all five rated it “no.”

Round two re-rated the frameworks still undecided. The third round was the modification, a two-hour face-to-face meeting, held after the committee had agreed on ground rules and selection criteria, and it settled most of the decisions, including 10 of the 16 frameworks kept.

Esmail then sent those 16 to 22 international experts in a separate survey, with consensus set at 70% rating a framework “yes.” None of the 16 reached it. Only when “yes” and “partially yes” were combined did one framework clearly lead, rated suitable by 19 of the 22 experts (86%) and top-rated by both groups. How you group the points on your scale can decide your result, which is why that grouping belongs in your consensus rule from the start.

What makes this thesis worth reading is how candidly Esmail examines the design. The limitations section reads like an examiner’s list of questions:

The thesis also notes that the 70% threshold “was arbitrary and determined a priori to survey administration,” and answers two of these concerns: a wide range of expertise on the committee, and a facilitator and ground rules to counter conformity. That’s the pattern to copy: name the weakness before your examiner does, then show what you did about it.

What an examiner would ask

Both studies passed, and both made reasonable choices. Even so, there are questions an examiner could put to each, and they’re the ones you’ll want to answer in your own chapter.

Of the classic Delphi: how were the 116 statements built from 104 responses, and could someone else have coded them differently? An audit trail answers that, and Hahn used an outside auditor for exactly this reason.

Of the modified Delphi: with five members and a unanimity rule, did one strong voice decide the result at the meeting? And when the international survey produced no framework at 70% on “yes,” was combining “yes” and “partially yes” a decision made in advance or after seeing the data? It’s the same question every Delphi has to answer: were the rules fixed before the results came in?

---

Writing up your Delphi study

The CREDES guidance on conducting and reporting Delphi studies, developed in palliative care research, is a widely cited reporting checklist. Even outside healthcare, it’s a useful structure for your methods and results chapters. Its main reporting points come down to these:

• Your expert criteria, how you recruited the panel, and the response rate in every round.
• Your consensus definition, set in advance, including what happens to items that don’t reach it.
• A flowchart of the process, round by round, including the analysis between rounds.
• The results of each round separately, so a reader can see consensus develop.
• What any modification was, and why, applied consistently.

Reporting is where many published studies fall short. In that same review of 80 quality-indicator studies, only 31 reported response rates for every round. Your chapter can do better than the literature you’re citing.

Two problems come up in the proposals we see for this design. First, a committee that doesn’t know the method needs it explained, with citations to the methodological literature rather than only to studies that used it. The short 2007 guide by Hsu and Sandford is still a widely cited introduction, so don’t drop it because of its date.

Second, if your Delphi built an instrument or a framework that the rest of your study relies on, give it proper space in your methodology chapter, not a line and an appendix. The panel, the rounds and the decisions you made are part of what makes your instrument credible.

---

The limitations of the Delphi method

Every design has limits, and examiners expect you to know this one’s. These are the five to address, with what you can do about each:

• Consensus isn’t correctness. The CREDES guidance is direct about this: consensus doesn’t necessarily mean the panel found the “correct” answer. Even the method’s inventors wrote that “the predictions obtained in the form of an opinion consensus may be lacking in reliability.” Present your findings as an agreed expert judgment, and report the items your panel disagreed on. Stable disagreement is informative too. CREDES notes that it highlights differences in perspectives. It may also flag where the evidence itself is thin.
• Consensus can hide dissent. Depending on how consensus was defined, “up to 40% of the experts do not agree with the consensus.” Report the full distribution of ratings, not just whether each item passed.
• Dropout can bias the result. If the panelists who disagree are the ones who leave, the remaining panel agrees more easily. Report response rates by round, and say whether the people who dropped out differed from those who stayed.
• The thresholds are arbitrary. No rule is correct, so justify yours with a precedent and apply it consistently.
• It takes time. Several rounds, each with a response window and analysis in between, can take months. Build that into your timeline, along with any ethics approval your panel needs.

None of these makes the Delphi a weak design. They make it a design that has to be reported carefully, and a well-reported Delphi study is a genuine contribution to your field. When you come to write your own limitations section, this video walks through five common limitations in dissertations, with examples.

Still have questions?

How many rounds should a Delphi study have?

Most Delphi studies use two or three rounds. Three is common for a classic design, where the first round uses open questions and the next two rate the resulting statements. More rounds can raise agreement slightly, but each one costs you panelists and time. Decide your maximum number of rounds, or your stopping rule, before you start.

How many experts do you need for a Delphi study?

There’s no fixed number. Ten to 15 experts is often cited as enough when your panel is homogeneous, and you’ll need more if it mixes different professional groups. One review of about 40 dissertations found panels ranging from 8 to 345 participants. Invite more people than you need, because some will drop out between rounds.

What is a modified Delphi study?

A modified Delphi study is any Delphi that departs from the classic design, and the term has no single agreed meaning. Two common modifications are starting with a list of statements drawn from the literature instead of open questions, and adding a face-to-face meeting of the panel. If you run one, state exactly what you modified and why.

What is a good consensus threshold?

No threshold is correct. In one review of 100 Delphi studies, percent agreement was the most common rule, with a median threshold of 75%. Many health studies combine a minimum median rating with a level of agreement or a maximum interquartile range. Whatever you choose, set it before round one and cite a precedent for it.

What does a Delphi questionnaire look like?

A first-round questionnaire in a classic Delphi asks a few open questions about the topic. Later rounds list statements built from those answers, each rated on a scale, often of 5, 7 or 9 points, for importance, appropriateness or agreement. Each new round usually shows panelists the group’s previous ratings, so they can reconsider their own.

Delphi study vs focus group: what’s the difference?

A focus group brings participants together to discuss a topic, so you hear how they reason, but stronger voices can steer the group. A classic Delphi study never brings panelists together. They answer anonymously in rounds and see only a summary of the group’s responses, although some modified versions add a meeting. A Delphi aims at consensus; a focus group usually explores a range of views.

Is the Delphi method qualitative or quantitative?

A classic Delphi is both. The first round collects open-ended answers, which are analyzed qualitatively and turned into statements. Later rounds ask the panel to rate those statements, and the ratings are analyzed with descriptive statistics, such as medians and percentages, to judge consensus. A Delphi that starts from a list drawn from the literature is mainly quantitative.

Who developed the Delphi technique?

The Delphi technique was developed at the RAND Corporation in the early 1950s, in research for the US Air Force. Norman Dalkey and Olaf Helmer described its best-known early experiment, which used seven experts and five rounds of questionnaires. Because the work was classified, it was only published about a decade later.

Can’t find your answer here? Ask a Grad Coach directly – the initial chat is free.

Speak with a friendly coach →

Cite-worthy sources

Cite this article

Jansen, D. (2026). How to Run a Delphi Study. Grad Coach. https://gradcoach.com/delphi-study/

Copy citationMore citation formats

• Birko, S., Dove, E. S., & Özdemir, V. (2015). Evaluation of nine consensus indices in Delphi foresight research and their dependency on Delphi survey characteristics: A simulation study and debate on Delphi design and interpretation. PLoS ONE, 10(8), e0135162.
• Boulkedid, R., Abdoul, H., Loustau, M., Sibony, O., & Alberti, C. (2011). Using and reporting the Delphi method for selecting healthcare quality indicators: A systematic review. PLoS ONE, 6(6), e20476.
• Dalkey, N., & Helmer, O. (1962). An experimental application of the Delphi method to the use of experts (RM-727/1-Abridged). RAND Corporation.
• Diamond, I. R., Grant, R. C., Feldman, B. M., Pencharz, P. B., Ling, S. C., Moore, A. M., & Wales, P. W. (2014). Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of Clinical Epidemiology, 67(4), 401-409.
• Esmail, R. (2020). Understanding the relationship between health technology reassessment and knowledge translation [Doctoral dissertation, University of Calgary].
• Hahn, C. (2025). Developing military service-related PTSD training curricula for entry level counselors: A Delphi study [Doctoral dissertation, University of Texas at San Antonio].
• Hsu, C.-C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), 1-8.
• Jünger, S., Payne, S. A., Brine, J., Radbruch, L., & Brearley, S. G. (2017). Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliative Medicine, 31(8), 684-706.
• McMillan, S. S., King, M., & Tully, M. P. (2016). How to use the nominal group and Delphi techniques. International Journal of Clinical Pharmacy, 38(3), 655-662.
• Niederberger, M., & Spranger, J. (2020). Delphi technique in health sciences: A map. Frontiers in Public Health, 8, 457.
• Skulmoski, G. J., Hartman, F. T., & Krahn, J. (2007). The Delphi method for graduate research. Journal of Information Technology Education, 6, 1-21.