Validity & Reliability in Research

Almost every quantitative dissertation has a paragraph headed “validity and reliability”, and it’s often the weakest paragraph in the methodology chapter. Students write that their instrument “is valid and reliable”, cite a textbook definition, and move on. Examiners notice, because a claim isn’t evidence, and everything in your results chapter rests on the measurements being sound.

Our coaches have read a lot of these paragraphs, having coached 10,000+ students through their dissertations. In this post, I’ll explain what validity and reliability mean, the main types of each, and the numbers examiners look for. Then I’ll show you how three real dissertations established theirs, including one that admitted it couldn’t.

---

Validity vs reliability at a glance

Both terms describe the quality of a measurement, whether that’s a survey scale, a test or a physical instrument. They ask different questions, though:

That last row is the one to remember. As one widely cited guide puts it, an instrument “cannot be valid unless it is reliable”. Reliability is necessary for validity, but it isn’t enough. The classic way to picture it is a target, where each dot is one reading and the bullseye is the true value:

The “reliable, not valid” target is the trap most students fall into. A high Cronbach’s alpha tells you your questions hang together. It doesn’t tell you they measure the concept your research questions are about.

---

What is validity?

Validity is whether an instrument measures what it’s supposed to measure, so that the conclusions you draw from its scores are justified. It isn’t a single property you either have or don’t. You build a case for it from several kinds of evidence, and the strength of that case is what an examiner judges.

Say you want to measure job satisfaction with a set of Likert-scale items, and every item asks about pay:

These four items would probably agree with each other nicely, so the scale could look reliable. But job satisfaction also covers your manager, your colleagues, the work itself and your chances of promotion. A scale that only captures pay measures pay satisfaction, so calling it job satisfaction isn’t valid, no matter how consistent the answers are.

The fix starts before you write or choose a single item. Define your construct from the theory you’re using, then check that your items cover that definition and nothing else. If your theory says job satisfaction has five dimensions, your scale needs all five.

For example, the Job Descriptive Index, one of the longest-standing job satisfaction measures, splits the concept into five facets and measures each one separately. Here’s what items covering those five might ask about (our own illustrations, not the index’s actual items):

The pay-only scale above covers one row of this table. That’s why it can’t be a valid measure of job satisfaction, no matter how well its four items agree with each other.

The main types of validity

Validity evidence comes in a few standard types. The first four are about your instrument:

• Face validity: whether the items look, on the surface, like they measure the concept. It’s usually considered the weakest type, because it’s a judgment, not a test.
• Content validity: whether the items cover every part of the concept, usually judged by a panel of experts.
• Construct validity: whether scores behave the way the theory says they should. That includes convergent validity (correlating with measures of related concepts) and discriminant validity (not correlating with unrelated ones), and it’s often tested with factor analysis.
• Criterion validity: whether scores match an established benchmark, either at the same time (concurrent) or later on (predictive), such as an aptitude test predicting first-year grades.

The other two are about your study as a whole, not your instrument.

Internal validity is whether your design lets you claim that one thing caused another. It’s the central worry in an experiment or a quasi-experiment, because something other than your intervention may explain the result. Say students who chose a tutoring program went on to score higher than those who didn’t. If the volunteers were more motivated to begin with, motivation could explain the gap just as well as the tutoring. Random assignment, a comparison group and measuring both groups beforehand are the usual ways to rule explanations like that out.

External validity is whether your findings generalize beyond your sample, to other people, settings or times. It depends heavily on your sampling. A survey of first-year students at one university tells you about those students, and it may not hold for students elsewhere or in later years. A convenience sample doesn’t rule a study out, but it does limit how far you can generalize, and your discussion should say so.

When someone asks about “validity” in your methodology chapter, they usually mean the instrument types. Your design’s internal and external validity belong in the design and limitations discussion.

---

What is reliability?

Reliability is whether an instrument gives consistent results when it measures the same thing under the same conditions. If your bathroom scale shows a different weight each time you step on it, you can’t trust any single reading, and nothing you calculate from those readings can be trusted either.

Reliability covers the people using the instrument, too. If two researchers code the same interview, or rate the same classroom observation, and reach different answers, that’s a reliability problem, even if the coding scheme itself is sound.

The main types of reliability

Which type you need depends on how you collect your data:

• Internal consistency: whether the items in a scale agree with each other. It’s the one you’ll report for almost any multi-item survey scale, usually as Cronbach’s alpha.
• Test-retest reliability: whether the same people get similar scores when they take the instrument again a little later. It’s usually reported as a correlation or an intraclass correlation coefficient (ICC).
• Inter-rater reliability: whether different coders or raters agree. Use Cohen’s kappa for categories with two coders (Fleiss’ kappa with more) and the ICC for scores.
• Parallel forms reliability: whether two versions of the same test give the same results. You’ll mostly meet this in educational testing.

Most survey-based dissertations report internal consistency for each scale. Add test-retest if your concept should be stable over time, and inter-rater reliability whenever human judgment turns data into codes or scores.

How high should Cronbach’s alpha be?

Alpha normally runs from 0 to 1 (a negative value often means an item needs reverse-scoring). The same guide to Cronbach’s alpha notes that reported acceptable values range “from 0.70 to 0.95”, and 0.70 is the minimum our coaches see cited most often in dissertations. Many also cite bands such as 0.70 for acceptable, 0.80 for good and 0.90 for excellent, as Tunstall’s thesis below does.

Two things catch students out:

• Higher isn’t always better. The same guide warns that an alpha above 0.90 may mean some items are redundant, “testing the same question but in a different guise”, and notes that a maximum of 0.90 has been recommended.
• Calculate it per scale, not for the whole questionnaire. Alpha assumes the items measure one concept. Run it across several concepts at once and the extra items can inflate the result, so report it for each scale and subscale.

If your alpha comes out low, check for reverse-worded items first (“I rarely feel stressed” in a stress scale). An item that runs the opposite way to the rest drags alpha down until you reverse-score it. In our coaches’ experience, that’s the most common cause of a disappointing alpha, and the easiest to fix.

For test-retest and inter-rater ICCs, one practical guideline treats values below 0.5 as poor, 0.5 to 0.75 as moderate, 0.75 to 0.9 as good and above 0.9 as excellent. Its authors recommend judging by the 95% confidence interval rather than the single estimate, which is worth doing in your own write-up.

---

Can a measure be reliable but not valid?

Yes. Internal consistency only tells you whether your items move together, and items can move together around the wrong concept. The pay-only scale above is the textbook case.

It also happens the other way within reliability itself: a scale can pass one reliability test and fail another. A recent clinical psychology doctorate from the University of Lincoln, featured in our psychology dissertation collection, shows it clearly. Charlotte Mary Tunstall tested a 13-item measure of vicarious trauma, the cumulative toll that exposure to clients’ trauma takes on helping professionals, with 418 of them in the UK.

On internal consistency, the scale did well, with an alpha of .86. It also correlated with an established measure of psychological distress, the DASS-21 (r = .65), which is good convergent validity evidence. Then 318 of the participants took it again, on average 19.3 days later:

The test-retest ICC was .208 (95% CI .013 to .365), so even the top of the interval sits in the “poor” band. What makes this thesis worth reading is that it doesn’t hide the result. It reports both numbers side by side and discusses what might explain the gap, including the possibility that vicarious trauma itself shifts over a couple of weeks.

That possibility matters. Test-retest reliability only counts against a measure when the concept should stay stable. If the experience genuinely changes from one fortnight to the next, a low retest score may say more about the experience than about the scale. That’s exactly the kind of discussion an examiner rewards, so if one of your reliability results disappoints, report it and interpret it rather than leaving it out.

---

Use an existing scale, or build one?

As a general rule of thumb, try to use an existing, validated scale wherever one exists. It’s the advice our coaches give more often than almost any other in this part of a dissertation, because a published scale arrives with its validity and reliability evidence already in the literature. A scale you write yourself arrives with none, and building that evidence is a study in its own right.

Finding one is usually a matter of looking in the right places:

• The methods sections and appendices of studies on your topic, which name the scales they used
• The paper that originally developed and validated the scale, which you’ll cite anyway
• Databases your library may subscribe to, such as PsycTESTS or the Mental Measurements Yearbook
• The scale’s authors, who you may need to email for permission and the full item list

Once you’ve chosen a scale, use it as published. If you change the wording, drop items or translate it, say so, explain why, and treat the changed version as needing fresh evidence.

Example 1: an adapted scale, re-tested

A PhD from Purdue University, featured in our education dissertation collection, shows what that fresh evidence looks like. Yujie Huang surveyed 1,061 undergraduates about culturally responsive teaching, their sense of belonging and their academic self-efficacy.

No published scale fitted the study’s setting, undergraduate math classes, exactly, so Huang adapted existing ones. The culturally responsive teaching scale, for example, started from a published 18-item scale for middle school students, dropped its subscale on affirming students’ home languages, and added relationship-building items adapted from a second instrument, giving 23 items. Because the scales had changed, the thesis treats them as three new scales and tests them accordingly.

Huang did three things worth following in your own study. First, an expert panel of four faculty members, three PhD students and one master’s student reviewed the questionnaire for face and content validity, and several items were reworded on their advice. Second, a pilot with 39 responses tested the survey before the main data collection. Third, the reliability table reports the alpha of the published scale each one started from, beside the alpha in this study’s own sample:

That last column matters. A scale that was reliable for its original authors may not be reliable for your sample, and an adapted one even less so, so examiners want both numbers. Notice, too, that the first scale’s 0.95 sits above the 0.90 mark discussed earlier, which is the kind of value an examiner may ask about.

Huang also ran factor analyses on the culturally responsive teaching and belonging scales, to check that the adapted items grouped the way the theory said they should. That’s construct validity evidence, and it’s the step an adapted scale needs most.

Example 2: a self-built test, honestly

Sometimes no suitable scale exists. A Doctor of Nursing Practice project from UMass Amherst, featured in our nursing dissertation collection, is a good example of what that costs. Alexa de Villier wanted to measure what primary care providers knew about diet and heart disease before and after an education session sent by email, found nothing suitable, and wrote a 17-item questionnaire on their views and knowledge, based on the literature. The methods section says so plainly:

The first half of that sentence is the right instinct. Saying what you didn’t establish is far better than implying you did. The second half is the part to avoid in your own work. Reliability belongs to a specific instrument used with a specific group, not to a response format, so a figure from a different study can’t stand in for your own.

With six providers taking part, a meaningful reliability analysis probably wasn’t realistic, which makes the honest admission the right call. In a larger study, the minimum for a self-built scale is an expert review for content validity, a pilot, and an alpha from your own data.

Writing it up in your methodology

Your validity and reliability section needs to answer three questions for each instrument: where it came from, what evidence supports it, and what you found in your own data. Here’s what that might look like, written by us using the figures from Huang’s study above:

A few sentences of evidence, each with a number or a named procedure, will carry more weight with an examiner than a page of definitions. Some departments want your own sample’s values in the methodology chapter, as here, and others in the results chapter beside your descriptive statistics, so check your handbook or a recent thesis from your program.

Common mistakes in this section

A few problems come up again and again when we review this part of a draft:

• A limitations discussion headed “validity and reliability”. Limitations are what your design couldn’t do. Validity and reliability are the evidence that your measures work. They belong in different sections.
• A high response rate offered as evidence of validity. A response rate tells you something about the risk of non-response bias, a kind of selection bias, not about whether your instrument measures the right thing.
• Software headings used as thesis headings. SPSS labels its output “Reliability”; your thesis should organize the results around your research questions instead.
• The original authors’ alpha reported, and nothing from your own data. Always run it on your sample.

Fix those four, and you’ll avoid most of the comments examiners write on this section. Your design’s constraints belong in a separate section, and we’ve covered how to write up your study’s limitations in detail.

What about qualitative research?

Validity and reliability come from measurement, so they fit qualitative research awkwardly. You aren’t measuring a construct with an instrument, and another researcher with the same interviews wouldn’t be expected to produce identical themes.

Most qualitative researchers use trustworthiness instead, an approach set out by Lincoln and Guba in the 1980s, with four criteria that roughly parallel the quantitative ones: credibility, transferability, dependability and confirmability. Each is supported by named procedures, such as member checking, an audit trail and a reflexive journal. We’ve explained each trustworthiness criterion and how to demonstrate it separately.

If your coding involves more than one coder working from a codebook, you may still report inter-coder agreement. Check first, though, whether your analysis method expects it, since not every qualitative approach treats agreement between coders as a goal.

Still have questions?

What’s the difference between validity and reliability?

Validity is whether an instrument measures what it’s meant to measure. Reliability is whether it gives consistent results each time. A scale that always reads 2kg heavy is reliable but not valid. You need both, and a measure can’t be valid unless it’s reliable, because inconsistent readings can’t be accurate.

Can research be valid but not reliable?

Not really. If a measure gives different results each time it measures the same thing, no single result can be trusted, so it can’t be valid. The reverse is common, though: a measure can be perfectly consistent and still measure the wrong thing.

What are the four types of validity?

The four usually listed for an instrument are face, content, construct and criterion validity. Construct validity is often split into convergent and discriminant validity, and criterion validity into concurrent and predictive. Internal and external validity are also types, but they describe a study’s design and generalizability, not an instrument.

What are the four types of reliability?

The four are internal consistency, test-retest, inter-rater and parallel forms reliability. Internal consistency, usually reported as Cronbach’s alpha, is the one most survey-based dissertations need. Inter-rater reliability matters whenever people code or score your data.

What is a good Cronbach’s alpha?

0.70 is the most widely cited minimum, and 0.80 or above is usually described as good. Values above about 0.90 can mean some items are redundant. Report alpha for each scale or subscale separately, using your own sample’s data.

What is the weakest form of validity?

Face validity is usually considered the weakest, because it only asks whether the items look like they measure the concept. It’s a judgment rather than a test. It’s still worth reporting, but it shouldn’t be your only validity evidence.

How do you ensure validity in a questionnaire?

Start with a clear definition of each construct from your theory. Use an existing validated scale where one exists, have experts review your items for content validity, and pilot the questionnaire. If your sample allows, a factor analysis adds construct validity evidence.

Do qualitative studies need validity and reliability?

Most qualitative studies use trustworthiness instead, judged by credibility, transferability, dependability and confirmability. Some qualitative researchers still use the terms validity and reliability, so follow your university’s guidance and your chosen method’s own literature. Either way, name the specific procedures you used.

Can’t find your answer here? Ask a Grad Coach directly – the initial chat is free.

Speak with a friendly coach →

Cite-worthy sources

Cite this article

Jansen, D. (2023). Validity & Reliability in Research. Grad Coach. https://gradcoach.com/validity-reliability-research/

Copy citationMore citation formats

• de Villier, A. (2025). The Mediterranean diet & heart disease: Measuring knowledge through provider education [Doctor of Nursing Practice project, University of Massachusetts Amherst].
• Huang, Y. (2019). Undergraduate students’ perceptions of culturally responsive teaching and their sense of belonging and academic self-efficacy in higher education [Doctoral dissertation, Purdue University].
• Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163.
• Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Sage.
• Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53-55.
• Tunstall, C. M. (2026). Confirmatory factor analysis and further validation of the Trent Measure of Vicarious Trauma (TMVT) [Doctoral thesis, University of Lincoln].