7.2.14 Reliability & Validity

Reliability 

• Reliability is achieved if a study has been set up so that the IV can be seen to affect the DV:
  • If the procedure is replicated, the study should show similar results
  • Replicating a study and finding similar results shows that the measure is consistent and not liable to fluctuate to any significant degree
• If a study is completed using a standardised procedure (see the previous page on procedures used in research) and obtains the same results, it can be said that is is reliable 
• Lab experiments are the most reliable method used in psychology as they:
  • Take place in neutral space under controlled conditions
  • Follow a standardised procedure
  • Use random allocation to conditions
  • Tend to use a control group as comparison to the experimental group
  • Generate quantitative data which is easy to compare and analyse

• Field experiments generate quantitative data and implement an IV but they are subject to an array of extraneous variables over which the research has no control, making them less reliable than lab experiments

• Natural experiments generate quantitative data but they use a naturally-occurring IV over which the research has no control making them less reliable than lab experiments

• There are both internal reliability and external reliability 
  • Internal reliability: The extent something is consistent with itself 
  • External reliability: The extent a test measure is consistent over time

Assessing Reliability 

• The reliability of a questionnaire can be assessed using two methods:
  • The test-retest method measures external reliability: 
    • The same participants are given the same questionnaire at separate time intervals (e.g. with a 6-month gap in between testing sessions) 
    • If the same result is found per participant then external reliability is established
  • The split-half method measures internal reliability:
    • The researcher splits the test in half and analyses the responses given to the first half of the questionnaire compared to the second half of the questionnaire
    • If similar responses are given in both halves then internal reliability is established
• Inter-observer reliability 
• For an observational study the researchers will have previously agreed on which specific behaviours should be recorded so that all observers are on ‘the same page’ prior to the observation 
• Researchers can test the reliability of their of their observations by comparing them with another researcher's recording of their behaviours: the level of consistency between the two records is then compared
• Inter-observer reliability is the level of consistency between two or more trained observers when they conduct the same observation, as follows:
  • All observers must agree on the behaviour categories and how they are going to record them before the observation begins
  • The observation is conducted separately by each observer to avoid conformity (i.e. one observer may be influenced by one or more other observers)
  • After the observational period:
    • The observers compare the two independent data sets (often designed as a tally chart)
    • They then test the correlation between the two sets
    • If there is a strong positive correlation between the sets then this shows that there is good inter-observer reliability and that the behaviour categories are reliable
• Establishing good inter-observer reliability means that there is less chance that researcher bias has interfered with the observation

Behaviour	Tally
Clenches fist	III
Frowns	IIII
Crosses arms	IIII III
Raises eyebrows	IIII

An example of a tally chart used to record categories of behaviour in an observation.

Validity 

• Validity focuses on accuracy 
• Accurately measuring what the study sets out to 
• What extent can the findings be generalised to the wider population and out of the research setting 
• There is both internal and external validity 
  • Internal validity: Measures whether the results are due to to the manipulation of the independent variable and not confounding variables. 
  • Internal validity can be improved by reducing investigator effects and demand characteristics. 
  • External validity: Measure whether the results can be generalised beyond the research setting, looking at;
    • Ecological validity: Whether it can be generalised to other settings 
    • Population validity: Whether it can be generalised to other people 
    • Temporal validity: When it can be generalised over time 
  • External validity can be improved by setting research/experiments in naturalistic environments   

Assessing Validity 

• Predictive validity: Assesses validity by predicting how well a test does at predicting future behaviour 
• Temporal validity: Assesses how valid it remains over time 
• Concurrent validity: Assesses through correlation, correlating scores from research already existing and known to be valid 
• Face validity: Assesses whether something is what it looks like, to what extent does the item look like what the test measures