Validity, Reliability, and their relationship

5. Validity Test & Reliability Test

and

Sampling Design & Procedures

Improving Response Rates

Scale Evaluation

Alternative Forms

Split-half

Internal Consistency

Validity, Reliability, and their relationship

  • Validity is the degree to which a study measures what it was designed to measure. It deals with the quality of measurement.
  • Reliability is the extent to which a variable is consistent in what is intended to measure, in other words, it is the consistency, dependability, or repeatability of measures.
  • Relationship between validity and reliability
  • Reliability does not necessarily tell whether the measurement is measuring what is supposed to be measured. Compared to validity, which addresses the issue of what should be measured, reliability is related to how it is measured. Therefore, in order to minimize our measurement error, both reliability and validity are examined.
  • A measure may be reliable but not valid, but it cannot be valid without being reliable. That is, reliability is a necessary but not sufficient condition for validity.
  • Reliability is a necessary, but not sufficient, condition for validity.

Validity

  • Content validity addresses whether the scales adequately measure the domain content of the construct. It is a subjective but systematic evaluation of how well the content of a scale represents the measurement task at hand. There is no objective statistical test to evaluate content validity. Researchers must carefully utilize specified theoretical descriptions of the construct to judge content validity.
  • Criterion validity reflects whether a scale performs as expected in relation to other variables selected (criterion variables) as meaningful criteria, i.e., is the proposed measures exhibit generally the same direction and magnitude of the correlation with other variables of which as the measures have already been accepted within the social science community.
  • The establishment of construct validity involves two major subdomains
  • convergent validity
  • discriminant validity

Construct validity

Convergent validity is the extent to which the scale correlates positively with other measures of the same construct.

To test for convergent validity, we can use Factor Analysis and examine the factor loadings and the significance level of each construct. When the factor loadings of intended constructs are all higher than .50, indicating convergent validity has been achieved. (common factor analysis – PAF) We can also use AVE to test convergent validity.

Discriminant validity is the extent to which a measure does not correlate with other constructs from which it is supposed to differ. In other words, it describes the degree to which one construct is not similar to any other construct that is theoretically distinct.

To test for discriminant validity, CFA can be used.

Reliability

Reliability can be defined as the extent to which measures are free from random error. Researchers must demonstrate instruments are reliable since without reliability, research results using the instrument are not replicable.

Reliability is estimated in one of four ways

Internal consistency

Split-half reliability

Test-retest reliability

Alternative forms

Reliability

Internal consistency reliability: estimation based on the correlation among the variables comprising the set (typically, Cronbach’s alpha).

Split-half reliability: estimation based on the correlation of two equivalent forms of the scale.

Test-retest reliability: Estimation based on the correlation between two (or more) administrations of the same item, scale, or instrument for different times, locations, or populations, when the two administrations do not differ on other relevant variables.

Alternative-forms reliability: two equivalent forms of the scale are constructed and the same respondents are measured at two different times, with a different form being used each time.

Cronbach’s alpha

Cronbach’s alpha, the coefficient of reliability, is frequently used to measure internal consistency and stability of an instrument (Churchill, 1979). It is the average of all possible split-half coefficients resulting from different ways of splitting the scale items.

Cronbach’s alpha varies from 0 to 1, and a value of 0.6 or less generally indicates unsatisfactory internal consistency reliability.

The Sampling Design Process

Define the Target Population

The target population is the collection of elements or objects that possess the information sought by the researcher and about which inferences are to be made. The target population should be defined in terms of elements, sampling units, extent, and time.

An element is the object about which or from which the information is desired, e.g., the respondent.

A sampling unit is an element, or a unit containing the element, that is available for selection at some stage of the sampling process.

Extent refers to the geographical boundaries.

Time is the time period under consideration.

Classification of Sampling Techniques

Convenience Sampling

Convenience sampling attempts to obtain a sample of convenient elements. Often, respondents are selected because they happen to be in the right place at the right time.

use of students, and members of social organizations

department stores using charge account lists

A Graphical Illustration of Convenience Sampling

Group D happens to assemble at a convenient time and place. So all the elements in this Group are selected. The resulting sample consists of elements 16, 17, 18, 19 and 20. Note, no elements are selected from group A, B, C and E.

A B C D E
1 6 11 16 21
2 7 12 17 22
3 8 13 18 23
4 9 14 19 24
5 10 15 20 25

Judgmental Sampling

Judgmental sampling is a form of convenience sampling in which the population elements are selected based on the judgment of the researcher.

test markets

purchase engineers selected in industrial marketing research

expert witnesses used in court

Graphical Illustration of Judgmental Sampling

The researcher considers groups B, C and E to be typical and convenient. Within each of these groups one or two elements are selected based on typicality and convenience. The

resulting sample consists of elements 8, 10, 11, 13, and 24. Note, no elements are selected

from groups A and D.

A B C D E
1 6 11 16 21
2 7 12 17 22
3 8 13 18 23
4 9 14 19 24
5 10 15 20 25

Quota Sampling

Quota sampling may be viewed as two-stage restricted judgmental sampling.

The first stage consists of developing control categories, or quotas, of population elements.

In the second stage, sample elements are selected based on convenience or judgment.

Population Sample
composition composition
Control
Characteristic Percentage Percentage Number
Sex
Male 48 48 480
Female 52 52 520
____ ____ ____
100 100 1000

What sampling technique do you recommend for DuPont Case?

Quota samples are most applicable for mall intercept interviews because they allow for more precision than regular judgmental sampling and mall intercept interviews are inherently non-probabilistic.

We can create control categories along age groups. For example

Age %

22–30 20

31–45 43

45–60 18

60+ 19

  • which represents the percentage of the sample size, which should be obtained from each category. Respondents are approached in the mall with the goal of achieving this age distribution.
  • In this case, we also want to bias our selection in terms of women, since they purchase most carpeting. Thus, we should purposely target women in these age groups at a 2 to 1 ratio to men.

A Graphical Illustration of
Quota Sampling

A quota of one element from each group, A to E, is imposed. Within each group, one element is selected based on judgment or convenience. The resulting sample consists of elements 3, 6, 13, 20 and 22. Note, one element is selected from each column or group.

A B C D E
1 6 11 16 21
2 7 12 17 22
3 8 13 18 23
4 9 14 19 24
5 10 15 20 25

Snowball Sampling

In snowball sampling, an initial group of respondents is selected, usually at random.

After being interviewed, these respondents are asked to identify others who belong to the target population of interest.

Subsequent respondents are selected based on the referrals.

A Graphical Illustration of
Snowball Sampling

Elements 2 and 9 are selected randomly from groups A and B. Element 2 refers elements 12 and 13. Element 9 refers

element 18. The resulting sample consists of elements 2, 9, 12, 13, and 18. Note, there are no element from group E.

A B C D E
1 6 11 16 21
2 7 12 17 22
3 8 13 18 23
4 9 14 19 24
5 10 15 20 25

Classification of Sampling Techniques

Simple Random Sampling

Each element in the population has a known and equal probability of selection.

Each possible sample of a given size (n) has a known and equal probability of being the sample actually selected.

This implies that every element is selected independently of every other element.

A Graphical Illustration of
Simple Random Sampling

Select five random numbers from 1 to 25. The resulting sample consists of population elements 3, 7, 9, 16, and 24.

A B C D E
1 6 11 16 21
2 7 12 17 22
3 8 13 18 23
4 9 14 19 24
5 10 15 20 25

Systematic Sampling

The sample is chosen by selecting a random starting point and then picking every ith element in succession from the sampling frame.

The sampling interval, i, is determined by dividing the population size N by the sample size n and rounding to the nearest integer.

When the ordering of the elements is related to the characteristic of interest, systematic sampling increases the representativeness of the sample.

Systematic Sampling

If the ordering of the elements produces a cyclical pattern, systematic sampling may decrease the representativeness of the sample.

For example, there are 100,000 elements in the population and a sample of 1,000 is desired. In this case the sampling interval, i, is 100. A random number between 1 and 100 is selected. If, for example, this number is 23, the sample consists of elements 23, 123, 223, 323, 423, 523, and so on.

A Graphical Illustration of
Systematic Sampling

Select a random number between 1 to 5, say 2.

The resulting sample consists of population 2,

(2+5=) 7, (2+5×2=) 12, (2+5×3=)17, and (2+5×4=) 22. Note, all the elements are selected from a single row.

A B C D E
1 6 11 16 21
2 7 12 17 22
3 8 13 18 23
4 9 14 19 24
5 10 15 20 25

Stratified Sampling

A two-step process in which the population is partitioned into subpopulations, or strata.

The strata should be mutually exclusive and collectively exhaustive in that every population element should be assigned to one and only one stratum and no population elements should be omitted.

Next, elements are selected from each stratum by a random procedure, usually SRS.

A major objective of stratified sampling is to increase precision without increasing cost.

The elements within a stratum should be as homogeneous as possible, but the elements in different strata should be as heterogeneous as possible.

The stratification variables should also be closely related to the characteristic of interest.

A Graphical Illustration of
Stratified Sampling

Randomly select a number from 1 to 5

for each stratum, A to E. The resulting

sample consists of population elements

4, 7, 13, 19 and 21. Note, one element

is selected from each column.

A B C D E
1 6 11 16 21
2 7 12 17 22
3 8 13 18 23
4 9 14 19 24
5 10 15 20 25

Cluster Sampling

The target population is first divided into mutually exclusive and collectively exhaustive subpopulations, or clusters.

Elements within a cluster should be as heterogeneous as possible, but clusters themselves should be as homogeneous as possible. Ideally, each cluster should be a small-scale representation of the population.

Then a random sample of clusters is selected, based on a probability sampling technique such as SRS.

For each selected cluster, either all the elements are included in the sample (one-stage) or a sample of elements is drawn probabilistically (two-stage).

A Graphical Illustration of
Cluster Sampling (2-Stage)

Randomly select 3 clusters, B, D and E.

Within each cluster, randomly select one

or two elements. The resulting sample

consists of population elements 7, 18, 20, 21, and 23. Note, no elements are selected from clusters A and C.

A B C D E
1 6 11 16 21
2 7 12 17 22
3 8 13 18 23
4 9 14 19 24
5 10 15 20 25

Strengths and Weaknesses of
Basic Sampling Techniques

Procedures for Drawing
Probability Samples

Procedures for Drawing
Probability Samples

Procedures for Drawing
Probability Samples

Procedures for Drawing
Probability Samples

Tennis’ Systematic Sampling
Returns a Smash

Tennis magazine conducted a mail survey of its subscribers to gain a better understanding of its market. Systematic sampling was employed to select a sample of 1,472 subscribers from the publication’s domestic circulation list. If we assume that the subscriber list had 1,472,000 names, the sampling interval would be 1,000 (1,472,000/1,472). A number from 1 to 1,000 was drawn at random. Beginning with that number, every 1,000th subscriber was selected.

A brand-new dollar bill was included with the questionnaire as an incentive to respondents. An alert postcard was mailed one week before the survey. A second, follow-up, questionnaire was sent to the whole sample ten days after the initial questionnaire. The net effective mailing was 1,396. Six weeks after the first mailing, 778 completed questionnaires were returned, yielding a response rate of 56%.

Discussion question 1

  • Discuss the advantages of convenience samples and when it is appropriate to use them.
  • Convenience sampling is the least expensive and least time consuming of all sampling techniques. The sampling units are accessible, easy to measure, and cooperative.
  • In spite of these advantages, this form of sampling has serious limitations. Many potential sources of selection bias are present, including respondent self-selection. Convenience samples are not representative of any definable population. Hence, it is not theoretically meaningful to generalize to any population from a convenience sample, and convenience samples are not appropriate for marketing research projects involving population inferences.
  • Convenience samples are not recommended for descriptive or causal research, but they can be used in exploratory research for generating ideas, insights, or hypotheses.
  • Convenience samples can be used for focus groups, pretesting questionnaires, or pilot studies. Even in these cases, caution should be exercised in interpreting the results. Nevertheless, this technique is sometimes used even in large surveys.

Discussion question 2

  • Discuss the advantages of systematic sampling.
  • Systematic sampling is less costly and easier than simple random sampling, because random selection is done only once.
  • Moreover, the random numbers do not have to be matched with individual elements as in simple random sampling. Because some lists contain millions of elements, considerable time can be saved. This reduces the costs of sampling. If information related to the characteristic of interest is available for the population, systematic sampling can be used to obtain a more representative and reliable (lower sampling error) sample than simple random sampling.
  • Another relative advantage is that systematic sampling can even be used without knowledge of the composition (elements) of the sampling frame. For example, every ith person leaving a department store or mall can be intercepted. For these reasons, systematic sampling is often employed in consumer mail, telephone, and mall intercept interviews.

Discussion question 3

  • Discuss the uses of nonprobability and probability sampling.
  • Nonprobability sampling is used in concept tests, package tests, name tests, and copy tests, where projections to the populations are usually not needed. In such studies, interest centers on the proportion of the sample that gives various responses or expresses various attitudes. Samples for these studies can be drawn using methods such as mall intercept quota sampling.
  • On the other hand, probability sampling is used when there is a need for highly accurate estimates of market share or sales volume for the entire market. National market tracking studies, which provide information on product category and brand usage rates, as well as psychographic and demographic profiles of users, use probability sampling. Studies that use probability sampling generally employ telephone interviews. Stratified and systematic sampling are combined with some form of random-digit dialing to select the respondents.

*

 
Do you need a similar assignment done for you from scratch? We have qualified writers to help you. We assure you an A+ quality paper that is free from plagiarism. Order now for an Amazing Discount!
Use Discount Code "Newclient" for a 15% Discount!

NB: We do not resell papers. Upon ordering, we do an original paper exclusively for you.

/by