Survey Research and Questionnaires
Survey research is a commonly used method of collecting information about a population of interest. There are many different types of surveys, several ways to administer them, and many methods of sampling. There are two key features of survey research:
- Questionnaires -- a predefined series of questions used to collect information from individuals
- Sampling -- a technique in which a subgroup of the population is selected to answer the survey questions; the information collected can be generalized to the entire population of interest
The two most common types of survey questions are closed-ended questions and open-ended questions.
- The respondents are given a list of predetermined responses from which to choose their answer
- The list of responses should include every possible response and the meaning of the responses should not overlap
- An example of a close-ended survey question would be, "Please rate how strongly you agree or disagree with the following statement: 'I feel good about my work on the job.' Do you strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, or strongly disagree?"
- A Likert scale, which is used in the example above, is a commonly used set of responses for closed-ended questions
- Closed-ended questions are usually preferred in survey research because of the ease of counting the frequency of each response
- Survey respondents are asked to answer each question in their own words
- Responses are usually categorized into a smaller list of responses that can be counted by the study team for statistical analysis
Considerations for Designing a Questionnaire
- It is important to consider the order in which questions are presented. Sensitive questions, such as questions about income, drug use, or sexual activity, should be put at the end of the survey. This allows the researcher to establish trust before asking questions that might embarrass respondents. Researchers also recommend putting routine questions, such as age, gender, and marital status, at the end of the questionnaire
- Double-barreled questions, which ask two questions in one, should never be used in a survey. An example of a double barreled question is, "Please rate how strongly you agree or disagree with the following statement: 'I feel good about my work on the job, and I get along well with others at work.'" This question is problematic because survey respondents are asked to give one response for two questions
- Researchers should avoid using emotionally loaded or biased words and phrases
Visit the following websites for more information about questionnaire design:
Glossary terms related to questionnaire design:
Surveys can be admininistered in three ways:
- Through the mail
- Advantage: Low cost
- Disadvantage: Low response rate
- By telephone
- Advantages: Higher response rates; responses can be gathered more quickly
- Disadvantage: More expensive than mail surveys
- Advantages: Highest response rates; better suited to collecting complex information
- Disadvantage: Very expensive
Visit the following website for more information about survey administration:
Glossary terms related to survey administration:
One of the primary strengths of sampling is that accurate estimates of a population's characteristics can be obtained by surveying a small proportion of the population. Four sampling techniques are described here:
Simple Random Sampling
- Simple random sampling is the most basic form of sampling
- Every member of the population has an equal chance of being selected
- This sampling process is similar to a lottery: the entire population of interest could be selected for the survey, but only a few are chosen at random
- Researchers often use random-digit dialing to perform simple random sampling. In this procedure, telephone numbers are generated by a computer at random and called to identify individuals to participate in the survey
- Cluster sampling is generally used when it is geographically impossible to undertake a simple random sample
- Cluster sampling requires that adjustments be made in statistical analyses
For example, in a face-to-face interview, it is difficult and expensive to survey households across the nation. Instead, researchers will randomly select geographic areas (for example, counties), then randomly select households within these areas. This creates a cluster sample, in which respondents are clustered together geographically.
- Stratified samples are used when a researcher wants to ensure that there are enough respondents with certain characteristics in the sample
- The researcher first identifies the people in the population who have the desired characteristics, then randomly selects a sample of them
- Stratified sampling requires that adjustments be made in statistical analyses
For example, a researcher may want to compare survey responses of African-Americans and Caucasians. To ensure that there are enough Afrian-Americans in the survey, the researcher will first identify the African-Americans in the population and then randomly select a sample of African-Americans.
- Common nonrandom sampling techniques include convenience sampling and snowball sampling
- Nonrandom samples cannot be generalized to the population of interest. Consequently, it is problematic to make inferences about the population
- In survey research, random, cluster, or stratified samples are preferable
Visit the following websites for more information about sampling procedures:
Glossary terms related to sampling procedures:
Measurement error is the difference between the target population's characteristics and the measurement of these characteristics in a survey. There are two types of measurement error: systematic error and random error.
- Systematic error is more serious than random error
- Occurs when the survey responses are systematically different from the target population responses
- For example, if a researcher only surveyed individuals who answered their phone between 9 and 5, Monday through Friday, the survey results would be biased toward individuals who are unemployed
- Sources of bias include
- Nonobservational error -- Individuals in the target population are systematically excluded from the sample, such as in the example above
- Observational error -- When respondents systematically answer surveys question incorrectly. For example, surveys that ask respondents how much they weigh will probably underestimate the population's weight because respondents are likely to underreport their weight
- Random error is an expected part of survey research, and statistical techniques are designed to account for this sort of measurement error
- Occurs because of natural and uncontrollable variations in the survey process, i.e., the mood of the respondent
For example, a researcher may administer a survey about marital happiness. However, some respondents may have had a fight with their spouse the evening prior to the survey, while other respondents' spouses may have cooked the respondent's favorite meal. The survey responses will be affected by the random day on which the respondents were chosen to participate in the study. With random error, the positive and negative influences on the survey measure balance out.
Visit the following website for more information about measurement error:
Glossary terms related to measurement error:
Nonresponse Rate Bias
Ethics of Survey Research
Respondents should give informed consent before participating in a survey. In order for respondents to give informed consent,
- The researcher must inform the respondents of the study's purpose, content, duration, and potential risks and benefits
- The researcher must inform the respondents that they do not have to answer all the survey questions
- The researcher must inform the resondents that they can stop participating in the study at any point
Confidentiality and Anonymity
It is absolutely imperative that researchers keep respondents' identities confidential. To ensure confidentiality, researchers should not link respondents' identifiers to their survey responses when using data. Common identifiers include names, social security numbers, addresses, and telephone numbers.
Anonymity is an even stronger safeguard of respondent privacy. If a researcher assures anonymity, it means that the researcher is unable to link respondents' names to their surveys.
Visit the following websites for more information about anonymity:
Glossary terms related to ethics:
Advantages and Disadvantages of Survey Research
- Sample surveys are a cost-effective and efficient means of gathering information about a population
- Survey sampling makes it possible to accurately estimate the characteristics of a target population without interviewing all members of the population
Survey sampling is particularly useful when the population of interest is very large or dispersed across a large geographic area.
- Surveys do not allow researchers to develop an intimate understanding of individual circumstances or the local culture that may be the root cause of respondent behavior
- Respondents often will not share sensitive information in the survey format
- A growing problem in survey research is the widespread decline in response rates
Elements of a research proposal and report
2005 © David S. Walonick, Ph.D.
All rights reserved.
Excerpts from Survival Statistics - an applied statistics book for graduate students.
All research reports use roughly the same format. It doesn't matter whether you've done a customer satisfaction survey, an employee opinion survey, a health care survey, or a marketing research survey. All have the same basic structure and format. The rationale is that readers of research reports (i.e., decision makers, funders, etc.) will know exactly where to find the information they are looking for, regardless of the individual report.
Once you've learned the basic rules for research proposal and report writing, you can apply them to any research discipline. The same rules apply to writing a proposal, a thesis, a dissertation, or any business research report.
The Research Proposal and Report
Research papers usually have five chapters with well-established sections in each chapter. Readers of the paper will be looking for these chapters and sections so you should not deviate from the standard format unless you are specifically requested to do so by the research sponsor.
Most research studies begin with a written proposal. Again, nearly all proposals follow the same format. In fact, the proposal is identical to the first three chapters of the final paper except that it's writtten in future tense. In the proposal, you might say something like "the researchers will secure the sample from ...", while in the final paper, it would be changed to "the researchers secured the sample from ...". Once again, with the exception of tense, the proposal becomes the first three chapters of the final research paper.
The most commonly used style for writing research reports is called "APA" and the rules are described in the Publication Manual of the American Psychological Association. Any library or bookstore will have it readily available. The style guide contains hundreds of rules for grammar, layout, and syntax. This paper will cover the most important ones.
Avoid the use of first person pronouns. Refer to yourself or the research team in third person. Instead of saying "I will ..." or "We will ...", say something like "The researcher will ..." or "The research team will ...".
A suggestion: Never present a draft (rough) copy of your proposal, thesis, dissertation, or research paper...even if asked. A paper that looks like a draft, will interpreted as such, and you can expect extensive and liberal modifications. Take the time to put your paper in perfect APA format before showing it to anyone else. The payoff will be great since it will then be perceived as a final paper, and there will be far fewer changes.
Style, layout, and page formatting
All text on the title page is centered vertically and horizontally. The title page has no page number and it is not counted in any page numbering.
Left margin: 1½"
Right margin: 1"
Top margin: 1"
Bottom margin: 1"
Pages are numbered at the top right. There should be 1" of white space from the top of the page number to the top of the paper. Numeric page numbering begins with the first page of Chapter 1 (although a page number is not placed on page 1).
Spacing and justification
All pages are single sided. Text is double-spaced, except for long quotations and the bibliography (which are single-spaced). There is one blank line between a section heading and the text that follows it. Do not right-justify text. Use ragged-right.
Font face and size
Any easily readable font is acceptable. The font should be 10 points or larger. Generally, the same font must be used throughout the manuscript, except 1) tables and graphs may use a different font, and 2) chapter titles and section headings may use a different font.
APA format should be used to cite references within the paper. If you name the author in your sentence, then follow the authors name with the year in parentheses. For example:
Jones (2004) found that...
If you do not include the authors name as part of the text, then both the author's name and year are enclosed in parentheses. For example:
One researcher (Jones, 2004) found that...
A complete bibliography is attached at the end of the paper. It is double spaced except single-spacing is used for a multiple-line reference. The first line of each reference is indented.
Bradburn, N. M., & Mason, W. M. (1964). The effect of question order on response. Journal of Marketing Research1 (4), 57-61.
Bradburn, N. M., & Miles, C. (1979). Vague quantifiers. Public Opinion Quarterly43 (1), 92-101.
Outline of chapters and sections
TABLE OF CONTENTS
CHAPTER I - Introduction
Statement of the problem
Significance of the study
Research questions and/or hypotheses
CHAPTER II - Background
Definition of terms
CHAPTER III - Methodology
Restate purpose and research questions or null hypotheses
Population and sampling
Instrumentation (include copy in appendix)
Procedure and time frame
Analysis plan (state critical alpha level and type of statistical tests)
Validity and reliability
Scope and limitations
CHAPTER IV - Results
CHAPTER V - Conclusions and recommendations
Summary (of what you did and found)
Discussion (explanation of findings - why do you think you found what you did?)
Recommendations (based on your findings)
Chapter I - Introduction
Chapter I begins with a few short introductory paragraphs (a couple of pages at most). The primary goal of the introductory paragraphs is to catch the attention of the readers and to get them "turned on" about the subject. It sets the stage for the paper and puts your topic in perspective. The introduction often contains dramatic and general statements about the need for the study. It uses dramatic illustrations or quotes to set the tone. When writing the introduction, put yourself in your reader's position - would you continue reading?
Statement of the Problem
The statement of the problem is the focal point of your research. It is just one sentence (with several paragraphs of elaboration).
You are looking for something wrong.
....or something that needs close attention
....or existing methods that no longer seem to be working.
Example of a problem statement:
"The frequency of job layoffs is creating fear, anxiety, and a loss of productivity in middle management workers."
While the problem statement itself is just one sentence, it is always accompanied by several paragraphs that elaborate on the problem. Present persuasive arguments why the problem is important enough to study. Include the opinions of others (politicians, futurists, other professionals). Explain how the problem relates to business, social or political trends by presenting data that demonstrates the scope and depth of the problem. Try to give dramatic and concrete illustrations of the problem. After writing this section, make sure you can easily identify the single sentence that is the problem statement.
The purpose is a single statement or paragraph that explains what the study intends to accomplish. A few typical statements are:
The goal of this study is to...
... overcome the difficulty with ...
... discover what ...
... understand the causes or effects of ...
... refine our current understanding of ...
... provide a new interpretation of ...
... understand what makes ___ successful or unsuccessful
Significance of the Study
This section creates a perspective for looking at the problem. It points out how your study relates to the larger issues and uses a persuasive rationale to justify the reason for your study. It makes the purpose worth pursuing. The significance of the study answers the questions:
Why is your study important?
To whom is it important?
What benefit(s) will occur if your study is done?
Research Questions and/or Hypotheses and/or Null Hypotheses
Chapter I lists the research questions (although it is equally acceptable to present the hypotheses or null hypotheses). No elaboration is included in this section. An example would be:
The research questions for this study will be:
1. What are the attitudes of...
2. Is there a significant difference between...
3. Is there a significant relationship between...
Chapter II - Background
Chapter II is a review of the literature. It is important because it shows what previous researchers have discovered. It is usually quite long and primarily depends upon how much research has previously been done in the area you are planning to investigate. If you are planning to explore a relatively new area, the literature review should cite similar areas of study or studies that lead up to the current research. Never say that your area is so new that no research exists. It is one of the key elements that proposal readers look at when deciding whether or not to approve a proposal.
Chapter II should also contain a definition of terms section when appropriate. Include it if your paper uses special terms that are unique to your field of inquiry or that might not be understood by the general reader. "Operational definitions" (definitions that you have formulated for the study) should also be included. An example of an operational definition is: "For the purpose of this research, improvement is operationally defined as posttest score minus pretest score".
Chapter III - Methodology
The methodology section describes your basic research plan. It usually begins with a few short introductory paragraphs that restate purpose and research questions. The phraseology should be identical to that used in Chapter I. Keep the wording of your research questions consistent throughout the document.
Population and sampling
The basic research paradigm is:
1) Define the population
2) Draw a representative sample from the population
3) Do the research on the sample
4) Infer your results from the sample back to the population
As you can see, it all begins with a precise definition of the population. The whole idea of inferential research (using a sample to represent the entire population) depends upon an accurate description of the population. When you've finished your research and you make statements based on the results, who will they apply to? Usually, just one sentence is necessary to define the population. Examples are: "The population for this study is defined as all adult customers who make a purchase in our stores during the sampling time frame", or "...all home owners in the city of Minneapolis", or "...all potential consumers of our product".
While the population can usually be defined by a single statement, the sampling procedure needs to be described in extensive detail. There are numerous sampling methods from which to choose. Describe in minute detail, how you will select the sample. Use specific names, places, times, etc. Don't omit any details. This is extremely important because the reader of the paper must decide if your sample will sufficiently represent the population.
If you are using a survey that was designed by someone else, state the source of the survey. Describe the theoretical constructs that the survey is attempting to measure. Include a copy of the actual survey in the appendix and state that a copy of the survey is in the appendix.
Procedure and time frame
State exactly when the research will begin and when it will end. Describe any special procedures that will be followed (e.g., instructions that will be read to participants, presentation of an informed consent form, etc.).
The analysis plan should be described in detail. Each research question will usually require its own analysis. Thus, the research questions should be addressed one at a time followed by a description of the type of statistical tests that will be performed to answer that research question. Be specific. State what variables will be included in the analyses and identify the dependent and independent variables if such a relationship exists. Decision making criteria (e.g., the critical alpha level) should also be stated, as well as the computer software that will be used.
Validity and reliability
If the survey you're using was designed by someone else, then describe the previous validity and reliability assessments. When using an existing instrument, you'll want to perform the same reliability measurement as the author of the instrument. If you've developed your own survey, then you must describe the steps you took to assess its validity and a description of how you will measure its reliability.
Validity refers to the accuracy or truthfulness of a measurement. Are we measuring what we think we are? There are no statistical tests to measure validity. All assessments of validity are subjective opinions based on the judgment of the researcher. Nevertheless, there are at least three types of validity that should be addressed and you should state what steps you took to assess validity.
Face validity refers to the likelihood that a question will be misunderstood or misinterpreted. Pretesting a survey is a good way to increase the likelihood of face validity. One method of establishing face validity is described here. How to make sure your survey is valid.
Content validity refers to whether an instrument provides adequate coverage of a topic. Expert opinions, literature searches, and pretest open-ended questions help to establish content validity.
Construct validity refers to the theoretical foundations underlying a particular scale or measurement. It looks at the underlying theories or constructs that explain a phenomena. In other words, if you are using several survey items to measure a more global construct (e.g., a subscale of a survey), then you should describe why you believe the items comprise a construct. If a construct has been identified by previous researchers, then describe the criteria they used to validate the construct. A technique known as confirmatory factor analysis is often used to explore how individual survey items contribute to an overall construct measurement.
Reliability is synonymous with repeatability or stability. A measurement that yields consistent results over time is said to be reliable. When a measurement is prone to random error, it lacks reliability.
There are three basic methods to test reliability : test-retest, equivalent form, and internal consistency. Most research uses some form of internal consistency. When there is a scale of items all attempting to measure the same construct, then we would expect a large degree of coherence in the way people answer those items. Various statistical tests can measure the degree of coherence. Another way to test reliability is to ask the same question with slightly different wording in different parts of the survey. The correlation between the items is a measure of their reliability. See: How to test the reliability of a survey.
All research studies make assumptions. The most obvious is that the sample represents the population. Another common assumptions are that an instrument has validity and is measuring the desired constructs. Still another is that respondents will answer a survey truthfully. The important point is for the researcher to state specifically what assumptions are being made.
Scope and limitations
All research studies also have limitations and a finite scope. Limitations are often imposed by time and budget constraints. Precisely list the limitations of the study. Describe the extent to which you believe the limitations degrade the quality of the research.
Chapter IV - Results
Description of the sample
Nearly all research collects various demographic information. It is important to report the descriptive statistics of the sample because it lets the reader decide if the sample is truly representative of the population.
The analyses section is cut and dry. It precisely follows the analysis plan laid out in Chapter III. Each research question addressed individually. For each research question:
1) Restate the research question using the exact wording as in Chapter I
2) If the research question is testable, state the null hypothesis
3) State the type of statistical test(s) performed
4) Report the statistics and conclusions, followed by any appropriate table(s)
Numbers and tables are not self-evident. If you use tables or graphs, refer to them in the text and explain what they say. An example is: "Table 4 shows a strong negative relationship between delivery time and customer satisfaction (r=-.72, p=.03)". All tables and figures have a number and a descriptive heading. For example:
The relationship between delivery time and customer satisfaction.
Avoid the use of trivial tables or graphs. If a graph or table does not add new information (i.e., information not explained in the text), then don't include it.
Simply present the results. Do not attempt to explain the results in this chapter.
Chapter V - Conclusions and recommendations
Begin the final chapter with a few paragraphs summarizing what you did and found (i.e., the conclusions from Chapter IV).
Discuss the findings. Do your findings support existing theories? Explain why you think you found what you did. Present plausible reasons why the results might have turned out the way they did.
Present recommendations based on your findings. Avoid the temptation to present recommendations based on your own beliefs or biases that are not specifically supported by your data. Recommendations fall into two categories. The first is recommendations to the study sponsor. What actions do you recommend they take based upon the data. The second is recommendations to other researchers. There are almost always ways that a study could be improved or refined. What would you change if you were to do your study over again? These are the recommendations to other researchers.
List references in APA format alphabetically by author's last name
Include a copy of any actual instruments. If used, include a copy of the informed consent form.