To Survey or Not to Survey? What a Good Question. 

Wondering if a survey is the right method for your study? You have come to the right place!

Surveys are a popular method of gathering data from individuals using a set of pre-written questions. The primary goal of a survey is to systematically collect and analyze data about various experiences, perceptions, characteristics, or behaviors within a group of individuals. 

The word “survey” is often used to reference the set of questions used to collect data when, in fact, it represents a multifaceted data collection process. For the purposes of this resource and to avoid any confusion, we refer to the instruments or sets of questions as “questionnaires,” and use the term “survey” to refer to the overarching data collection process depicted below. 

Surveys can be a valuable and versatile way to collect data about respondent experiences and perceptions in a wide range of situations. In particular, surveys can be a good approach when the following conditions are met: 

• Study objectives aim to quantify experiences, behaviors, or opinions for a particular audience. 

• Data will be collected from a large or diverse audience. 

• The study is focused on collecting feedback from stakeholders to gauge overall engagement, satisfaction, or identify opportunities for improvement. 

• The audience may prefer to share their opinions or experiences anonymously. 

• You can confidently develop a short list of all possible responses for each question.  

While surveys are a versatile tool for collecting opinion-based data, they may not always be the most appropriate or effective approach. You should consider another method if: 

• The research topic is highly sensitive or personal, and respondents may be uncomfortable sharing information even with the promise of anonymity. 

• The study aims to gain holistic understandings of complex phenomena.  

• Audiences might be facing challenges that hinder informed participation. These may include but are not limited to: literacy challenges, language barriers, lack of access to necessary technologies, et cetera. 

• Topics are highly politicized and biased questions may compromise the validity of results. 

• There are limited resources for analyzing the large amounts of data generated by a survey. 
  • This can be particularly true when surveys collect open text responses for analysis. 
• The intended audience suffers from survey fatigue and may not be motivated to respond.  

• You want to ask multiple open-ended (i.e., free response) questions.  

Although survey processes are widely used for collecting experience and perception-based data, there are inherent limitations of this method.  

• Design Challenges: A survey is only as good as the quality of the questionnaire it uses. Designing a clear and unbiased questionnaire is essential in minimizing biased responses and misinterpreted questions. The distribution method, whether digital, paper, or phone based, should also match the needs of the individuals you are sampling. 

• Sampling Issues: Sampling issues can exacerbate various biases and ultimately provide a non-representative sample for analysis. Ensuring that appropriate sampling methods are employed helps minimize bias and strengthens the representativeness and generalizability of results. Consider both who you are surveying and how many individuals you need to sample in order to get a useful response rate. 

• Response Bias: Respondents may provide answers they consider to be more “acceptable” or otherwise misrepresent their true opinions. This can be especially relevant when surveying audiences about sensitive topics. 

• Low Response Rates: Survey fatigue, poorly timed or placed distribution, and lack of engagement in your target audience often result in low response rates, which impacts the utility of findings.