Creating and Deploying Quality Web-based Surveys

From E-Learning Faculty Modules


Contents

Module Summary

Online surveys are a common tool in higher education environments. These are used to conduct social sciences research, evaluate programs, evaluate instructors, poll employees, and conduct other work. Such cloud-based tools (hosted on remote servers) are not your grandma’s survey tool. These enable users to script and code, customize surveys, collect a wide range of information, offer rich branching options, enable automated work, and have built-in data visualizations for online dashboards and exportable and printable reports. Many survey tools (as research suites) even include built-in analytics. Many are using such complex tools without formal training in the art and science of building surveys. This short article introduces some insights on how to create and deploy quality web-based surveys.

Takeaways

Learners will...

  • review some of the quality standards for online survey research
  • review some of the basics about how to design and pilot-test an online respondent-administered survey
  • consider some “best practices” in survey creation and deployment
  • explore some of the technological affordances of modern online survey system
  • consider some in-tool data analytics features for data from online survey research

Module Pretest

1. What are some quality standards for online survey research?

2. What are some of the basics about how to design and pilot-test an online respondent-administered survey?

3. What are some best practices in survey creation and deployment?

4. What are some of the technological affordances of modern online survey systems?

5. What are some in-tool data analytics features for data from online survey research?


Main Contents

This section provides some contents based on the five questions above.

1. What are some quality standards for online survey research?

Optimally, an online survey instrument is designed to collect relevant information from target populations in the most ethical, legal, and objectively accurate way as possible.

IRB oversight. A survey instrument is not created independent of a research context or a particular domain of study. Rather, it should be a part of a thoughtful research design, and it should go through “human subjects review” by a properly empowered institutional review board (IRB). The IRB works to ensure that the research will not cause harm to the participants or to others in society, and they ensure that the data collected during the research is properly handled during and after the research—to ensure that participants’ privacy and well-being are not compromised. Ultimately, research should benefit humanity—particularly the social groups represented by the target participants, and more broadly, others. To submit a survey to the IRB, it is a good idea to have an informed consent write-up to be as accurate as possible. No deception should be included. The rationales for the research should be clear. The methodology should be clear. There should be an accurate and thought-through data plan for how the data will be handled. IRBs have forms that may be filled out for full disclosure.

Once a survey has been vetted and found to be “exempt” or otherwise approved, if there are any material changes to the original survey or deployment methods or mistakes, the researcher (or research team) should go back to the IRB for more advisement and oversight. Likewise, if there are new ways to use the collected information not addressed in the early approved research, it seems wise to return for more feedback. IRB approval does not protect against legal liabilities, but it is a step in the right direction and may prevent some lapses.

Intellectual property and privacy. Beyond the concerns of the IRB (which will not be discussed further here), there are other issues of legality. For example, the online survey may be built from another’s instrument. In some cases, instruments have been commercialized and require payment before they may be legally used. There are subscription databases through which such instruments may be explored, but rights to use such instruments have to be negotiated with the legal owners of those instruments. In other cases, instruments are freely available for use, and researchers may use such works in whole or in part. However, originators of the instruments will expect proper citations and crediting. In this context, others’ intellectual property rights should be respected, not contravened. Another potential legal consideration involves people’s privacy rights. To this end, if people are videotaped, audio-recorded, photographed, or otherwise recorded to serve as part of a multimedia-rich survey, each person has to have signed a legal release before such captures are done, and they have to be properly informed about the limited uses of their recorded likeness in the online survey. Media releases should accurately represent the full uses of the created media, and they should be legally binding.

Accessibility. Online surveys should be fully Section 508 compliant. All audio and video should have closed captioning (or some form of timed text that has the informational equivalency of the original non-textual media). Images should include alt-texting. Colors should be sufficiently high-contrast, and they should not be the only mode through which information is conveyed. Data tables should be screen readable, and as each cell is “read,” it should be clear what the in-cell data refers to. (In other words, the column header and the row header for that cell should be clear.)

If complex question types are used—such as matrix questions, gaps analysis questions, slider questions, hotspot questions, and others—there should be mitigations for those who are using other devices than keyboards to interact with the online survey. Online questions that do not allow keyboard inputs (and keyboard shortcut inputs) are not accessible.

Design for objectivity. While researcher may have a particular preference for the results he / she / they may want to see, a survey instrument should be designed to be as objective as possible—to counteract human subjectivity and cognitive biases as much as possible. The survey designers need to avoid priming respondents to give particular answers. They should understand the nuances in messaging that may lead to systematic biases in the use of the instrument. For example, the uses of particular language or imagery or video may prime particular responses. The order of scaled options in a multiple-choice question affects selection (often, respondents will select the first one and not read through to all the choices, and even if they do, the earlier selections tend to be preferable). To work against such tendencies, some survey designers will randomize order selection…or they will organize options in a least-to-most way (instead of most-to-least).

The limits of self-report. Survey research, at heart, is limited not only because of reach but also because of human tendencies towards self-deception...and towards social-desirable responding. In other words, the respondents themselves have motivations to provide information that may be skewed based on internal and external factors, self-fulfillment and social fulfillment dynamics.

Beyond the design of the survey, there are other ways that results may be manipulated. For example, the few students in a course may be asked to assess an e-book that their professor created. Even if the survey is “anonymized” or identities are kept “confidential,” all learners are likely identifiable in a small class, and the power differential between the professor (who is being assessed) and the learner (who is doing the assessing) is skewing. Survey respondents may think that their responses will affect their grade (and in some cases, it may). Another potential manipulation is the use of excessive reward. If respondents are promised more than a small participation award or a chance at a small participation award, they may engage in research that compromises their own privacy or dignity. IRB review generally includes the analysis of promised rewards to ensure that there is not an imbalance in the promised potential rewards.

Then, too, the actual analysis of the data may introduce skew. Which data is included, and which is not? How is the data cleaned and processed? How are statistical analyses run and interpreted? Which statistical methods are chosen, and were these proper to the research context and the type of data used?

Data sharing. Online survey systems often have built-in ways to share the collected data—through data tables and simple data visualizations (bar charts, pie charts, line graphs, and others)—through online dashboards. Data visualizations, of course, are summary data. On the surface, they do not enable probing of the underlying data. However, while these online dashboards may seem somewhat innocuous, dashboards often enable users to view the data as data tables, which may be exported with all the data columns for analysis in external tools. In other words, unless dashboards can be locked down, they have a high potential of leaking all the underlying data. Worse yet, if the shared datasets were not properly masked and de-identified, all that identifier data is spilled with other information.

With the recent moves towards reproducible research and the concomitant release of datasets to other researchers and the broad public, the dataset should be professionally de-identified and protected against re-identifiability. Sometimes, only a few datapoints about individuals is needed to re-identify, particularly with individuals who have access to other comparable databases against which to compare the survey-originated dataset.

Multi-language translations. With the affordances of UTF-8 character set (of Unicode), any of the languages expressed on the WWW and Internet may be expressed in web-based survey tools. Such language capabilities, further enhanced by Google Translate, Google Transliteration, Google’s YouTube’s voice-to-text auto-transcription, means that a much wider audience may be accessed. These capabilities are not without some problems: foremost, accuracy, given the polysemous nature of languages.

Automated tools that enable survey makers to translate their surveys into dozens of other languages should have native language speakers analyze the output and make corrections before anything is deployed. Also, those deploying surveys to different parts of the world should have individuals vet the imagery, the videos, and the audio, to ensure that offensive communications were not included. Unintentionally offensive communications will not only be off-putting but can mean that certain populations are not reached and voices are not heard.

Ideally, the survey design and its deployment should be the best research that can be done in that time within real world limits.


2. What are some of the basics about how to design and pilot-test an online respondent-administered survey?

A majority of the online surveys are respondent-administered surveys. In other words, these online surveys are created and sent out by email or posted as open-links on social media sites. Given that the researcher is not directly available, there are some design features that will aid in the research work:

  • Over-explain the survey purpose early on.
  • Use a forced-response after the informed consent information, to ensure that the survey respondent has read and understood the informed consent and their own rights, and offer a binary choice of “yes” or “no.” If the person selects “no,” then he or she may be sent to the end of the survey.
  • Verify that the respondent is of legal age to respond to the survey. Again, here, use a forced response of “yes” or “no.” Those who decline may be branched off to the conclusion of the survey. (The second and third bullets are usually combined into one.)
  • Include the researcher or research team’s names, affiliations, and contact information.
  • Use clear and basic language.
  • Be respectful at every phase.
  • Add customizations—such as reiterations of the respondent’s responses—to show acknowledgment (and to elicit further insights).
  • Close-ended questions should enable respondents to choose “other” and to add textual elaboration of the “other.”

Once a survey respondent has read and understood why the research is being conducted and what the benefits are, and so on, it is important to maintain the trust and the respondent’s momentum and goodwill in responding.

  • When designing the survey, it is important to think sequentially. Doing a walk-through as a respondent can give a sense of where potential confusions may arise.
  • Each section of a survey should be labeled.
  • The survey taker’s time should be valued, so questions should not be repeated except for where they are being used as a validator. (“Survey fatigue” is a real phenomenon, and this should be avoided. Survey takers should not experience frustration, unless that is a necessary state for the research. Also, the introduction of frustration will often mean survey taker drop-out or attrition, for “censored” and incomplete responses.)
  • There should be a progress bar that shows how much survey is left. If a survey will take longer than normal, then the setting should allow a save and return to complete feature.
  • The overall look-and-feel of a survey should be maintained.
  • If semantic differential scales (like the Likert) are used, the least-to-most or most-to-least sequence should be maintained throughout the survey. Consistency is important. Also, the coding scores behind the scale should be consistent and correct (for statistical analysis). If the scoring values of the multiple-choice scales were improperly set, it is still simple to just recode values before exporting the data for analysis.
  • The survey should be tested against misstatements of fact.
  • The survey should be tested against misspellings.
  • If branching is used, there should not be any dead ends for respondents. (Every function should be tested…and if a revision is made to an online survey, all the back-end scripting should be double-checked again since code is sometimes glitch, and some developers may have added in contorted illogic. Also, there can be survey designer error because of trade-offs in a system. For example, using a “prevent ballot box stuffing” toggle means that survey respondents who may be using public computers at a library—such as university students—will be blocked from responding to a survey if anyone else accessed that survey from that machine prior. Unintentional design flaws can be problematic.)
  • If a survey is to be used for longitudinal research over long stretches of time, it should be designed intelligently for long-term use. If there are revisions, the original data structures may have to be protected for easy across-time analysis (with new questions shunted off to the end of the original survey).
  • To be relevant, survey instruments will need to be revised. It helps to have some indicators to suggest when revisions are necessary and some ways to know where changes are needed.


3. What are some best practices in survey creation and deployment?

Different researchers and research teams will have their own best practices for survey creation and deployment. Simply, it is important to have a clear research design (objectives, methods, target populations, methodology, and so on). It helps to design a survey to have a clear sequence and trajectory. The survey should be written in simple English (or whatever base language is used). Any multimedia used should be included for a purpose and should be non-skewing (if possible). The survey should be designed to the technological affordances of the online survey system.

If branching logic is used in the survey, there should be flowcharts or function diagrams that show the various paths, and the logics that determine each juncture.

The draft survey should be pilot-tested. An alpha test involves a check of the survey through all branches to ensure all technical functions work…across browsers and devices. Online surveys should be security tested against findability by browser spiders…participation by robots (and / or fraudsters who would engage in “ballot box stuffing”)…and so on. There should also be tests against data leakage. This test may also involve checks for legality (see above) and accessibility. (Proper design, though, means that early lapses in legality and accessibility are addressed in the design and development process. Such rookie mistakes should not be made, and such problems should be caught sufficiently early in the process.)

Also, especially for large-scale surveys, there should be a test of the data that comes from the survey. (If this is not done, it is possible to capture data in an inefficient way which will make human processing and analysis of the data much more expensive and frustrating. For example, if a question can be asked with a list of known possible answers, that is always preferable than making an open-ended question because of the problems of noise in human-provided data.) There are automated tests that create synthetic data to enable such testing, but in such cases, there should be a large number of such records made to explore every eventuality in a complex survey, and this should be bolstered by human testing of every possible branch. Later, when that survey has been revised and is ready to be administered (distributed), it is important to delete all faux data, so that is not included with the actual survey responses. All faux data responses should also be decremented, so quotas will not be clogged with synthetic data. (Real data should not ever be contaminated with test data.)

The alpha test is usually conducted by the in-house development team. A beta test includes some members of the “public,” with individuals who represent the target demographics of interest. Any issues identified in these processes should be addressed.

Beyond the functionality of the survey instrument for the local research purpose, it is important to review standards for such instruments in terms of reliability (same results across time, ceteris paribus) and validity (construct validity)…and item analysis for construct, convergent, and discriminant validity. If the instrument is supposed to be predictive, there should be ways to validate or invalidate that. (A good source for this is E. Ruel, W.E. Wagner, III, and BJ. Gillespie’s “The Practice of Survey Research: Theory and Applications.”)


4. What are some of the technological affordances of modern online survey systems?

To simplify, contemporary online survey systems have a few major new affordances. They include the following:


Web affordances:

  • Easy integration of multimedia in inline frames and directly (built-in video players)
  • UTF-8 and multilingual expression
  • Google Translate integration


Scripting and programming:

  • using light programming to customize the survey experience (through acknowledging respondents’ responses, and others)
  • using conditionals to set branching logic and in-system actions (including email triggers and contact list populating triggers)
  • using display logic to enable contingency questions (questions which appear only if certain conditions are met)
  • setting quotas to automate particular behaviors on achievement of quota
  • using embedded data to capture data or to apply logics involving math or to apply logics involving dates
  • using conditions to set branches (through branching logic)
  • using geographical information (whether from IP addresses, area codes, zip codes, city/states, or other indicators) for conditionals
  • the setting of default answers to enrich the setup of questions
  • the uses of authenticator contact lists against which respondents need to verify identities before proceeding
  • the uses of CAPTCHA to verify humanity (and non-bothood)

There are some programs that may be deployed in online surveys that enable accessing third-party APIs, such as for Google Maps. All online survey systems limit executable code on their sites to protect users and data, but some light code may be run to capture data from a third-party service.


Built-in features:

  • auto testing a designed survey through the generation of synthetic data
  • auto accessibility-testing to ensure the proper accommodations for accessibility (some accessibility standards, not all)
  • security protections against manipulation
  • protection of uploaded files against hacking


Data analytics:

  • easy data exports (at macro and micro levels)
  • cross tab analysis
  • text analysis
  • application programming interfaces (APIs) and generated tokens for enabling direct data download to qualitative data analytics tools

and others


5. What are some in-tool data analytics features for data from online survey research?

It is well beyond the purview of a summary piece to include information on how to analyze survey data. The practices of particular fields and domains inform how survey data may be used. In general, close-ended questions are analyzed quantitatively, and open-ended questions (for text or file upload feedback) are analyzed based on various forms of content analysis, theme and sub-theme extraction, sentiment analysis, and so on. For all online survey data, there are computational analysis tools and methods.

For online survey suites, there are built-in tools that enable analysis. These may include light statistical analysis, data visualizations (frequency counts), cross-tab analysis, and text analysis.

In many cases, the best bet is to explore the data initially online but then to export the raw data and analyze in other tools especially designed for different types of analytics (including machine learning analysis techniques). (It is important to make sure that all the collected data is fully exported, especially with larger datasets. Also, pristine master files of data should be archived and protected before any data cleaning or processing, to protect against researcher or data analyst mistakes in data handling.)

Examples

(Forthcoming.)

How To

To summarize very briefly:

1. Learn the domain field well. 2. Learn the rules for online survey research in that domain. 3. Learn the web-based survey system and its various functionalities. 4. Design a survey tool thoroughly for the studied phenomenon. (If you're going to err, do so on the side of rigor.) 5. Go through IRB oversight. Make the proper changes. 6. Alpha test. 7. Beta test. 8. Revise the survey tool. 9. Deploy. 10. Conduct full analysis of the captured data. 11. Improve the survey instrument. 12. Share.

Depending on the local context, the above may / may not be relevant.

Possible Pitfalls

Some common pitfalls have been mentioned above, but this list is certainly not comprehensive. There are many ways to do something right, and many ways to do something wrong. The idea is to anticipate potential problems and head those off. It is wise to access intelligent and professional and ethical counsel in order to use survey research correctly.

Module Post-Test

1. What are some quality standards for online survey research?

2. What are some of the basics about how to design and pilot-test an online respondent-administered survey?

3. What are some best practices in survey creation and deployment?

4. What are some of the technological affordances of modern online survey systems?

5. What are some in-tool data analytics features for data from online survey research?

References

Ruel, E., Wagner III, W.E., & Gillespie, B.J. (2016). The Practice of Survey Research: Theory and Applications. Los Angeles: SAGE.

Extra Resources

Ruel, E., Wagner III, W.E., & Gillespie, B.J. (2016). The Practice of Survey Research: Theory and Applications. Los Angeles: SAGE.