Advanced Qualtrics

From E-Learning Faculty Modules


Contents

Module Summary

At K-State, the survey tool available to campus researchers is powered by the Qualtrics Research Suite. This tool is used not only for research surveys but also to support the provision of services, to register individuals for events, and to deliver mass-scale trainings (because of the application programming interfaces or APIs which enable the broadscale capturing of user performance data for information systems). This tool is a powerful one which enables access to various online affordances, such as the inclusion of multimedia, survey customizations (enabled by piped text and branching logic, among other features), the easy automated extraction of data (such as through panels), and other features. As such, it is helpful to be aware of some of these features for the design of surveys, the downloading of data, and the uses of some features to support data analytics (in particular, cross-tabulation analyses).


Image:KStateSurveyScreenshot.jpg

Takeaways

Learners will...

  • Review some basics of creating an accessible online survey
  • Review the various question types available in Qualtrics
  • Review advanced features of setting up a survey in Qualtrics, including setting default answers; using branching logic; using piped text for customizations; setting triggers for automated survey responses; adding display logic to questions; setting up a loop and merge feature; setting up a “carry forward” feature; setting quotas; using the Google Translate feature; customizing the conclusion of a survey, and protection a survey against various types of online manipulation
  • Learn how to set up panels
  • Analyze their surveys using item analysis
  • Learn how to extract data (in an analyzable way) from surveys in Qualtrics and to apply advanced in-tool data analytics to the data
  • Learn how to archive a survey and related data offline in a way that is reconstitutable in Qualtrics


Module Pretest

1. What are some ways to create an “accessible” online survey? What is a feature in Qualtrics that may help in this accessibility aspect?

2. What are the various question types available in Qualtrics? (These include the following: Text/Graphic Questions; Multiple-Choice Questions; Matrix Table Questions; Text Entry Questions; Slider Questions; Rank Order Questions; Side by Side Questions; Constant Sum Questions; Pick, Group and Rank Questions; Hot Spot Questions; Heat Map Questions; Gap Analysis Questions; Drill Down Questions; Net Promoter Score (NSP) / Customer Loyalty Questions; and two types of Hidden Questions (Timing and “Meta Info Questions”); CAPTCHA verification; and File Upload Questions

3. What are some advanced features available in setting up a survey in Qualtrics? (These include the following: setting default answers; using branching logic; using piped text for customizations; setting triggers for automated survey responses; adding display logic to questions; setting up a loop and merge feature; setting up a “carry forward” feature; setting quotas; using the Google Translate feature; customizing the conclusion of a survey, and protection a survey against various types of online manipulation.

4. What are panels, and how are these used in the Qualtrics Research Suite?

5. What is a survey analysis feature (including an item analysis one) in Qualtrics?

6. What are some tips on how to extract data (in an analyzable way) from surveys in Qualtrics? What are some in-tool advanced data analytics in Qualtrics?

7. How does one archive a survey and its related data offline in a way that is reconstitutable in Qualtrics?


Main Contents

This main area is organized based on the seven pre-test questions. Given the complexity of the topic, note that some of this is located on Qualtrics and also in a downloadable handout (pdf format).


1. What are some ways to create an “accessible” online survey? What is a feature in Qualtrics that may help in this accessibility aspect?

  • File contents in digital files and websites should be keyboard accessible for access and navigation. They should not require mouse actions, for example.
  • Use course file types in universal product formats.
  • Ensure that digital files are human accessible and machine-readable.
  • Properly name digital documents. Structure text documents.
  • Use clear, simple English.
  • Label informational graphics with “alt text.”
  • Transcribe and label audio and video.
  • Make accessible PowerPoint™ slideshows, if these are used in the survey.
  • Use color in an accessible (high contrast) way. Use labels in addition to color.
  • Summarize and label data tables. Do not use HTML tables for anything other than data.
  • Support user control of automations and sequenced actions, as much as possible. Allow use control of time. Use time limitations reasonably and possibly even sparingly.
  • Use accessible-format questions in Qualtrics.


2. What are the various question types available in Qualtrics? (These include the following: Text/Graphic Questions; Multiple-Choice Questions; Matrix Table Questions; Text Entry Questions; Slider Questions; Rank Order Questions; Side by Side Questions; Constant Sum Questions; Pick, Group and Rank Questions; Hot Spot Questions; Heat Map Questions; Gap Analysis Questions; Drill Down Questions; Net Promoter Score (NSP) / Customer Loyalty Questions; and two types of Hidden Questions (Timing and “Meta Info Questions”); CAPTCHA verification; and File Upload Questions


To experience the respective questions, please view the following training survey through a browser:

Demo “Training” Survey: https://kstate.qualtrics.com/SE/?SID=SV_5gTuBG8ZWa94tMx


3. What are some advanced features available in setting up a survey in Qualtrics? (These include the following: setting default answers; using branching logic; using piped text for customizations; setting triggers for automated survey responses; adding display logic to questions; setting up a loop and merge feature; setting up a “carry forward” feature; setting quotas; using the Google Translate feature; customizing the conclusion of a survey, and protection a survey against various types of online manipulation.

Advanced features are summarized in the two following sources:

Training Handout: http://www.k-state.edu/ID/ExploringAdvancedQualtricsHandout.pdf

Demo “Training” Survey: https://kstate.qualtrics.com/SE/?SID=SV_5gTuBG8ZWa94tMx


4. What are panels, and how are these used in the Qualtrics Research Suite?


Panels are collections of individuals' names and emails. They may be captured from any number of sources and placed in datasets and used in Qualtrics. They may be captured on-the-fly based on responses to surveys and even particular responses to survey questions or certain technologies used...to "thin-slice" the individuals in a population.


5. What is a survey analysis feature (including an item analysis one) in Qualtrics?


To understand how respondents interacted with the survey—such as what times of day they accessed the survey and how long they took to interact with the survey, a researcher may go to the Reporting Tab and view Survey Statistics.

To extract data about the survey (and the item analysis):

Reporting (tab) -> Survey Statistics


6. What are some tips on how to extract data (in an analyzable way) from surveys in Qualtrics? What are some in-tool advanced data analytics in Qualtrics?


When extracting data, it is important to ensure that the data is analyzable in software tools outside of Qualtrics. To do so, it is important to first output a report…and then access the data in .xl / .xlsx / .csv format. One important point is to fully expand tables to capture the full data (if there are more than 100 responses for a particular question).

Downloaded data should be in properly named tables (often linked to the particular question number).

Oftentimes, data for complex questions will have to be downloaded en masse and then broken apart into different pieces (separate datasets) for analysis.

Quantitative or numeric data should be clearly labeled and structured for analytics in SPSS, SAS, or other data analytics tools.

Text data should be left in .csv format because of the affordances of sorting and organizing in Excel. These may be transcoded into straight .txt in order to be used in a Word file for other processing…in Word, in NVivo 11 Plus, and other data analytics tools.

Spatial data may be extracted for processing in Google Maps and ArcGIS.

Thinking about Survey Re-use

During the extraction and analysis of survey data, researchers would do well to note what corrections may be made on the survey itself to enhance the data extraction and analysis if that survey may be re-used at a later date (or if the survey itself will be part of a shared instrument for other researchers).

  • For example, open-source questions should be linked to the prior multiple choice ones instead of being built as a stand-alone question (or else the “Survey Statistics” will show very low participation for many text-based follow-on questions).
  • If respondents are asked to indicate locations, it will help to offer a full list of possible entries instead of an open text field because people often have very poor spelling. The use of a text field means that data analyst type is spent trying to clarify what people meant.
  • If people are asked to respond to close-ended questions, it may help to have a catch-all open text response for those who do not see their preferred response in the closed set.
  • If enumerators are used to capture data from respondents who are non-literate and non-numeric, they should be trained to phrase responses based on what the respondents say instead of using their own phrasing for a range of insights.

Again, it is important to pilot test surveys so that they are flexible and non-leading in their design.


7. How does one archive a survey and its related data offline in a way that is re-constitutable in Qualtrics?


To save a survey offline (in .qsf or "Qualtrics Survey Format" format) and the related data (in .csv format), please do the following:

To download the survey: Open the survey -> Advanced Options (button) dropdown -> Export Survey (.qsf file) -> Select location for the save

To download the related data: View Results (tab) -> Download Data -> Set parameters -> Click .csv format (.csv file) -> Select location for the save

To reconstitute the survey and data (import responses) from offline sources: My Surveys (tab) -> Create Survey -> Quick Survey Builder -> Advanced Options -> Import Survey -> Choose File -> Open -> Import

(OR)

View Results (tab) -> Responses (button) -> Advanced Options -> Import Responses -> Browse -> Verify Fields (Show or “Ignore”) -> Import

  • Incomplete data (for example, a non-response) is not re-imported.


File Formats: “.qsf” stands for “Qualtrics survey format”. “.csv” stands for “comma-separated values” format.

Examples

Please review the sample surveys in the built-in Qualtrics libraries of surveys and survey templates.

How To

The "how to" depends on way too many variables than can be addressed here.

Possible Pitfalls

This section about “pitfalls” is not about the Qualtrics Research Suite tool per se.

One pitfall relates to the use of this article. First, do note that this information is not comprehensive, and it will likely date out in time since features are constantly changing. Also, this information is presented with a certain point-of-view and may be limited by that. Users may want to access Qualtrics documentation directly (a link is below). Qualtrics support personnel are highly skilled and responsive, and they can provide insights for unique design needs. (Please read their documentation first before contacting their Help personnel though.)

Another pitfall relates to the work of setting up and deploying surveys for research (and other applied uses). It is critical to think through a survey design from the beginning all the way through the data analytics side and to pilot-test the survey (and then to document these steps, for the Methodologies section of publications and presentations). The reason why such due diligence is necessary is that a poorly designed survey means poor data and poor publications. It also may mean a lot of hard work during data analysis to de-noise data.

The last pitfall here relates to legal issues related to surveys. One involves the need for all researchers who use this tool to have gone through recent institutional review board (IRB) training. Surveys are understood to relate to human research and therefore generally needs to go through human subjects review. Oftentimes, surveys are exempted from oversight, but it is important to offer informed consent to respondents and other features in order to be legal. Having an IRB committee look over the research method and the survey design may be insightful for the researcher, and it may provide some legal cover if there are questions sometime down-the-line. Part of this review involves looking at whether the participants of the research stand to gain from the work, broadly speaking. A researcher cannot offer outsized inducements for participation. There should be ways to contact researchers if there are concerns. If research changes over time, often, research has to undergo further IRB oversight and approval, particularly if the research change is substantial.

How researchers handle the data is another aspect of legal responsibility. Another aspect of legality involves ensuring that the survey itself does not contravene various laws. A survey may not infringe on others’ intellectual property, including copyright. That means that any included multimedia may only be via embed text (not an uploaded copyrighted video); all images should be copyright-free (or owned by the individual offering the copyright).

Another aspect of legality involves protection of privacy. Individuals’ names and likenesses should not be used in a survey without their explicit and documented / recorded permission. If respondents’ identities are captured in the survey, it should be with their explicit permission. That information should only be used within the limits of the approved research. (Generally, information that is collected is used; otherwise, that data should not be collected in the first place.)

Accessibility (Section 508) is another consideration. If multimedia is used, there should be a text equivalent of that data (such as alt-texting, captioning, and so on). Every aspect of a survey should be screen-readable. A survey should be able to be interacted with using only a keyboard (not a mouse) because of mobility challenges. Color should not be used as the only way some information is delivered. Data tables should be properly described. There are a range of other considerations as well. Some built-in questions in Qualtrics are by nature inaccessible, and those are identified by the “Check Survey Accessibility” tool in Qualtrics.

Module Post-Test

1. What are some ways to create an “accessible” online survey? What is a feature in Qualtrics that may help in this accessibility aspect?

2. What are the various question types available in Qualtrics? (These include the following: Text/Graphic Questions; Multiple-Choice Questions; Matrix Table Questions; Text Entry Questions; Slider Questions; Rank Order Questions; Side by Side Questions; Constant Sum Questions; Pick, Group and Rank Questions; Hot Spot Questions; Heat Map Questions; Gap Analysis Questions; Drill Down Questions; Net Promoter Score (NSP) / Customer Loyalty Questions; and two types of Hidden Questions (Timing and “Meta Info Questions”); CAPTCHA verification; and File Upload Questions

3. What are some advanced features available in setting up a survey in Qualtrics? (These include the following: setting default answers; using branching logic; using piped text for customizations; setting triggers for automated survey responses; adding display logic to questions; setting up a loop and merge feature; setting up a “carry forward” feature; setting quotas; using the Google Translate feature; customizing the conclusion of a survey, and protection a survey against various types of online manipulation.

4. What are panels, and how are these used in the Qualtrics Research Suite?

5. What is a survey analysis feature (including an item analysis one) in Qualtrics?

6. What are some tips on how to extract data (in an analyzable way) from surveys in Qualtrics? What are some in-tool advanced data analytics in Qualtrics?

7. How does one archive a survey and its related data offline in a way that is reconstitutable in Qualtrics?


References

Extra Resources

Qualtrics FAQ Overview: http://www.qualtrics.com/university/researchsuite/research-resources/other-resources/faqs/

Demo Survey: https://kstate.qualtrics.com/SE/?SID=SV_5gTuBG8ZWa94tMx

Training Handout: http://www.k-state.edu/ID/ExploringAdvancedQualtricsHandout.pdf

Using Qualtrics for Online Trainings: http://www.slideshare.net/ShalinHaiJew/using-qualtrics-for-online-trainings