DEALING WITH BOTS IN QUALTRICS SURVEYS

What are bots, and why do I need to worry about them?

Bots are automated scripts that can be used to complete a survey. Many were designed for legitimate purposes, such as testing a survey before launch. But, because some surveys pay people to take part, or enter them in a prize draw, some people use bots to take surveys hundreds or even thousands of times in an attempt to get these survey rewards.

Survey attempts by bots are not legitimate data, so the first problem is that you’ve got loads of extra responses and you need to work out which ones are legitimate and which are not. The info below can help you work this out. Plus, I’ve outlined some of the tools in Qualtrics that can help minimise these issues. There may be similar tools in other survey platforms.

The second problem is that, sometimes, the people who are running the bots can get really nasty if you don’t give them survey rewards. I’ll talk about this below, too.

Bot attacks seem to be increasing. In this article, I’ll talk about what to do if you’re hit by a bot, why some surveys are more at risk than others, the tools that are available in Qualtrics, and different ways that you might like to use these tools, which will depend on your needs.

What’s the difference between a bot and a duplicate response?

Sometimes people may take surveys multiple times without necessarily meaning to do so. Imagine you have a survey that’s been running for months. It’s possible someone took it in June, and completely forgot about it, taking it again in October. Or, it’s possible that people might be on a mailing list. Usually, we can track who has and hasn’t taken part, so we can target reminders to people who haven’t yet done it, but that’s not always possible. If that’s the case, then it’s possible that we’ll need to send reminders to everyone, and people might think that they haven’t taken it.

Generally, these have minimal impact. If you identify duplicates (e.g., the same IP address and similar answers, or if you collect something like email addresses, a couple of attempts from the same email address), I usually will reward them once and keep their first attempt for analysis, removing subsequent attempts. Generally, it’s fairly safe to assume they’re trying to help. If you don’t reward them at all, you’ll just turn more and more people away from taking surveys, and it’s hard enough to get people to do surveys!

A bot attack, in comparison, is when one or more bots have completed dozens, hundreds or even thousands of attempts. All of these attempts may or may not contain the same answers to each question. The main point is that they’re automated responses. The bots not reading the questions, they’re not giving you good data, they’re just trying to rip you off.

So what do you do if you're hit by a bot? Some of the tools below will head off duplicates and bots, while some will just work on bots.

What do I do if I’m hit by a bot?

The first thing is to work out if you’re dealing with duplicates or bots, or both. A few duplicates will happen in every survey, and you should consider this during data cleaning (page coming soon).

If you’ve got the following protections in place, it’ll be a lot easier to deal with bots. But, if you haven’t got any protections in place, then it could take a bit more work to identify these responses, depending how sophisticated the bot is.

Here are some things you an do:

Don’t panic! It happens. You haven’t done anything wrong.
First, see if you can work out which of your survey responses are legitimate attempts, which are innocent duplicates, and which are bot attacks. Get an idea for the scale of the problem. You don’t need to work out every single attempt at this point. What you’re trying to work out is how many of your responses are real, and how many might be bots. The reason for this is because, usually, you’re aiming for a particular number of “good completes”, so you’ll want to know if you’re close to that or not. Let’s say you’re aiming for 1000 good survey responses, and overnight your number of responses jumps from 300 to 3,000. If you’ve been hit by a bot, you may still only be near 300 good completes.
If you’re experienced with surveys, you likely already know what to do. But, if you’re relatively new to them, or don’t do them that often, talk to someone with more survey experience.

WORK OUT IF THEY’RE LEGIT ATTEMPTS OR NOT

REPORT TO ETHICS OR OTHER BODY IF NEEDED

BE PREPARED FOR POSSIBLE ABUSE, AND HEAD IT OFF IF POSSIBLE

CONSIDER KEEPING STUDENTS’ CONTACT INFO OFF SURVEYS, E.G., ON THE INFORMATION SHEET

Which surveys are most at risk?

Generally, if you’re offering to pay each participant (say with a voucher or a small cash incentive), or if you’re offering a prize draw, you’ve got a pretty good chance of being hit by a bot. This is especially the case for surveys that are advertised on social media, like Twitter, Facebook, Reddit, etc.

If you’re not paying people to take part, then you’ve got less chance of being hit by a bot because the person running the bot doesn’t have much to gain. But, it can still happen because it’s really inexpensive for them to run the bot through as many surveys as possible, so they may not be particularly choosy about it.

This is pretty annoying because these are common ways to get people to take part in surveys! I’m definitely NOT saying to avoid doing these things. But I AM saying that if you are recruiting people on social media platforms, you’ll want to bring in some of the protections below.

Bot detection in Qualtrics

Qualtrics calls this “Fraud detection”, and you can see their page on this topic here: https://www.qualtrics.com/support/survey-platform/survey-module/survey-checker/fraud-detection/ . Don’t worry, I’ll run through it with you too.

Please note that Fraud detection may not be available on all Qualtrics licenses. You may need to pay more to access at least some of these features.

Some people have worked out that you can create a “bot”, or an automated script, to take surveys many times in the hope of getting money for taking part. We might see hundreds or even thousands of automated bot attempts.

If it’s your survey that’s hit by bots, it can be a pretty unpleasant experience. Not only will you have to work out which