Experimental studies tease out effects based on high control of participation. While participants use tools developed by researchers (allowing high control), the control is limited in computer-mediated tools. Researchers cannot examine what subjects actually do during the study. For example, a research subject might not properly familiarise themselves to a media vignette in framing studies, thus making the research setup invalid, or might even answer questions incorrectly. (However, the problem is not unique to research via interactive systems only. The same thing could happen with a traditional survey in laboratory settings.) As crowdwork platforms are aimed to work on tasks that require human intelligence to provide the right solution, many algorithms and other solutions to measure the quality of responses have been proposed (e.g. Li et al., 2017, Section Quality control). However, these approaches assume that there is a correct answer against which we can compare other responses. In experiments this is rarely the case; rather, we seek to evaluate differences in answers. To build a baseline, researchers often use attention checks, questions where participants are polled about content or asked to enter a specific response to check that they are focusing on the task. Alternatively, digital platforms can be developed to collect data on the response process, which is also known as paradata. We can measure how much time was spent on a page of the survey or if a particular browser window was kept open. Further details could include cursor location when the survey was responded to (e.g. Horwitz et al., 2016), all pressed keys or even contextual data such as camera photos (web cameras are nowadays embedded in almost all laptops and desktops) or the physical location of the device. These can help to understand responders' behaviour and even invalidate some responses based on their performance, thus expanding beyond what researchers could do with a paper-and-pen survey.
Where interactive systems may provide additional leverage is deception. Social scientists have often used deception in their experimental studies. Classics include Milgram (1963) on obedience and Asch (1951) on social conformity. Both used paid actors who presented themselves as other volunteer participants but in fact followed a script defined by the researchers. As people interact primarily with systems, they are not aware of other actors - as the saying goes: `on the Internet, nobody knows you're a dog'. For example, running a replication of Asch (1951) could be an experimental room where a subject sees eight other bots (casted as other subjects in the experiment), claiming that two lines are of equal length before answering the question about if the lines are of equal length. (As the online environment might have a different degree of social pressure, the results might not replicate.) Furthermore, as researchers control the environment where interaction occurs, they could include deception. For example, instead of (real) download numbers, Salganik et al. (2006) could have manufactured numbers to further identify what kinds of differences in download numbers lead to popularity and trends.
These deceptions play out on strengths of digital artefacts in data collection. Experimental participants only have access to information we provide to them, at least in principle. For example, Munger (2017) used different fake user profiles to explore how race and perceived social status impact reactions on anti-racist messages in Twitter. In practice, for an experimental subject it seemed that a white or black user with a high or low number of followers commented on their use of racist slurs (there were four different experimental conditions plus one control condition where no intervention was available). The deception teased out relevant theoretical effects through an experimental design, showing how even an established platform can be used in innovative ways through technical and sociological imagination. Digital tools allow precipice approaches to manipulate users, often without their knowledge. This is partly because digital artefacts are opaque to users - the algorithms, interfaces and systems are developed by mystic companies and we just use them. (For example, you have most likely been part of several experiments already today, as companies like Google and Facebook run several experiments every day to examine the best design of their website, best algorithm to rank content or even just to decide what is the best of 41 different shades of blue to use in the toolbar - an experiment run at Google. Do you know if you have participated in any experiments via online platforms?) John (2019) examined Facebook's service showing the number of friendship connections between conflict areas, such as Israelis and Palestinians. His conclusions were that the displayed numbers were âbullshitâ (his words) but were used by several researchers to illustrate the impact Facebook has on society. This case illustrates how difficult it is to not trust digital artefacts. Experimental deceptions may benefit from this kind of trust, creating an immersion where users correctly believe the system.
The effect of deception can be further enhanced through users' personal data, available to some degree through online services. For example, it is possible to create a Twitter feed that includes content from accounts you follow, collected through Twitter's API. Some of the content could be manipulated to create experimental conditions, creating an experimental study where participants see some parts of the reality. To some degree, this is not novel and could have been done previously with extra efforts. For example, Gerber et al. (2008) pulled people's voting record, a public data set in the United States, and used that when designing a letter to send to experimental participants. With digital artefacts, the benefit is to access much more personal and user-generated content - and people are not only willing to do such work but share such data often with startups and other commercial entities.
Our discussion on control and deception opportunities is part of the experimental design or planning the experiments so that they end up correctly studying what researchers wanted to study. Interactive artefacts as research instruments do not revolutionise the execution of experimental research. They might make it cheaper and more accessible through online platforms, allow running more experiments outside the laboratory settings and thus increase ecological validity; might allow stronger standardisation of the participation process through virtual reality or robots; or might allow immersive deceptions. However, the core questions of samples and control do not vanish; rather, they need reconsideration. Some opportunities to mitigate these challenges may come from collecting paradata during an experimental process. Furthermore, increasing sample sizes can improve the statistical process, thus allowing scholars to make stronger statements. These opportunities and challenges in experimental design must be weighted with opportunities the study has for scholarship and research ethics.