Scholars seek to understand and limit potential mistakes and errors emerging from their analysis approaches. Quantitative social scientists use several processes to examine that their results are valid. They examine if data follow a normality assumption before using statistical tests, such as Students' t-tests, and explore if there are biases in regression models by examining residuals. The reliability of measurements, or constructs, is evaluated using other kinds of statistical tests, such as Cronbach's alpha or Cohen's kappa values. Qualitative social scientists working on content classification use intercoder reliability to identify if their classification is solid. Even more interpretative qualitative work, such as grounded theory methods, may use research teams and discussions about the interpretations and exemplars to illustrate how the conclusions emerge from the data. Similarly, computer scientists and physicists examine questions related to how good the data for making interpretations are and to understand what could cause errors in their analysis. As people other than social scientists have started to work on social data, these concerns are increasingly addressed across all researchers working on computational social sciences.
Avoiding mistakes and errors is often discussed under terms of internal validity, external validity and reliability - in both quantitative and qualitative research traditions. Discussing internal validity puts emphasis on how good the relationship is between a theoretical concept and observed phenomenon. In particular, it focuses on examining other phenomena that might be in play as well. The famous quote `correlation does not imply causation' highlights internal validity concerns: Seeing a correlation between variables does not yet mean that there is a causal relationship. In qualitative research, demonstrating data through vignettes is one way to illustrate the connection between data and theoretical ideas. Some scholars also highlight that the credibility should be checked by involving users. External validity instead highlights the essential question of generatability or transferability: to what degree were the results specific to a particular context - like time, culture or platform - and did not inform us about phenomena outside its context. For example, in psychology (and other fields) it has been observed that research subjects are often WEIRD: Western, educated, industrialised, rich and democratic. At some point, the most studied population was psychology students. However, it has been shown that results from these settings do not always hold when seeking to speak about humans outside the target population. Finally, reliability examines how well the research procedures give the same output with the same inputs. A classic example of poor reliability is a weight scale that gives out random numbers. While it is able to quantify weight, weighting the same object several times would lead to many different results. In quantitative work, reliability is an essential criterion for any survey instrument, while in qualitative work, reliability relates more to the transparency and openness of the subjectivity of the research settings. This summary does not do full justice to the debate on reliability and validity. In both research genres, extensive work has been written about these questions, even proposing alternative terms beyond validity and reliability. Nonetheless, issues in these aspects of the research â internal and external validity, reliability or some alternative more emphasised terms - might lead the results or interpretation to be incorrect. Therefore, any discussion based on these results or implications they have for our society would similarly be wrong.
All research methods have their own practices to focus on questions of internal validity, external validity and reliability. These practices are, to some degree, field specific and even change over time (e.g. , ). Therefore, this chapter does not provide a comprehensive list of potential threads to validity and reliability. Rather, we illustrate areas where validity and reliability concerns exist and discuss some ways to address these. However, as these methods and the scientific practices about them are constantly evolving, one must always be careful to understand the established practices before applying these methods.