There are several threads to the validity and reliability that emerge from computational approaches. Challenges rise from what a researcher does to what a computer does and includes not only the existing researcher but also a network of other researchers and libraries and tools they have developed. A software bug could lead to misconnection between a theoretical concept and its empirical operationalisation in code. Algorithmic black boxes can make replication more challenging or impact the overall validity of the research process. Lack of a clear research process can open opportunities for innovation but can risk reliability, especially if someone tries to replicate an ill-described process from a paper.
Furthermore, the challenges may not be in the specific methods and its execution but instead may relate to a lack of understanding contexts or problems in the data collection process, leading to a loss of external validity or even internal validity. Therefore, some of the challenges are not novel to computational methods but highlight difficulties in studying society more broadly. These kinds of issues are difficult to tackle, as they are wide and range across different settings. As the research processes are open, it may be difficult to identify best practices for particular methodologies to increase validity and reliability. Partly, these issues are because computational social science is in its early days, where mistakes and reflections have not yet occurred.
We highlight that some of the easy mitigation strategies follow traditions and practices in social sciences. We especially focus on discussing triangulation as a tool to improve validity and reliability. Triangulation may take several forms. Scholars have identified researcher triangulation (i.e. changing the person doing analysis), method triangulation (i.e. experimenting different methods to see if they lead to different outcomes), data triangulation (i.e. using different data sources) and theory triangulation (i.e. examining the phenomena using different theory lenses) (e.g. Denzin, 2007). These are extremely common in qualitative research practices, but even quantitative researchers use similar triangulation practices. Intercoder reliability checks could be seen as researcher triangulation, and often practitioners first conduct a descriptive analysis via visualisations followed by hypothesis testing. This is why triangulation practices provide a suitable way to start identifying approaches to increase validity and reliability, but translation work is required to ensure they work with computational methods. Other kinds of suitable forms of triangulation include:
[d]ifferences were found in the underlying implementations of the Microsoft .NET and Java mathematics libraries. The differences caused otherwise identical calculations to produce divergent results, even for relatively simple equations. As might be expected, cumulative use of such calculations was found to produce widely divergent results.Their experiences suggest that it may be beneficial, even while demanding in terms of time and resources, to use different libraries to understand the phenomena more broadly.
While triangulation, broadly conceived, provides several beneficial approaches to understand validity and reliability, there is a novel factor creating threats to validity and reliability that we must acknowledge. Computational social sciences highlight how code is becoming critical for research, and research is becoming software intensive. Therefore, techniques to manage your code are essential to avoid bugs in the code, which can significantly decrease the validity and reliability of the analysis. We already discussed best practices in Chapter 8. These practices, in particular automated testing and code reviews, help address code-related mistakes. Therefore, it is advisable to use these best practices to ensure validity and reliability are taken seriously.
Finally, there are increasing calls among scholars to embrace a more reflective use of computational methods in (social) sciences. van Es et al. (2018) call for tool criticism, stating that
[i]n this reflexive and critical practice, the limitations and presuppositions built into the tool and its output need to be put under scrutiny, as well as usersâ interactions with the tool. [- -] the tools themselves are non-neutral, and afford particular kinds of use, and how output, such as visualisation, is always already imbued with particular conventions, and manipulations.Similarly, Kennedy et al. (2014) comment that
different combinations of researchers, tools, partners and contexts might produce different results [- -] We suggest that a tension between exploring the potential and recognising problems is also integral to working critically with digital methods, to efforts to simultaneously use and critique them. The use of digital methods is often motivated by a desire to produce results, so digital methods themselves produce the expectation that data will be found, that results will be produced, and that actions might be taken. Acting to realise these expectations can sometimes get in the way of thinking critically about them.Authors state that we must engage in critical scholarship about the methods we are using and ask questions related to why and how computational methods are used.
These concerns may relate to the novelty of computational methods in many disciplines of social sciences. Historically, users of novel methods have been held more accountable and asked to be more elaborative than those using established methods (Alastalo, 1947).< Algorithmic data analysis or large-scale network analysis are still novel methods for many social scientists. Similarly, the use of digital trace data or other forms of big data are emerging in social sciences. What these calls for critical reflection highlight for me is that we should not become fascinated by the methods and only their opportunities. Rather, it is important to remember that computational methods are complementing our toolbox for social sciences. There are cases where it is wise to use computational methods for analysis, but similarly there can be cases where drawbacks are larger than advantages. An aspect that we have not touched extensively in other chapters is that these methods can only capture what is in the data. Because of this, some social scientists might see these methods as following a positivist paradigm of science. That is, it applies quantitative analysis and seeks to draw law-like generalisations from society. In response to these tendencies, scholars have named other approaches to study the society, identified challenges to present society using these methods and called out challenges related to classifications. This topic has gained a lot of scholarly attention and discussion (e.g. , ), highlighting how sensitive the question is for social scientists. Therefore, one area of critical reflection is to discuss what aspects of society we can study with these methods and in what cases we should use other methods to gain a richer picture. However, quite recently, (Concannon et al., 2018) developed what they refer to as a feminist perspective to conduct computational text data analysis. Their case shows that these methods may be used together with critical scholarship, like feminism, when correctly using frameworks from critical scholarship as well.
Finally, it is not always clear why computational methods are used in academic research. First, we must acknowledge that academic research is part of the wider society. In the late 2010s, and potentially early 2020s, there have been increasing calls to develop data-driven approaches in businesses and governments. Big data and artificial intelligence - meaning various things - have been integrated in common jargon in these communities. Similarly, we have witnessed the increased importance of digital platforms in society, thus making them legitimate research interests for a wider body of social scientists than previously. These trends have also been witnessed in academia. The question partly relates to the relevancy and legitimacy of academic research. For example, Savage and Burrows (2009) speak about a crisis of empirical sociology, caused by others, like commercial entities, advancing the study of society using novel methods and data sources. They invite academic sociologists to move forward from traditional data sources to use non-representative materials. Furthermore, like other parties in society, academia is reacting to the increasing hype of big data and artificial intelligence by focusing more research to it. This is because computational methods and data about society enter research fields. It seems that experimental and proof-of-concept papers are published, increasing their volume. Partly, research funding bodies have allowed specific resources for computational research under different artificial intelligence, digital humanities and social sciences and computational methods initiatives. Because of factors like these, research benefiting from computational methods seems to be increasingly trendy and further driving researchers towards these methods. Therefore, critical reflection can unravel many motivations beyond scholarly rigour and pure academic interests, which shape the research agenda and methods used in academic research. As academia is fundamentally a social enterprise, this is not surprising. Similar research trends have been observed in academia way before this hype.
Understanding potential underlying motivations and factors may benefit by critically reflecting if computational methods should be used to answer a research question. Like all research methods, computational methods have limitations because of validity and reliability concerns. And as the saying goes: With great powers come great responsibility. This chapter showed several areas where it is justified to question the validity and reliability of research when computational methods are used, which invites us to reflect on if the chosen approaches are best suited for the problem. The overall goal of such reflection can be seen as a cost-benefit analysis. There are particular benefits from using computational methods, but increasingly we must acknowledge their limitations, challenges and other factors that may make them unsuitable for our research question. I believe that computational methods should be used only when benefits for scholarship are larger than the costs. Computational methods are not a one-size-fits-all solution. Rather, one should be aware of potential issues and challenges these methods bring up and articulate in the scholarship about how these issues have been addressed and identified.
Finally, the book has shown computational methods for social sciences are rapidly evolving. Therefore, we are still understanding the best practices for these methods, ensuring not only a clear connection to social science theory but also identifying how concerns related to validity and reliability can be addressed. Questions about how code and data are developed and maintained and how the use of methods is described will be essential, as those are aspects that are less present in traditional research. Naturally, the same challenge we observed with ethics is also present in questions of validity and reliability. The best practices or `cookbooksâ can evolve to each discipline separately, as each discipline has their own practices and developments. For example, when examining computer-science-emerging research about humans, () showed that the practices of using human-annotated data have diverse practices, some of them seeming rather alienated for anyone who has read about social science methods on classification. Therefore, we as a community still have a lot of work to understand how good research should look and how to achieve it when using computational methods.