Tijdschrift voor Taalbeheersing - Volume 38, Issue 2, 2016

Research integrity in quantitative, empirical studies: problems and potential solutions

In the social sciences, there is an important debate about how to do good empirical research and how to publish about it. As this journal, the Tijdschrift voor Taalbeheersing, also regularly publishes empirical studies, the present article aims to open the debate in the field of language and communication. After outlining the major problems associated with research integrity and developing potential solutions to these problems, we invited researchers from the field of language and communication to present their views, which are published as reactions to this paper in the current issue of this journal.

Integrity in research: hunger for perfection and production

Solutions for observed integrity problems in scientific practice often involve reorganizations of the publication system, but systemic solutions do not warrant changes in culture. Perfect form does not guarantee interesting content. Next to looking at the system, academics should also look at and reevaluate their own criteria for publishing a paper, or hiring new staff. Hunger for perfection and production may affect scientific integrity.

Empirically sound conclusions: A guide for language researchers

Recently, there has been (and still is) a lot of discussion about the replicability of scientific findings. Hornikx and Batenburg (2016) try to explain the non-replicability in science, and discuss the situation for language research. We make the point that empirically sound conclusions will contribute to the replicability of science. In this commentary, we discuss several practices that will help the empirical language researcher to draw more solid conclusions from their data.

Conduct matters: On rules for scientific integrity1

Discussions of scientific integrity of research in discourse studies (taalbeheersing) should not be limited to experimental research only, for rules for scientific conduct apply to all sorts of research: experimental and observational, quantitative and qualitative, empirical and theoretical. Nowadays, written codes are available that specify simple and clear principles of honesty, carefulness, reliability, verifiability, impartiality, independence, and responsibility. In response to breaches of these principles, Hornikx and Batenburg (2016) make a plea for new rules and practices that pertain to the macro and the micro level of the scientific ecosystem. Their proposals contain further specifications of available rules for what is logically and socially acceptable and thereby suggest that the present codes do not suffice and that new and stricter rules are an appropriate and effective response to transgressions. It might be wise to consider another question first: Do scientists in our discipline take the existing rules sufficiently serious? My answer, derived from my own experience and environment in research and teaching, is negative. Therefore, rather than new rules, we need more interaction with students, PhD’s and colleagues on matters of integrity.

More messages, of course – but how many and what kind of messages?

It is solid advice to employ more messages in one’s experiments when conducting research on message variables’ effects. The question is how many messages should be used? And what characteristics should these messages have? Compared to the insights and guidelines available for the selection mode and number of participants when aiming to generalize to a population, such insights and guidelines are sorely lacking when aiming to generalize over messages.

Sustainable Research

To increase the effectiveness of communication research in terms of theoretical and practical yield, I propose a three-pronged approach: 1) Reduce the temptation to make less ethical decisions by changing publication policies; 2) Increase the power of the statistical tests, also by maximizing effect size; 3) Focus the research effort on the domain of sustainability.

Research quality: dichotomous versus meta-analytical thinking

The success of statistical reform as a means of improving the quality of quantitative research requires that individual researchers change their interpretation of the results of statistical analyses. For instance, if researchers do not change their habits of dichotomously interpreting p-values, publishing studies regardless of statistical significance may lead to incorrect interpretations of the empirical evidence across studies. Such a publication policy would be beneficial, however, if researchers think meta-analytically and focus on the estimation of effect sizes and confidence intervals.

False positives: No lack of integrity, but an excess of flexibility

Hornikx and Batenburg discuss the important problem of false positives in the literature. One of the causes for false positives they give is a faulty incentive system that stimulates unethical behavior. In this reply I argue that false positives are mainly caused by human error, misunderstanding of statistics, and underpowered studies. I suggest a small adaptation to the solutions that were offered.

Scarcity is good, but selection must improve

Hornikx and Batenburg (2016) demonstrate that publication pressure is an important reason why the social sciences face a problem with replicability. They argue that researchers might use questionable methods because the opportunities to publish are scarce. This paper, however, suggests that more room for publications will not lead to better research practices. Instead the scientific community should focus on stimulating good methods: (i) use power analyses to make sure an experiment is adequately sensitive and (ii) value the effect size over statistical significance.

How transparent can we be?

In their contribution Hornikx and Batenburg (2016) raise a number of issues related to the transparency and integrity of quantitative research in particular. In this contribution two additional issues are raised that may hinder the accessibility and accountability of our research results. One is related to studies based on the use of confidential materials – especially in the context of qualitative studies – and the other to the implications of the new ‘meldplicht datalekken’ (the duty to report data leaks) that has been introduced in the Netherlands per January 1, 2016, as part of the Personal Data Protection Act. Together they demonstrate the limits of our strive for scientific transparency.

Research integrity in quantitative, empirical studies: the perspective of researchers in language and communication

The article by Hornikx and Batenburg (2016) invited researchers in language and communication to reflect on problems and possible solutions related to research integrity in conducting and reporting quantitative, empirical research. The reactions from colleagues in the field underline the problems that were described, and present a number of solutions to these problems. In general, these solutions can be summarized under the heading of meta-analytic thinking: one study does not provide a yes-or-no answer to a given answer; it should be designed to provide more robust results (e.g., through replications, multiple message designs), and should be reported in such a way as to be replicable in the research community (e.g., transparency, registration reports).