 |
| Academic journals [Source: http://www.wun.ac.uk/wun/research/view/world-class-universities] |
1) Hypothesis preregistration. Currently, many academic studies gather large datasets and then search for trends/patterns within that dataset, without first stating their hypotheses. Such a post-hoc analysis, also called 'fishing', can lead to a Texas Sharpshooter falacy, which often arises when researchers have a large amount of data at their disposal, but only focus on a small subset of that data. The fallacy is characterized by a lack of a specific hypothesis prior to the gathering of data, or the formulation of a hypothesis only after data have already been gathered and examined. The fallacy occurs when the same data is used to construct and test the same hypothesis (hypotheses suggested by the data).
Suggested solution: Researchers pre-register their hypothesis, research designs, and analysis plans with a journal and publish them in advance, for example with the Open Science Foundation. Such a pre-registration allows for stronger explanatory and confirmatory research. An example of a journal that already practices hypothesis registration is Learning at Scale.
2) Many academic publications only publish the summaries of their data. This does not allow other researcher to verify the data analysis or re-use the dataset for further studies.
Suggested solution: Open data approaches allow researchers, whenever possible, feasible, and ethical, to make their dataset available for inspection and additional analysis. This also allows replication studies and an in-depth inspection by reviewers.
3) Reviewers and editors of academic journals often spend a lot of time reviewing submitted manuscripts. The academic publishing process is considered a 'dialogue' between the authors, the editor, and the reviewers. However, this dialogue happens usually behind closed doors (masked reviews) and the often substantial amount of work by the reviewers does not receive any official recognition.
Suggestion solution: Open reviews publish the peer review comments alongside the article. This gives the reviewers credit for their work and enables readers to judge for themselves how well the paper responded to the critique of peers. It could even be considered that reviewers and editors receive credit for the paper (similar to the credits in a movie that list all people who contributed in some form). Some journals even list reviewer reports with a separate DOI. Open review can also reveal the identity of authors and reviewers (rather than a single blind or double blind review). Open review could encourage a more authentic academic dialogue. Open reviews with a publication of peer review reports can contribute to greater transparency (see some examples here).