Improving academic publishing through hypothesis registration, open reviews, and open data

Academic journals
[Source: http://www.wun.ac.uk/wun/research/view/world-class-universities]

There are three promising approaches for improve academic publishing:

1) Hypothesis preregistration. Currently, many academic studies gather large datasets and then search for trends/patterns within that dataset, without first stating their hypotheses. Such a post-hoc analysis, also called 'fishing', can lead to a Texas Sharpshooter falacy, which often arises when researchers have a large amount of data at their disposal, but only focus on a small subset of that data. The fallacy is characterized by a lack of a specific hypothesis prior to the gathering of data, or the formulation of a hypothesis only after data have already been gathered and examined. The fallacy occurs when the same data is used to construct and test the same hypothesis (hypotheses suggested by the data).

Suggested solution: Researchers pre-register their hypothesis, research designs, and analysis plans with a journal and publish them in advance, for example with the Open Science Foundation. Such a pre-registration allows for stronger explanatory and confirmatory research. An example of a journal that already practices hypothesis registration is Learning at Scale.

2) Many academic publications only publish the summaries of their data. This does not allow other researcher to verify the data analysis or re-use the dataset for further studies.

Suggested solution: Open data approaches allow researchers, whenever possible, feasible, and ethical, to make their dataset available for inspection and additional analysis. This also allows replication studies and an in-depth inspection by reviewers.

3) Reviewers and editors of academic journals often spend a lot of time reviewing submitted manuscripts. The academic publishing process is considered a 'dialogue' between the authors, the editor, and the reviewers. However, this dialogue happens usually behind closed doors (masked reviews) and the often substantial amount of work by the reviewers does not receive any official recognition.

Suggestion solution: Open reviews publish the peer review comments alongside the article. This gives the reviewers credit for their work and enables readers to judge for themselves how well the paper responded to the critique of peers. It could even be considered that reviewers and editors receive credit for the paper (similar to the credits in a movie that list all people who contributed in some form). Some journals even list reviewer reports with a separate DOI. Open review can also reveal the identity of authors and reviewers (rather than a single blind or double blind review). Open review could encourage a more authentic academic dialogue. Open reviews with a publication of peer review reports can contribute to greater transparency (see some examples here).

Order of academic authorship (and how to avoid conflicts)

Photo by Shironosov/ iStock

Academic papers are often co-authored by several authors, from two to several hundred (for example in CERN physics studies).

Rules for the order of multiple authors vary by discipline, but generally there are three possibilities:

• By degree of contribution: Authors are listed in descending order of contribution. The principal investigator is often placed last in the author list. However, some universities want to change this practice by only having the principal investigator listed if he/she actively contributed to that particular paper.
• Alphabetically: Authors are listed in alphabetical order (by family name).
• Random: Authors are listed in randomized order (rarely used).

In theory, deciding on an order of authors should be a straightforward process. However, disputes over the order of authors can ensue. Ilakovac (2007) in the Canadian Medical Association Journal reported that over two-thirds of 919 surveyed authors disagreed with their coauthors regarding contributions of each author.

There are several suggestions how to avoid authorship conflicts (compiled from source #1 and source #2):

1) Who counts as an author? Only people who made substantial, direct, intellectual contributions should be listed as authors. All authors should meet the following three criteria, and all those who meet the criteria should be authors:
a. Scholarship: Contribute significantly to the conception, design, execution, and/or analysis and interpretation of data.
b. Authorship: Participate in drafting, reviewing, and/or revising the manuscript for intellectual content.
c. Approval: Approve the manuscript to be published.
People who provided technical services, administration, acquisition of funding, collection of data, editorial writing, or general supervision should not be listed as co-authors but mentioned in the acknowledgement section (see argument here).

2) Responsibilities of lead author: As a practical matter in the case of publications with multiple authors, one author should be designated as the lead author. The lead author assumes overall responsibility for the manuscript, and also often serves as the managerial and corresponding author, as well as providing a significant contribution to the research effort. A lead author is not necessarily the principal investigator or project leader. The lead author is responsible for:
a. Authorship: Including as co-authors all and only those individuals who meet the authorship criteria set forth in this policy.
b. Approval: Providing the draft of the manuscript to each individual contributing author for review and consent for authorship. The lead author should obtain from all coauthors their agreement to be designated as such and their approval of the manuscript. A journal may have specific requirements governing author review and consent, which must be followed.
c. Integrity: The lead author is responsible for the integrity of the work as a whole, and ensuring that reasonable care and effort has been taken to determine that all the data are complete, accurate, and reasonably interpreted.

3) Responsibilities for co-authors: All co-authors of a publication are responsible for:
a. Authorship: By providing consent to authorship to the lead author, co-authors acknowledge that they meet the authorship criteria set forth in section 1 of this policy. A coauthor should have participated sufficiently in the work to take responsibility for appropriate portions of the content.
b. Approval: By providing consent to authorship to the lead author, co-authors are acknowledging that they have reviewed and approved the manuscript.
c. Integrity: Each co-author is responsible for the content of all appropriate portions of the manuscript, including the integrity of any applicable research.

4) Talk about contributions early: Co-authors should talk early in the process about the order of authorship, degree/kind of contribution, and the decision-making process (how decisions are made and who has final say if a consensus is not reached). The order of authors is a collective decision of the authors (not just by the lead author).

5) Deal with disagreements: If disagreement arises, make every effort to resolve the dispute locally among the authors. If necessary, get the principal investigator or the Ombuds office involved.

6) Revise degrees of contributions: If authorship seems straightforward, the order of authors can be arranged in advance but with the caveat that this could change if contributions change significantly. Create a culture of transparency and revisit the issue of order of authorship periodically in case contributions (or assumptions about contributions) have changed. Towards the end of the writing process, each co-author should describe his/her contributions and what he/she thinks every other author contributed (this can reveal misunderstandings and provides the opportunity for clarification).

7) Approve final document: Each co-author should review drafts and approve the final version before submission.

8) Describe contributions for reader: When submitting to the journal, include a short description of each co-author's contributions and how order was assigned to help readers interpret roles correctly.

References:
Ilakovac V, Fister K, Marusic M, Marusic A (January 2007)."Reliability of disclosure forms of authors' contributions".Canadian Medical Association Journal 176 (1): 41–6.doi:10.1503/cmaj.060687

Open access academic publishing

The traditional academic publishing process is flawed. Tax-money funded research is published in journals that are only available through very expensive subscriptions. The public gets charged twice (for the research and access to the publications). Research should be made more accessible so  it can be used for decision-making and further research.

You can learn about open access academic publishing at Open Access Week: http://www.openaccessweek.org/

You can get open access to scientific publications here:

• Directory of Open Access Journals: http://www.doaj.org/home
• The Public Library of Science (PLoS): http://www.plos.org/

The video below, illustrated by Jorge Cham from PhD Comics, discusses some of the main elements of the open access debate:

Academic Workflow of an Education Researcher

As an academic researcher, I use a number of software and hardware tools in concert. To illustrate my academic workflow, I created the diagram below [Click on image to enlarge].

Academic Workflow Diagram

"Input" includes the tools I use to collect and organize documents. I distinguish between different document types and input sources. My two favorite tools to organize documents are DevonThink and PersonalBrain. DevonThink allows organizing documents into a hierarchical folder structure. The in-built AI supports storting out new documents into matching folders. PersonalBrain is a dynamic 3D mindmap that allows annotating and tagging weblinks and documents. I currently have over 8000 weblinks organized in PersonalBrain.

"Output" includes the tools I use to write and publish documents. To write academic publications, I use Scrivener to create an outline of the documents and work on drafts. I enter reference tags from Sente into Scrivener. Once I reach a final draft, I export the text from Scrivener to Word and use Sente to generate a bibliography.

How a single person "wrote" 85,000 books

Philip M. Parker is professor of Management Science an international business school based in France. In the last five years alone, he published over 85,000 books on a wide variety of topics: Rotary pumps; Avocados: A Medical Dictionary; Toilet Brushes; Dictionary for Romanian Crossword Puzzles; Golf Bags in India; or Chinese Prawn Crackers in Japan.

How is possible that a single person could publish so many books on such a variety of topics? Because Parker doesn't write books, he compiles them. Parker created a (patentend) computer algorithm that automatically generates a book from a big database. It takes the program about 20 minutes to complete a book.

The books are not printed unless somebody actually orders one. All the books are available on amazon.com.

Read the full article on Parker's book-writing software on Neatorama How to write 85000 books.

Huge stack of books from Crestock Creative Photos