Improving academic publishing through hypothesis registration, open reviews, and open data

Academic journals
[Source: http://www.wun.ac.uk/wun/research/view/world-class-universities]

There are three promising approaches for improve academic publishing:

1) Hypothesis preregistration. Currently, many academic studies gather large datasets and then search for trends/patterns within that dataset, without first stating their hypotheses. Such a post-hoc analysis, also called 'fishing', can lead to a Texas Sharpshooter falacy, which often arises when researchers have a large amount of data at their disposal, but only focus on a small subset of that data. The fallacy is characterized by a lack of a specific hypothesis prior to the gathering of data, or the formulation of a hypothesis only after data have already been gathered and examined. The fallacy occurs when the same data is used to construct and test the same hypothesis (hypotheses suggested by the data).

Suggested solution: Researchers pre-register their hypothesis, research designs, and analysis plans with a journal and publish them in advance, for example with the Open Science Foundation. Such a pre-registration allows for stronger explanatory and confirmatory research. An example of a journal that already practices hypothesis registration is Learning at Scale.

2) Many academic publications only publish the summaries of their data. This does not allow other researcher to verify the data analysis or re-use the dataset for further studies.

Suggested solution: Open data approaches allow researchers, whenever possible, feasible, and ethical, to make their dataset available for inspection and additional analysis. This also allows replication studies and an in-depth inspection by reviewers.

3) Reviewers and editors of academic journals often spend a lot of time reviewing submitted manuscripts. The academic publishing process is considered a 'dialogue' between the authors, the editor, and the reviewers. However, this dialogue happens usually behind closed doors (masked reviews) and the often substantial amount of work by the reviewers does not receive any official recognition.

Suggestion solution: Open reviews publish the peer review comments alongside the article. This gives the reviewers credit for their work and enables readers to judge for themselves how well the paper responded to the critique of peers. It could even be considered that reviewers and editors receive credit for the paper (similar to the credits in a movie that list all people who contributed in some form). Some journals even list reviewer reports with a separate DOI. Open review can also reveal the identity of authors and reviewers (rather than a single blind or double blind review). Open review could encourage a more authentic academic dialogue. Open reviews with a publication of peer review reports can contribute to greater transparency (see some examples here).

Tools to visualize connections between academic publications

Academic researchers publish a lot of papers. One estimate puts the count at 1.8 million articles published each year, in about 28,000 journals (in 2012). The challenge is to make sense of all these publications.

With the raise of visualization tools, one would expect that there are some great tools out there that provide researchers with easy to use visualizations of how papers and authors are connected.

I distinguish between three different types of visualization tools: Impact factor, co-authors, and citations. Unfortunately, there currently seems to be no tool available that combines all three.

1. Tools to visualize the impact factor of a journal (impact factor diagrams)
Eigenfactor [free] offers dynamic graphs of the impact factor of journal by field. One can choose the field of interest, e.g. chemistry, education, or physics. The dynamic graph shows the journals with the highest impact factors on a timeline from 1997 to the present (slider below the graph). Eigenfactor is an academic research project co-founded by Jevin West and Carl Bergstrom and sponsored by the Bergstrom Lab in the Department of Biology at the University of Washington.

Eigenfactor 'motion graphs' show the impact factor of journals (on a timeline)

2. Tools to visualize the relations between co-authors (co-author mapping)
Do you want to see which researchers published together as co-authors? The 'Visual Explorer' of Microsoft Academic Search [free] focuses on the relations between people. After selecting a certain author, one can choose 'co-author path' in the left hand menu. The dynamically generated diagram show the co-authors of this researcher ('co-author graph'), how this researcher is connected to another researcher ('co-author path' - similar to the Six degrees of Kevin Bacon index), and who cited this researcher ('citation graph').

The 'Visual Explorer' by Microsoft Academic search shows relations between co-authors

3. Tools that visualize the citations between papers (citation mapping)
Do you want to know which papers cited a certain publication (forward mapping)? Do you want to see which papers were cited by a publication? Papercube [free] provides a visual navigation of academic citation networks. The data is a static snapshot of the CiteSeer digital library from 2004. The tools only serves to demonstrate the features but is unfortunately no longer updated.

Papercube shows academic citation networks (based on 2004 data)

Another tool to visualize citation networks is the Web of Science citation graph [paid through libraries]. This tool generates a dynamic graph of citations (forward and backwards). One can click on a specific publication to see details and to re-center the diagram.

Web of Science citation graph

The VOSviewer (which stands for 'visualizing scientific landscapes) is a free stand-alone tool developed by Leiden University. The VOSviewer tool can be used for constructing and visualizing bibliometric networks. These networks may for instance include journals, researchers, or individual publications, and they can be constructed based on co-citation, bibliographic coupling, or co-authorship relations. Reference lists can be imported from Web of Science, Scopus, and PubMed.

VOSviewer

SciMAT (developed by the University of Granada, Spain) is an open source (GPLv3) software tool developed to perform a science mapping analysis under a longitudinal framework. SciMAT allows  to conduct co-word, author co-citation, journal co-citation, author co-citation, bibliographic coupling, journal bibliographic coupling and author bibliographic coupling.

SciMAT

A list of other software for mapping scientific publications:

• Bibexcel

• CiteSpace

• CoPalRed

• IN-SPIRE

• Loet Leydesdorff’s software

• NetworkWorkbench Tool

• Science of Science Tool (Sci2)

• VantagePoint

• Pajek Wiki http://pajek.imfm.si
• OSviewer www.vosviewer.com
• NodeXL http://nodexl.codeplex.com
• NetDraw http://sites.google.com/site/netdrawsoftware
• Cytoscape www.cytoscape.org

Given the advances in data analytics, it is surprising (and disappointing) that there are no better designed tools for academics to explore and discover connections between publications.

Celebrating a ResearchGate milestone

Order of academic authorship (and how to avoid conflicts)

Photo by Shironosov/ iStock

Academic papers are often co-authored by several authors, from two to several hundred (for example in CERN physics studies).

Rules for the order of multiple authors vary by discipline, but generally there are three possibilities:

• By degree of contribution: Authors are listed in descending order of contribution. The principal investigator is often placed last in the author list. However, some universities want to change this practice by only having the principal investigator listed if he/she actively contributed to that particular paper.
• Alphabetically: Authors are listed in alphabetical order (by family name).
• Random: Authors are listed in randomized order (rarely used).

In theory, deciding on an order of authors should be a straightforward process. However, disputes over the order of authors can ensue. Ilakovac (2007) in the Canadian Medical Association Journal reported that over two-thirds of 919 surveyed authors disagreed with their coauthors regarding contributions of each author.

There are several suggestions how to avoid authorship conflicts (compiled from source #1 and source #2):

1) Who counts as an author? Only people who made substantial, direct, intellectual contributions should be listed as authors. All authors should meet the following three criteria, and all those who meet the criteria should be authors:
a. Scholarship: Contribute significantly to the conception, design, execution, and/or analysis and interpretation of data.
b. Authorship: Participate in drafting, reviewing, and/or revising the manuscript for intellectual content.
c. Approval: Approve the manuscript to be published.
People who provided technical services, administration, acquisition of funding, collection of data, editorial writing, or general supervision should not be listed as co-authors but mentioned in the acknowledgement section (see argument here).

2) Responsibilities of lead author: As a practical matter in the case of publications with multiple authors, one author should be designated as the lead author. The lead author assumes overall responsibility for the manuscript, and also often serves as the managerial and corresponding author, as well as providing a significant contribution to the research effort. A lead author is not necessarily the principal investigator or project leader. The lead author is responsible for:
a. Authorship: Including as co-authors all and only those individuals who meet the authorship criteria set forth in this policy.
b. Approval: Providing the draft of the manuscript to each individual contributing author for review and consent for authorship. The lead author should obtain from all coauthors their agreement to be designated as such and their approval of the manuscript. A journal may have specific requirements governing author review and consent, which must be followed.
c. Integrity: The lead author is responsible for the integrity of the work as a whole, and ensuring that reasonable care and effort has been taken to determine that all the data are complete, accurate, and reasonably interpreted.

3) Responsibilities for co-authors: All co-authors of a publication are responsible for:
a. Authorship: By providing consent to authorship to the lead author, co-authors acknowledge that they meet the authorship criteria set forth in section 1 of this policy. A coauthor should have participated sufficiently in the work to take responsibility for appropriate portions of the content.
b. Approval: By providing consent to authorship to the lead author, co-authors are acknowledging that they have reviewed and approved the manuscript.
c. Integrity: Each co-author is responsible for the content of all appropriate portions of the manuscript, including the integrity of any applicable research.

4) Talk about contributions early: Co-authors should talk early in the process about the order of authorship, degree/kind of contribution, and the decision-making process (how decisions are made and who has final say if a consensus is not reached). The order of authors is a collective decision of the authors (not just by the lead author).

5) Deal with disagreements: If disagreement arises, make every effort to resolve the dispute locally among the authors. If necessary, get the principal investigator or the Ombuds office involved.

6) Revise degrees of contributions: If authorship seems straightforward, the order of authors can be arranged in advance but with the caveat that this could change if contributions change significantly. Create a culture of transparency and revisit the issue of order of authorship periodically in case contributions (or assumptions about contributions) have changed. Towards the end of the writing process, each co-author should describe his/her contributions and what he/she thinks every other author contributed (this can reveal misunderstandings and provides the opportunity for clarification).

7) Approve final document: Each co-author should review drafts and approve the final version before submission.

8) Describe contributions for reader: When submitting to the journal, include a short description of each co-author's contributions and how order was assigned to help readers interpret roles correctly.

References:
Ilakovac V, Fister K, Marusic M, Marusic A (January 2007)."Reliability of disclosure forms of authors' contributions".Canadian Medical Association Journal 176 (1): 41–6.doi:10.1503/cmaj.060687

Crowdsourcing book publishing

UK-based company Unbound is a crowdfunding company for book publishing. Similar to kickstarter, authors pitch their book ideas directly to you. If you back a project before it reaches its funding target, you get your name printed in the back of every copy and immediate behind-the-scenes access to the author’s shed. If any project fails to hit its funding target, you get refunded in full.

Open access academic publishing

The traditional academic publishing process is flawed. Tax-money funded research is published in journals that are only available through very expensive subscriptions. The public gets charged twice (for the research and access to the publications). Research should be made more accessible so  it can be used for decision-making and further research.

You can learn about open access academic publishing at Open Access Week: http://www.openaccessweek.org/

You can get open access to scientific publications here:

• Directory of Open Access Journals: http://www.doaj.org/home
• The Public Library of Science (PLoS): http://www.plos.org/

The video below, illustrated by Jorge Cham from PhD Comics, discusses some of the main elements of the open access debate:

How to get the most out of attending research conferences

Research conference presentation

How to get the most out of attending research conferences

Pre

-Look at the program: Search for key players names or keywords.

-Contact people to meet for lunch or dinner

-Look at attractions of the conference location

During

-Have your business cards ready (e.g. in the back of your name badge holder).

-Observe examples of good and bad presentation strategies.

-Take notes of cutting edge research findings and methods (that you might use for your own work)

-Network and socialise: meet new people, meet old friends, connect other people, talk to mentors.

-Identify hot topics (for possible future research/ grant proposals).

-Extend your horizon by attending talks from areas other than your own research.
-Take breaks. Going to events non-stop is exhausting. Go see some sights of the location.

Post

-Send follow-up emails to people you talked to.

-Go through your notes and extract valuable information.

-Share interesting notes with the community, e.g. blog.

-Read interesting papers presented at the conference.

Additional tips on successfully navigating conferences can be found here: http://www.gradhacker.org/2012/06/25/successfully-navigating-conferences/

Internet and global inequality

The internet promises the provide equal access to information independent of location. However, according to research by the Oxford Internet Institute (published in the book, Geographies of the World's Knowledge), user-generated content is not equally contributed: Especially the U.S. dominates the production of user content. The authors suggest that the causes for this could be a) that people in less developed countries have less leisure time to produce content for free, and b) that people without money can't afford education necessary to produce quality content.

Unfortunately, the internet is not a magic wand that makes the world more equal.


Source: The Internet Does Not Solve Global Inequality - The Atlantic

Open access to research journals

(Source: http://www.tdl.org/2010/02/open-journal-systems-and-academic-freedom/)

Academic publishing is a strange business: Researchers do not get paid by journals for their work, and neither do reviewers or editors. However, people (and libraries) interested in reading the journal need to pay exorbitant fees to access the articles. Without a full subscription, a single article can cost well over $30 (all written, reviewed, and edited by people for free). In addition, university research is mostly funded by tax money, as is the money used by public libraries to pay for access to articles written by the same researchers.

A current boycott of over 8,000 researchers of one of the largest academic publishers, Elsevier, aims to expose and change their agressive business practices (See The Cost of Knowledge).

It is in the very nature of academic knowledge that it should be widely accessible. More and more researchers are looking into alternatives, for example by making their articles available for free on their own websites, or by publishing in Open Access journals (see Directory of Open Access Journals or Public Library of Science (PLoS) journals).

Read more about Open Access of academic journals here.

Academic Workflow of an Education Researcher

As an academic researcher, I use a number of software and hardware tools in concert. To illustrate my academic workflow, I created the diagram below [Click on image to enlarge].

Academic Workflow Diagram

"Input" includes the tools I use to collect and organize documents. I distinguish between different document types and input sources. My two favorite tools to organize documents are DevonThink and PersonalBrain. DevonThink allows organizing documents into a hierarchical folder structure. The in-built AI supports storting out new documents into matching folders. PersonalBrain is a dynamic 3D mindmap that allows annotating and tagging weblinks and documents. I currently have over 8000 weblinks organized in PersonalBrain.

"Output" includes the tools I use to write and publish documents. To write academic publications, I use Scrivener to create an outline of the documents and work on drafts. I enter reference tags from Sente into Scrivener. Once I reach a final draft, I export the text from Scrivener to Word and use Sente to generate a bibliography.

Template for how journalists write about scientific publications

Here is a generic template how web-journalists write about scientific publications. Funny, but also holds some truth (as all good humor).

This is a news website article about a scientific finding | Martin Robbins | Science | guardian.co.uk