Showing posts with label research. Show all posts
Showing posts with label research. Show all posts

Friday, October 21, 2016

Improving academic publishing through hypothesis registration, open reviews, and open data


Academic journals
[Source: http://www.wun.ac.uk/wun/research/view/world-class-universities]
There are three promising approaches for improve academic publishing:

1) Hypothesis preregistration. Currently, many academic studies gather large datasets and then search for trends/patterns within that dataset, without first stating their hypotheses. Such a post-hoc analysis, also called 'fishing', can lead to a Texas Sharpshooter falacy, which often arises when researchers have a large amount of data at their disposal, but only focus on a small subset of that data. The fallacy is characterized by a lack of a specific hypothesis prior to the gathering of data, or the formulation of a hypothesis only after data have already been gathered and examined. The fallacy occurs when the same data is used to construct and test the same hypothesis (hypotheses suggested by the data).

Suggested solution: Researchers pre-register their hypothesis, research designs, and analysis plans with a journal and publish them in advance, for example with the Open Science Foundation. Such a pre-registration allows for stronger explanatory and confirmatory research. An example of a journal that already practices hypothesis registration is Learning at Scale.

2) Many academic publications only publish the summaries of their data. This does not allow other researcher to verify the data analysis or re-use the dataset for further studies.

Suggested solution: Open data approaches allow researchers, whenever possible, feasible, and ethical, to make their dataset available for inspection and additional analysis. This also allows replication studies and an in-depth inspection by reviewers.

3) Reviewers and editors of academic journals often spend a lot of time reviewing submitted manuscripts. The academic publishing process is considered a 'dialogue' between the authors, the editor, and the reviewers. However, this dialogue happens usually behind closed doors (masked reviews) and the often substantial amount of work by the reviewers does not receive any official recognition.

Suggestion solution: Open reviews publish the peer review comments alongside the article. This gives the reviewers credit for their work and enables readers to judge for themselves how well the paper responded to the critique of peers. It could even be considered that reviewers and editors receive credit for the paper (similar to the credits in a movie that list all people who contributed in some form). Some journals even list reviewer reports with a separate DOI. Open review can also reveal the identity of authors and reviewers (rather than a single blind or double blind review). Open review could encourage a more authentic academic dialogue. Open reviews with a publication of peer review reports can contribute to greater transparency (see some examples here).

Monday, June 29, 2015

Wednesday, July 16, 2014

Great movies for science geeks and science teachers

Mad Scientist
(Source: http://content.science20.com/)
Many movies portrait scientists (see a list here), but only few try to give realistic depiction of science and scientists (sometimes in a historical context). Scientists are often either portrayed as mad loners (see image above) or a wizard-technicians (e.g. forensic scientists or computer hackers in almost every crime TV show).

Here is a short list of movies that do a better job at representing science (and show the scientist as a positive hero):


Name of the movie
Focus
Longitude (2000)
Technology: A clockmaker needs to collect convincing evidence to support the reliability of his method to measure the position of ships out in the ocean.
Astronomy. An astronomer strives to collect convincing evidence that the royal calendar is inaccurate.
Scientists run experiments to explore an alien virus.
Architecture. The challenges of building a Japanese castle.
Agora (2009)
Astronomy. Conflict between religion and science.

Wednesday, March 12, 2014

Farewell to the One-Laptop-Per-Child (OLPC) project

XO-1 laptop
(Source: OLPC news)
The OLPC website officially reports that "OLPC is dead" and so is the development  of the XO-1 laptops. Sections of the OLPC project seem to be still operational but the full effort has subsided.

The idealistic project who aimed to close/narrow the digital divide by giving every child a specially designed laptop did unfortunately not succeed to reach its ambitious goals.

The rugged, low-power XO-1 laptop (also known as the $100 Laptop, the Children's Machine, and the 2B1) is an inexpensive sub-notebook computer that offered many interesting design features:

  • To improve internet connection in areas with limited access, the XO-1 used a WiFi mesh networking protocol to allow many machines to share Internet access as long as at least one of them can see and connect to a router or other access point.
  • Operation system: Slimmed-down version of Fedora Linux and a GUI named Sugar 
  •  Video camera, a microphone, and a hybrid stylus/touch pad. 
  • Power: The XO-1 was designed to use minimal amounts of power. In addition to a standard plug-in power supply, human power and solar power sources are available, allowing operation far from a commercial power grid.
  • Dual screen mode: Reflective (backlight off) monochrome mode for low-power use in sunlight (ebook mode) and backlit color mode.
  • Water-resistant membrane keyboard, designed for the small hands of children.
Farewell, OLPC and XO-1. We need more idealistic projects like this.

Thursday, February 13, 2014

Tangible user interface of the future

MIT Media Lab presents a prototype for a tangible user interface. A Microsoft Kinect and a camera detect a user's hand movements and replicate them on a custom-built table with hundreds of moveable pins.
While this prototype is far from being portable, it offers a glimpse of what future tangible interfaces might look like.


Thursday, April 4, 2013

Order of academic authorship (and how to avoid conflicts)

Photo by Shironosov/ iStock

Academic papers are often co-authored by several authors, from two to several hundred (for example in CERN physics studies).

Rules for the order of multiple authors vary by discipline, but generally there are three possibilities:
  • By degree of contribution: Authors are listed in descending order of contribution. The principal investigator is often placed last in the author list. However, some universities want to change this practice by only having the principal investigator listed if he/she actively contributed to that particular paper.
  • Alphabetically: Authors are listed in alphabetical order (by family name).
  • Random: Authors are listed in randomized order (rarely used).
In theory, deciding on an order of authors should be a straightforward process. However, disputes over the order of authors can ensue. Ilakovac (2007) in the Canadian Medical Association Journal reported that over two-thirds of 919 surveyed authors disagreed with their coauthors regarding contributions of each author.

There are several suggestions how to avoid authorship conflicts (compiled from source #1 and source #2):

1) Who counts as an author? Only people who made substantial, direct, intellectual contributions should be listed as authors. All authors should meet the following three criteria, and all those who meet the criteria should be authors:
a. Scholarship: Contribute significantly to the conception, design, execution, and/or analysis and interpretation of data.
b. Authorship: Participate in drafting, reviewing, and/or revising the manuscript for intellectual content.
c. Approval: Approve the manuscript to be published.
People who provided technical services, administration, acquisition of funding, collection of data, editorial writing, or general supervision should not be listed as co-authors but mentioned in the acknowledgement section (see argument here).

2) Responsibilities of lead author: As a practical matter in the case of publications with multiple authors, one author should be designated as the lead author. The lead author assumes overall responsibility for the manuscript, and also often serves as the managerial and corresponding author, as well as providing a significant contribution to the research effort. A lead author is not necessarily the principal investigator or project leader. The lead author is responsible for:
a. Authorship: Including as co-authors all and only those individuals who meet the authorship criteria set forth in this policy.
b. Approval: Providing the draft of the manuscript to each individual contributing author for review and consent for authorship. The lead author should obtain from all coauthors their agreement to be designated as such and their approval of the manuscript. A journal may have specific requirements governing author review and consent, which must be followed.
c. Integrity: The lead author is responsible for the integrity of the work as a whole, and ensuring that reasonable care and effort has been taken to determine that all the data are complete, accurate, and reasonably interpreted.

3) Responsibilities for co-authors: All co-authors of a publication are responsible for:
a. Authorship: By providing consent to authorship to the lead author, co-authors acknowledge that they meet the authorship criteria set forth in section 1 of this policy. A coauthor should have participated sufficiently in the work to take responsibility for appropriate portions of the content.
b. Approval: By providing consent to authorship to the lead author, co-authors are acknowledging that they have reviewed and approved the manuscript.
c. Integrity: Each co-author is responsible for the content of all appropriate portions of the manuscript, including the integrity of any applicable research.

4) Talk about contributions early: Co-authors should talk early in the process about the order of authorship, degree/kind of contribution, and the decision-making process (how decisions are made and who has final say if a consensus is not reached). The order of authors is a collective decision of the authors (not just by the lead author).

5) Deal with disagreements: If disagreement arises, make every effort to resolve the dispute locally among the authors. If necessary, get the principal investigator or the Ombuds office involved.

6) Revise degrees of contributions: If authorship seems straightforward, the order of authors can be arranged in advance but with the caveat that this could change if contributions change significantly. Create a culture of transparency and revisit the issue of order of authorship periodically in case contributions (or assumptions about contributions) have changed. Towards the end of the writing process, each co-author should describe his/her contributions and what he/she thinks every other author contributed (this can reveal misunderstandings and provides the opportunity for clarification).

7) Approve final document: Each co-author should review drafts and approve the final version before submission.

8) Describe contributions for reader: When submitting to the journal, include a short description of each co-author's contributions and how order was assigned to help readers interpret roles correctly.

References:
Ilakovac V, Fister K, Marusic M, Marusic A (January 2007)."Reliability of disclosure forms of authors' contributions".Canadian Medical Association Journal 176 (1): 41–6.doi:10.1503/cmaj.060687

Tuesday, November 6, 2012

Job searching strategies for science education researchers

Below are the slides of a short presentation I gave at the Institute for Innovation in Science and Mathematics Education (IISME) on the topic of how science education researchers can find jobs after graduation.

Friday, October 26, 2012

Open access academic publishing

The traditional academic publishing process is flawed. Tax-money funded research is published in journals that are only available through very expensive subscriptions. The public gets charged twice (for the research and access to the publications). Research should be made more accessible so  it can be used for decision-making and further research.

You can learn about open access academic publishing at Open Access Week: http://www.openaccessweek.org/

You can get open access to scientific publications here:


The video below, illustrated by Jorge Cham from PhD Comics, discusses some of the main elements of the open access debate:

Wednesday, July 11, 2012

How to get the most out of attending research conferences


Research conference presentation

How to get the most out of attending research conferences

Pre
-Look at the program: Search for key players names or keywords.
-Contact people to meet for lunch or dinner
-Look at attractions of the conference location

During
-Have your business cards ready (e.g. in the back of your name badge holder).
-Observe examples of good and bad presentation strategies.
-Take notes of cutting edge research findings and methods (that you might use for your own work)
-Network and socialise: meet new people, meet old friends, connect other people, talk to mentors.
-Identify hot topics (for possible future research/ grant proposals).
-Extend your horizon by attending talks from areas other than your own research.
-Take breaks. Going to events non-stop is exhausting. Go see some sights of the location.

Post
-Send follow-up emails to people you talked to.
-Go through your notes and extract valuable information.
-Share interesting notes with the community, e.g. blog.
-Read interesting papers presented at the conference.

Additional tips on successfully navigating conferences can be found here: http://www.gradhacker.org/2012/06/25/successfully-navigating-conferences/

Tuesday, July 10, 2012

A review of 20 years of research in the Learning Sciences

Janet Kolodner
Below are my notes from the keynote talk given at the 10th bi-annual international conference of the Learning Sciences by Janet Kolodner, one of the founders of the field of Learning Sciences.

The Learning Sciences started in the late 1980s with the foundation of the "Journal of the Learning Sciences" (JLS). The Learning Sciences (LS) are a multi-disciplinary field that aims to develop models of learning in real-world situations. LS combines constructs and methods from a variety of fields, such as cognitive science, computer science, educational psychology, anthropology, science education research, and linguistics. Findings from the Learning Sciences appeal to multiple audiences and aim to have an impact on education.

Basic observations:
-In the 21st century, everyone needs to continue learning to fully participate in the workforce and citizenry (because available tools are changing fast).
-Education may be the way to close the equity gap.
-Schools (in the dominant industrial-model as we know them) can't do it all.

After 20 years of Learning Sciences research, what do we know about learning?
1) Learning deeply (as a process of mental model building) requires sustained and long-term effort and requires a lot of help (facilitation, scaffolding): Helping learners build mental models.
a) From a cognitive perspective, learning is a process of iteratively constructing, revising, and connecting concepts (mental models, ideas, etc.): Factual knowledge and procedural knowledge.
-Becoming fluid at reasoning skills is an iterative process of composing and debugging sequences of procedural knowledge ("How-to") [Schank & Abelson; Anderson; Newell, Rosenbloom & Laird]
b) We can only learn on the edges of what we already know (Zone of proximal development) [Vygotsky]
-We realise a need for learning if we can't do something we want to do, something turns out differently than expected, or something happens that we can't explain [Schank & Abelson; Anderson]
c) Revising one's mental models requires a great deal of reflection and interpretation [Bransford; Collins; Schank]
-Framing first, details later [Collins]
-Repeated deliberate practice: Help learners to repeatedly experience the result of their decisions and interpret and reflect on those results to debug their reasoning.
-Promote asking questions and wonder to identify holes in one's mental models.
-Tell only when learners are ready (A time for telling); Give learners a role in telling their peers.
-Give learners learning goals that are relevant to them to put in the mental energy to revise mental models.
-Help learners to connect ideas (mental models) by helping them recognise and revisit prior ideas [Linn]

2) Promoting and sustaining active engagement is key to learning: Intrinsic motivation
a) Personal and epistemological connections promote engagement [Papert; Resnick]
-Learners connect when they can connect to prior knowledge.
-Learners connect when they are interested, e.g. by using authentic real-world settings such as real audiences.
b) "Communities of Practice" [Lave, Wenger]
-Encouraging community (affinity groups, interest groups, communities of practice) encourages engagement.
-Community can be created in formal and informal learning environments by creating activities that promote shared values and ways of doing [Kolodner]
-Community engagement gives learners opportunities to imagine what they might be.
c) "Thick authenticity" for maintaining integrity [Schaffer and Resnick]
-Solving authentic problems relevant to the world learners live in by having available the same kinds of tools and resources professionals would have
-Big questions or challenges promote personal connection [Krajcik; Edelson; Reiser; Kolodner]
-Assessment should be authentic, embedded and meaningful to learners.
d) Turning over agency to learners allows learners to decide for themselves to participate [Holland]\
-Knowledge building with a community results in collective cognitive responsibility, mental model building and knowledge integration [Scardamalia; Bereiter; Linn].
e) Provide scaffolds that promote success (including productive failure) and self-assessment

3) The one-size-fits-all approach does not work, as everyone's interests and Zone of Proximal Development (ZPD) are different (This also applies to teachers as learners).
_____________________________________

A vision for learning environments of the future:
-Less emphasis on "lessons" and more emphasis on "challenges": Lessons are about doing something, challenges are about achieving something.
-Lectures will be for purposes identified by learners. Lectures will be often short, interactive, and impromptu. Lectures should be recorded to allow revisiting.
-Reading will be for purposes identified by learners to answer questions they have raised.
-Classrooms will be places where students and teachers address challenges together. Solving design challenges provide rich learning opportunities. Challenges can be of local relevance. Challenges can be done in groups (both within class and external).
-Learning will be purposeful: Skills and practices for living a life; for stewarding a planet; for health and well-being; for someday joining a workforce; for actively participating in civic life.
-Assessment will be purposeful: Assessment not only for accountability (e.g. current standardized assessments) but for the purpose of scaffolding and promoting self-monitoring.
-Learning technologies will be integrated in purposeful ways, e.g. models (show hidden mechanisms); simulations (allow to understand what happens when); communications to connect with communities, resources, and experts; videos; visual interfaces to access and analyse data; sharing data with real audiences; embedded assessments. Needed: Technologies should not be stand-alone but integrated across disciplines, grades and time (learning trajectories); used for learning and assessment; used by teachers and students; used in and out of school. Learning often happens during the interaction between community members while using technology - make that part of the design.

Some good examples:
-Students learn about chemistry through the challenge of improving the air quality in their community.
-Students learn about the process of science by improving cooking recipes.

How can Learning Sciences research gain more influence?
1) Practice what we preach! e.g. in teacher education courses, curriculum design
2) Building on each others' work to focus on integration: Work on coordinated projects. Integrate findings.
3) Shift towards implementing research in the community: Work with interpreters and integrators (who take the best of our ideas and products and hold them into integrated platforms and packages). Not every researcher has to take research to practice, but all should be willing to make their findings and products available.
4) In order to successfully implement technology in the classroom, teachers need to know how to use it.
5) We need to evaluate under what circumstances technology and instructional strategies are effective: First, identify why and under what circumstances technology works effectively in small trials (understand affordances and challenges); second, large controlled random trials; third, implement your innovative product (scale up, but not before rigorous testing). To make implementations successful, they need appropriate teacher professional development.
6) Work on what's important: Start by identifying challenges of national or international importance.
7) Learning science research should draw from at least three different literatures (cross- and inter-disciplinary), otherwise your work is not important enough.

Saturday, May 26, 2012

The dark side of augmented reality

Augmented Reality (Source: http://extensiveideas.com/)
Augmented reality, for example by wearing google googles, adds a digital layer to your surroundings. You can see superimposed information about buildings, people, shopping, directions, news, etc.

This New Scientist article (see below) discusses the possible dark sides of augmented reality: Digital graffiti, cyber-bullying, digital political demonstrations, or cyber-stalking.

Law makers need to catch up to regulate the use of augmented reality information - and hopefully find a sensible balance between giving users creative freedom and protecting them.

Augmented reality offers a new layer of intrigue - opinion - 25 May 2012 - New Scientist:

'via Blog this'

Identifying factors for longevity

Longevity (Source: http://www.councilforresponsiblegenetics.org/)
A 2012 Scientific American article (see link below) reported that certain genes, more precisely SNPs, occur more frequently in people who live over a 100 years than in a random population sample.

However, twin studies suggest that genetics only account for approximately 20-30% of an individual's chance of surviving to age 85.

Research by the Blue Zones research project suggests that environmental factors such as diet and a supportive community are important to live a long life.

Live Long and Proper: Genetic Factors Associated with Increased Longevity Identified: Scientific American:

'via Blog this'

Friday, May 11, 2012

Is technology-enhanced learning effective?

Image source: http://telscenter.org/projects/tels
Is Technology-Enhanced Learning Effective? What the Research Says about TEL.


The posting below is a reprint of tomorrow's professor newsletter vol 64, issue 4 (http://cgi.stanford.edu/~dept-ctl/cgi-bin/tomprof/postings.php). This short article looks at what recent research says about the effectiveness of technology-enhanced learning. It is prepared by the Research and Evaluation Team, Office of Information Technology, University of Minnesota - Twin Cities. http://z.umn.edu/research. In an effort to make research in the educational technology field more accessible, OIT's Research & Evaluation team produces frequent brief synopses of important recent studies. These synopses may be freely shared and used for non-profit academic purposes.http://z.umn.edu/briefs. For further information contact Dr. J.D. Walker (jdwalker@umn.edu).

Instructors interested in technology-enhanced learning (TEL) frequently want to know whether digital technology is educationally effective. Their question is not whether students like digital technology, or whether students are engaged by it, but instead whether it enhances student learning outcomes.

Despite a growing body of research into TEL, it is hard to give a simple answer to this question, in part because TEL studies are frequently deeply embedded in a particular context, which makes it difficult to know how well the studies generalize outside of that context.

A recent thorough and methodologically sound meta-analysis[1] by Barbara Means and colleagues for the U.S. Department of Education helps to address this problem by providing an overview of conclusions that are supported overall by the research on TEL. Means' primary concern was to compare the effectiveness of courses with an online component[2] to fully face-to-face courses.

Means used a stringent selection procedure in selecting studies for the meta-analysis, limiting the studies to those that used a comparative research design, measured learning outcomes objectively, controlled statistically for possible differences between control and treatment samples, and reported effect sizes for student learning outcomes. This procedure yielded 50 contrasts from studies conducted between 1996 and 2008.

* The DOE Meta-analysis: Findings

Online versus face-to-face : Means found that an average effect size of +0.20 (p < .001) standard deviations favoring the courses with an online component. This means that, on average, students in courses with an online component outperformed students in face-to-face courses by a small but statistically significant amount, after controlling for other factors.

Means is careful to say, however, that this finding almost certainly does not represent a pure effect of technology, or of the delivery method used in the different courses. Instead, online courses were associated with other instructional conditions, such as increased learning time, different materials, and enhanced opportunities for collaboration, which are the likely mechanisms through which they achieved superior results.

Factors that had no effect : The meta-analysis analyzed the influence of a large number of potential moderator variables and found that the main effect holds independent of the vast majority of these variables, including:

* learner type (K-12, undergraduate, graduate/professional);
* subject matter (medical/health care, others);
* type of knowledge tested (declarative, procedural, strategic); and
* type of computer-mediated communication with peers and with instructor (asynchronous only versus asynchronous plus synchronous).

Factors that had an effect : Several other moderator variables were statistically significant, however or nearly so, including:

* Blended learning : The authors separated purely online education from "blended" or "hybrid" conditions, or courses in which face-to-face instruction is enhanced or supplemented by online materials and/or activities. They then compared each of these separately to fully face-to-face conditions. Completely online instruction had an advantage of +0.05 (p = .46, not significant) standard deviations over purely face-to-face instruction, while the advantage of blended instruction over face-to-face was +0.35 (p < .001).

* Curriculum and instructional methods : When students in the online condition were exposed to a different curriculum and/or instructional methods from students in the face-to-face condition, the advantage of the online condition was +0.40 (p < .001); when these factors were equivalent across conditions, it was +0.13 (p < .05). This finding suggests that the positive effects of using online technology in education are enhanced when an instructor adapts curriculum and instructional approach to the use of technology.

* Type of online learning experience : Instructor-directed, expository learning had an effect size of +0.39 (p < .01); collaborative, interactive instruction, +0.25 (p < .001); independent, active online learning, +0.05 (not significant).

* Time on task : When students in the online condition spent more time on task than students in the face-to-face condition, the advantage of the online condition was +0.45; otherwise it was +0.18. This difference approached the threshold of statistical significance (p = .06).

Variants of online learning : Means' study also examined different ways of implementing TEL. In other words, in addition to comparing online and face-to-face learning, the meta-analysis addressed studies which compared different learning conditions all of which involve online technology.

This part of the analysis found that some frequently recommended online learning practices did not result in improved learning outcomes. These included providing multimedia in online learning materials (e.g. enhancing text with static graphic and embedded video), and incorporating quizzes into the online environment.

A group of studies which examined the effects of giving learners control over online resources produced mixed results, with some studies favoring providing learner control and others yielding null results.

However, it does appear that learning in the online environment can be improved by individualizing instruction (providing feedback and guidance customized to each learner's performance) and by promoting learner reflection (through prompts designed to foster student self-assessment and metacognition).

* Discussion and Best Practices

Blending works : This study adds to a growing consensus around the conclusion that the most effective type of instruction combines the online and face-to-face environments. Other meta-analyses which reach this conclusion are Bernard et. al (2004) and Zhao et. al (2005).

Adapting instruction works : Means' work supports the thoughtful adaptation of instructional methods and materials to the online environment, and forthcoming research by OIT's Research and Evaluation Team reaches a similar conclusion.

Self-directed online learning is not the best : Collaborative or instructor-directed online learning achieved results superior to those attained through independent, self-directed online learning, which may provide a partial explanation for why online learning has not proven to be a money-saver for cash-strapped educational institutions.


References:

1 Means, B., Toyama, Y., Murphy, R., Bakia, M., Jones, K. (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning. Center for Technology in Learning, U.S. Department of Education. Retrieved May 10, 2010 fromhttp://www.ed.gov/about/offices/list/opepd/ppss/reports.html .

2 "Courses with an online component" included courses which merely supplemented an unchanged face-to-face course with online materials, as well as courses delivered entirely online with no face-to-face interaction.