A Retraction Notice Not Retrieved: Wrong DOI

February 25th, 2024
by jodi

Part 2 of an occasional series on the Empirical Retraction Lit bibliography

Our systematic search for the Empirical Retraction Lit bibliography EXCLUDES retraction notices or retracted publications using database filters. Still, some turn up. (Isn’t there always a metadata mess?)

While most retraction notices and retracted publications can be excluded at the title screening stage, a few make it through to the abstract screening, and, for items with no abstracts, to the full-text screening. Today’s example is “Retraction of unreliable publication“. Kept at the title-screening stage**; no abstract; so it’s part of the full-text screening. PubMed metadata would have told us it’s a “Retraction of Publication” – but this particular record came from Scopus.

The Zotero-provisioned article, “Clinical guidelines: too much of a good thing“, had nothing to do with retraction so I went back to the record (which had this link with the Scopus EID). To see what went wrong, I searched Scopus for EID(2-s2.0-84897800625) which finds the Scopus record, complete with an incorrect DOI: 10.1308/xxx which today takes me to a third article with another DOI.***

Scopus Preview is even more interesting because it shows the EMTREE terms “note” and “retracted article” (which are not so accurate in my opinion):

In my 2020 Scientometrics article, I cataloged challenges in getting to the full-text retraction notice for a single article. It’s not clear how common such errors are, nor how to systematically check for errors.

I’m continuing to think about this, since, for RISRS II, I’m on the lookout for metadata disasters (in research-ese: What are the implications of specific instances of successes and failures in the metadata pipeline, for designing consensus practices?)

This particular retrieval error is due to the wrong DOI – which could affect any article (not just retraction notices). I’ve reported the DOI error to the Scopus document correction team.

It’s helpful that working on the Empirical Retraction Lit bibliography surfaces anomalous situations.

**Keeping “Retraction of unreliable publication” for abstract screening may seem overgenerous. But consider the title “Retractions”. Surely “Retractions” is the title of a bulk retraction notice! Nope, it’s a research article in the Review of Economics and Statistics by Azoulay, Furman, Krieger, and Murray. Thanks, folks. While plurals are more likely than singulars to signal research articles and editorials I try to keep vague/ambiguous titles for a closer look.

***For 10.1308/xxx Crossref just lists this latest article. Same with Scopus.

But my university library system has multiple results – a mystery!

Tags: , , , , , ,
Posted in Empirical Retraction Lit, information ecosystem, library and information science, scholarly communication | Comments (0)

Today in The Hill: Science is littered with zombie studies. Here’s how to stop their spread.

November 26th, 2023
by jodi

My newest piece is in The Hill today: Science is littered with zombie studies. Here’s how to stop their spread.

Many people think of science as complete and objective. But the truth is, science continues to evolve and is full of mistakes. Since 1980, more than 40,000 scientific publications have been retracted. They either contained errors, were based on outdated knowledge or were outright frauds. 

Identifying these inaccuracies is how science is supposed to work. …Yet these zombie publications continue to be cited and used, unwittingly, to support new arguments. 

Why? Almost always it’s because nobody noticed they had been retracted. 

Science is littered with zombie studies. Here’s how to stop their spread. Jodi Schneider in The Hill

Thanks to The OpEd Project, the Illinois’ Public Voices Fellowship, and my coach Luis Carrasco. Editorial writing is part of my NSF CAREER: Using Network Analysis to Assess Confidence in Research Synthesis. The Alfred P. Sloan Foundation funds my retraction research in Reducing the Inadvertent Spread of Retracted Science including the NISO Communication of Retractions, Removals, and Expressions of Concern (CREC) Working Group.

Tags: , , ,
Posted in information ecosystem, scholarly communication | Comments (0)

Last call for public comments: NISO RP-45-202X, Communication of Retractions, Removals, and Expressions of Concern

November 26th, 2023
by jodi

I’m pleased that the draft Recommended Practice, NISO RP-45-202X, Communication of Retractions, Removals, and Expressions of Concern (CREC) is open for public comment through December 2, 2023. I’m a member of the NISO Working Group which is funded in part by the Alfred P. Sloan Foundation in collaboration with my Reducing the Inadvertent Spread of Retracted Science project.

The NISO CREC Recommended Practice will address the dissemination of retraction information (metadata & display) to support a consistent, timely transmission of that information to the reader (machine or human), directly or through citing publications, addressing requirements both of the retracted publication and of the retraction notice or expression of concern. It will not address the questions of what a retraction is or why an object is retracted.

NISO CREC

Tags: , , , , ,
Posted in future of publishing, information ecosystem, Information Quality Lab news, library and information science, scholarly communication | Comments (0)

Towards a better process for scoping review updates: Empirical Retraction Lit

November 19th, 2023
by jodi

Part 1 of an occasional series on the Empirical Retraction Lit bibliography

In 2020, my team released the first version of the Empirical Retraction Lit bibliography, updated a number of times. The last updates are July 2021 (content); September 2021 (taxonomy); December 2022 (JSON/web design giving access to the taxonomy).

The bibliography is part of my Alfred P. Sloan-funded project, Reducing the Inadvertent Spread of Retracted Science, and it has also been an avenue for me to experiment with literature review automation and bibliography web tools. Since August members of my lab have been writing up a review on post-retraction citation, building on work a number of people have done on the review over the past several years. To update the content, we’re also working on a systematic search and screening process.

I expect a substantial number of new items. In July 2021 we had 385 items. After that I’d been estimating perhaps 7 new papers a month, which would mean ~175 new items July 2021-August 2023 (since our systematic search was September 5, 2023). That ballpark number seems plausible now that I’m in the middle of full-text screening. 2+ years is a very long time in retraction research, especially with the attention retraction has been receiving in the past few years!

A number of questions arise in trying to optimize a scoping review update process. Here are just a few:

  • Is the search I used last time adequate? Should it be updated or changed in any way?
    • Is date-based truncation appropriate for my search?
    • Is it appropriate to exclude certain items from the initial search (e.g., data, preprints)?
    • Is there a high-precision way to exclude retraction notices and retracted publications when the database indexing is insufficient?
    • Could citation-based searching in one or several sources replace multi-database searching on this topic? What are its precision and recall?
    • Are there additional databases that should be added to the search?
    • Is additional author-based searching relevant?
  • What is the most painless and effective way to deduplicate items? (Complicated in my case by retraction notices; retracted publications; and non-English language items that have multiple translations.)
  • Which items without abstracts may be relevant in this topic?
  • What is the shortest item that can make a research contribution in this topic?
  • Is original, “empirical” research a clear and appropriate scope?

Ideally the Empirical Retraction Lit bibliography will become a living systematic review that relies on as much automation and as little human effort as appropriate, with an eye towards monthly updates. My experimentation with existing automation tools makes this plausible. Routinely looking at a few papers a month seems feasible as well, especially since I could repurpose the time I spend in ad-hoc tracking of the literature, which has missed a number of items compared to systematic searching (even some high-profile items in English!).

Automation is also becoming more palatable now that I’ve found errors from the laborious human-led review: at least 2 older, in-scope items that were not included in the July 2021 version of the review, presumably because they were poorly indexed at the time of previous searches; and an informally published item that appears to have been erroneously excluded, presumably due to confusion that we were only excluding data and preprints when the bibliography included a related item.

Of course, for the website, there are a number of open questions:

  • How can the bibliography website be made useful for authors, reviewers, and editors? Awareness of related publications becomes even more important because the retraction literature is very scattered and diffuse!
  • How can the complex taxonomy of topics be made clear?
  • Would “suggest a publication” be a useful addition?

My aims in writing are:

  • To share my experience about the tools and processes I’m using.
  • To documenting the errors I make and the problems I have. This:
    • will remind me in the future (so I make different mistakes or try different tools).
    • can inform tool development.
    • can inspire systematic analysis of pragmatic information retrieval.
  • To identify possible collaborators for the Empirical Retraction Lit bibliography scoping review and website; for finalizing the post-retraction citation review; and for writing other reviews from the scoping review.
  • To solicit feedback from various communities working on pragmatic information retrieval, systematic review automation, retraction, library technology, scholarly publishing metadata workflows, literature review methodology, publication ethics, science of science,… more generally.

Tags: , , , ,
Posted in Empirical Retraction Lit, literature reviewing | Comments (0)

QOTD: What policymakers need…

November 17th, 2023
by jodi

What policymakers need are things that we [researchers] don’t value so much. Meta-analyses. What do we know about a given topic. If we survey 1000 papers on a given topic, what does the preponderance of the evidence say about a thing? Obviously the incentive structures in academia are about smaller mechanisms, or about making those smaller distinctions in the body of knowledge and that’s how knowledge advances and research advances, but for the policymaking space, you need to be able to translate that, like hundreds of years, half a century, decades of what we know about education, about inequality, about the STEM fields, about the research ecosystem, into that.

Alondra Nelson, discussion after the 2023 Sage-CASBS Award Lecture at the Center for Advanced Study in the Behavioral Sciences, November 16, 2023 [video]

Tags: , , ,
Posted in policymaking, scholarly communication | Comments (0)

Information Quality Lab at the 2023 iSchool Research Showcase

November 14th, 2023
by jodi

My Information Quality Lab presents 14 posters as part of the iSchool Research Showcase 2023, Wednesday noon to 4:30 PM in the Illini Union. View posters from 12 to 1; during the break between presentation sessions 2:15-2:45; and 3:30-4:30 PM.

Visualizing Race in Medicine
Chris Wiley

Three-Dimensional Archiving of Native American Artifacts at the Spurlock Museum
David Eby

Harold Baron Digital Archival Research and Publication Project
Divya Pathak

Disinformation Tactics and Designing to Deceive
Emily Wegrzyn

Who Needs a Main Entry, Anyway?
Liliana Giusti Serra, José Fernando Modesto da Silva

Epistemological Responsibility in Law and Science: Sharing the burden
Ted Ledford

How Computable is Scientific Knowledge?
Yuanxi Fu

Unified Framework for Evaluating Confidence in Research Synthesis
Hannah Smith, Yuanxi Fu, Jodi Schneider

Using argument graphs to audit reasoning in empirical scientific publications
Heng Zheng, Yuanxi Fu, Jodi Schneider

Activist Organizations and Their Strategies to Influence the Legalization of Medical Cannabis in Brazil
Janaynne Carvalho do Amaral, Jodi Schneider

Assessing Citation Integrity in Biomedical Publications: Annotation and NLP Models
Janina Sarol, Shufan Ming, Jodi Schneider, Halil Kilicoglu

Can ChatGPT Augment PDF-to-Text Conversion Errors in Scientific Publications?
Janina Sarol, Xuhan Zhang, Tanisha Roman, Jodi Schneider

Analyzing Retraction Indexing Quality in Subject-Specific and Multidisciplinary Databases
Malik Salami, Jodi Schneider

How Knowledge Intermediaries Gather and Make Sense of COVID-19 Information: An Interview Study
Togzhan Seilkhanova, Jodi Schneider

[Updated: 14!]

Tags: , , ,
Posted in Information Quality Lab news | Comments (0)

What can two-way communication between scientists and citizens enable?

September 24th, 2023
by jodi

The Washington Post quoted NIH researcher Paul Hwang: “Amazing findings in medicine are sometimes based on one patient”.

The findings here are a breakthrough discovery in a disease called ME/CFS – commonly known as chronic fatigue syndrome or myalgic encephalomyelitis – which led to a recent PNAS paper. This is an amazing moment: Without biomarkers, it’s been a contested disease “you have to fight to get”.

What really strikes me, though, is the individual interactions that created a space for knowledge production: an email from one citizen (Amanda Twinam) to one scientist (Paul Hwang); “serendipitous correspondence” from another scientist (Brian Walitt) with access to “an entire population” (9 of the 14 tested for the PNAS paper were similar to Amanda). Reading the literature, writing well-timed correspondence, and “hearing about” synergistic work going on in another lab all seem to have contributed.

Mady Hornig, a researcher not involved in the project, told the reporter: “It’s not very common that we do all of these … steps, having doctors who are really persistent about what is happening with one individual and applying a scientific lens.”

But what if we did?


Dumit, Joseph (2006). Illnesses you have to fight to get: Facts as forces in uncertain, emergent illnesses. Social Science & Medicine, 62(3), 577–590. https://doi.org/10.1016/j.socscimed.2005.06.018

Wang, Ping-yuan, Ma, Jin, Kim, Young-Chae, Son, Annie Y., Syed, Abu Mohammad, Liu, Chengyu, Mori, Mateus P., Huffstutler, Rebecca D., Stolinski, JoEllyn L., Talagala, S. Lalith, Kang, Ju-Gyeong, Walitt, Brian T., Nath, Avindra, & Hwang, Paul M. (2023). WASF3 disrupts mitochondrial respiration and may mediate exercise intolerance in myalgic encephalomyelitis/chronic fatigue syndrome. Proceedings of the National Academy of Sciences, 120(34), e2302738120. https://doi.org/10.1073/pnas.2302738120

Vastag, Brian (2023, September 19). She wrote to a scientist about her fatigue. It inspired a breakthrough. Washington Post. https://www.washingtonpost.com/health/2023/09/17/fatigue-cfs-longcovid-mitochondria/ Temporarily open to read via this gift link.

Tags: , , , ,
Posted in information ecosystem, random thoughts, scholarly communication | Comments (0)

Medical/Life Sciences Graduate or Undergraduate Student Hourly – Biomedical Literature Annotation for Citation Accuracy/Integrity (10 hours per week, spring semester) – School of Information Sciences – University of Illinois at Urbana-Champaign

January 13th, 2023
by jodi

The ScienceNLP Lab and the Information Quality Lab at the School of Information Sciences (iSchool) are seeking a University of Illinois at Urbana-Champaign student to read and annotate health-related literature. Knowledge and training and knowledge in a medical/life sciences field such as biology, chemistry, bioinformatics, food science and nutrition, or bioengineering is essential. The hourly will work an average of 10 hours per week for spring semester, under the co-supervision of Dr. Halil Kilicoglu and Dr. Jodi Schneider. The project focuses on assessing biomedical publications for citation accuracy and integrity. Your role in this project will be to locate citation statements in biomedical articles and assess their accuracy with respect to the cited articles. You will collaborate with other annotators on this task. This work is part of the project Natural Language Processing to Assess and Improve Citation Integrity in Biomedical Publications, funded by the Office of Research Integrity (ORI).

Project Description: While citations play a fundamental role in scientific knowledge diffusion and research assessment, they are often inaccurate (e.g., citation of non-existent findings), undermining the integrity of scientific literature and distorting the perception of available evidence. A recent meta-analysis showed that 25.4% of medical articles contained a citation error. A bibliometric analysis revealed that inaccurate citations of a letter published in 1980 may have contributed to the opioid crisis. The project will develop and validate resources and models that aid stakeholders in assessing biomedical publications for citation accuracy and integrity. The publicly available annotated corpus you help create will be used to develop natural language processing/artificial intelligence (NLP/AI) models for assessing reporting quality in biomedical articles.

Duties include:

  • Reading and annotating biomedical publications for citation integrity/accuracy
  • Contribution to development of annotation guidelines
  • Contribution to scientific presentations and publications

Required qualifications:

  • Background in a field such as: medicine, life sciences, including biology, chemistry, bioinformatics, food science and nutrition, bioengineering, or a related field.
  • Excellent English reading comprehension skills
  • Excellent communications skills in written and spoken English
  • Excellent analytical/critical thinking skills
  • Effective time management skills, attention to detail

Preferred qualifications:

  • Interest in topics such as trustworthy science, research rigor/quality, reproducibility
  • Interest in biomedical data science, bioinformatics, or related fields
  • Availability for multiple semesters

Interested candidates should send their CV/resume and a short statement of purpose drawing attention to their training in medicine or life sciences (e.g., biology, chemistry, bioinformatics, food science and nutrition, bioengineering, or a related field) to Halil Kilicoglu (halil@illinois.edu) and Jodi Schneider (jodi@illinois.edu). Review of applications will begin immediately. Applications will be accepted until the position is filled.

Posted on Handshake and the Virtual Job Board.

Tags: , , , , , , , , , , , , , ,
Posted in Information Quality Lab news | Comments (0)

Managing time demands: Francis Wade’s skill ladders

January 12th, 2023
by jodi

For diagnosing problems with time management, the best thing I’ve found so far is Francis Wade’s idea of “skill ladders”, described in the 2014 Perfect Time-Based Productivity: How to rescue your peace of mind as time demands increase (Amazon; book website).

For instance, the “capturing” ladder has 7 levels, with the bottom “not even trying to commit tasks to memory” and the top “always using a backed up electronic device that is never far away”:

Skill ladder for capturing, from Francis Wade’s Perfect Time-Based Productivity: How to rescue your peace of mind as time demands increase

He lists multiple ladders:

A downloadable PDF form provides a checklist to identify where you are on each ladder.

A summary of his diagnosis-based approach to managing time demands is available from The Evergreen Guide to Choosing Your Next Task Management App (2017).

More recently he organized the Task Management & Time Blocking Virtual summit (2020 & 2021 & 2022, with the 2023 event planned for Thursday through Saturday March 2-4). You can find more of Francis Wade’s recent thinking via Framework Consulting and in his Quora answers.

Tags: , , , , ,
Posted in time | Comments (0)

Graduate Hourly Position: Metadata Quality Investigation

September 19th, 2022
by jodi

Start Date: ASAP

Descriptions, Responsibilities, and Qualifications
This project offers an excellent opportunity for a University of Illinois Urbana-Champaign MSLIS student interested in metadata, data quality, database search, information retrieval and related topics. The incumbent will collect information about how well databases track retracted information, under the mentorship of Dr. Jodi Schneider, Assistant Professor and Director of the Information Quality Lab. The project will produce data analyses and reports to support a NISO Working Group in information gathering about how to improve metadata quality and display standards for retracted publications, in the Alfred P. Sloan foundation grant “Reducing the Inadvertent Spread of Retracted Science II: Research and Development towards the Communication of Retractions, Removals, and Expressions of Concern Recommended Practice”.

We will first search multidisciplinary databases (Scopus and Web of Science) as well as other sources (e.g., Crossref, Retraction Watch) for retracted publications. Then, we will compile a list of known retracted publications across these sources. We will compare across sources to identify retracted publications that have inconsistent information about whether or not they are retracted. We will also calculate which percentage of retracted publications indexed in the source are correctly indexed as retracted. We will then investigate how retractions are indexed in specific domain databases, using established retraction type indexing in biomedicine (PubMed, PubMed Europe)and psychology (PsycINFO), and investigating how retraced publications are indexed in chemistry (CAS SciFinder) and engineering (IEEE Xplore). We will also manually check indexing on a small dataset in search engines such as Google Scholar and Semantic Scholar.

Duties include:

  • Searching databases
  • Collating publication data
  • Deduplicating publication data
  • Documenting all aspects of the projects
  • Producing project memos and reports

Required Qualification:

  • Enrollment in the Master’s in Library and Information Science program at the University of Illinois at Urbana-Champaign
  • Interest in topics such as metadata, data quality, database search, etc.
  • Interest in quantitative research using publications as data
  • Detail orientation
  • Excellent communication skills in written and spoken English

Preferred Qualifications:

  • Available for continued work in spring 2023
  • Project management experience
  • Experience with quantitative data
  • Experience in database searching
  • Experience manipulating data using spreadsheet software (e.g., Excel) and/or scripting languages (e.g., R or Python)
  • Interest in reproducibility and open science
  • Interest or experience in writing research reports and/or publications

Compensation: paid as a graduate hourly through the University of Illinois, $20/hour for 10-15 hours a week.

Application Procedures: Interested candidates should send a cover letter and resume to Dr. Jodi Schneider in a single PDF file named Lastname-metadata-hourly.pdf to jodi@illinois.edu

Review of applications will begin immediately. Applications will be accepted until the position is filled. All applications received by Sunday October 2, 2022, will receive full consideration.

Posted on Handshake and on the iSchool website

Tags: , , , , , , ,
Posted in information ecosystem, Information Quality Lab news, scholarly communication | Comments (0)