Sunken Worlds, P-Hacking, and Verifying
Leave a commentFebruary 5, 2025 by libroshombre
Though nearly 2,500 years apart, Einstein and Aristotle had the same thought: “The more you know, the more you know you don’t know.” As former secretary of defense Donald Rumsfeld famously said, “There are known knowns. These are things we know that we know. There are known unknowns … But there are also unknown unknowns.” But did he really say that? Never hurts to check. Duke University Library’s article, “Who Said That? Librarian Tips for Verifying Quotes,” recommended using authoritative reference sources, like Bartlett’s Familiar Quotations, or “using Google Books and searching by the author or person’s name plus keywords or phrases from the quote.” The point is, in this age of misinformation verification is crucial. For example, a number of broadcast outlets carried “18 of the Craziest and Most Unusual Town Names in Texas” that I verified using the reliable Texas Handbook. Ding Dong was named after Zulis Bell and his nephew Bert who ran a country store that had a sign with two bells and the words “Ding Dong”), Bug Tussle came “from an incident in the 1890s when a swarm of insects spoiled an ice cream social”), and in the 1870s Joseph Brown operated a mill in his tiny post office-less community where he kept a box for locals to leave letters and dimes for postage that he relayed to the nearest post office, and the village became known as Dime Box.
Anything seen, read, or heard over the internet needs verification to be trusted. I like several YouTube science channels, such as Anton Petrov’s “Whatdamath”, and celebrity scientist Neil DeGrasse Tyson’s “Star Talk,” but I still research their topics I find of interest. Recently Petrov talked about LLVPs (Large Low-Level Velocity Provinces) that are defined by Harvard.edu as “continent-sized low-velocity zones,” that is, seismic waves are disrupted when they move through them. LLVPs are huge blobs of some unknown sort of matter found mostly under Africa on one side of the planet, and under the mid-Pacific Ocean on the other. It’s uncertain what the blobs are composed of and how they got there. However, a paper by Yulong Su was published last month in the professional journal Nature wherein he and his co-authors propose that the LLVPs are composed of “partially molten subducted ocean crust.” In short, they think the LLVPs got there by tectonic plates sliding under other plates and partly melting as they approach Earth’s hot core. LiveScience.com’s article on the phenomena by Harry Baker was published two weeks ago and is titled “Scientists Discover ‘Sunken Worlds’ Hidden Deep Within Earth’s Mantle That Shouldn’t Be There.” Despite its inflammatory headline, the article itself is quite informative. Though first discovered in the 1980s, new seismic technology is permitting higher resolution mapping of the Earth’s mantle, but “these mysterious blobs appear in places they should not, leaving researchers scratching their heads.”
On an episode of Star Talk titled “My Response to Terrence Howard,” Tyson dismantles a carecent public accusation by actor Terrence Howard, who appeared on a recent Joe Rogan podcast and claimed that when he gave a short paper describing his new mathematical theory to Tyson that he was rude, abusive, and dismissive when he responded. His theory was “1 x 1 = 2”, and Howard said “How can it equal one? If one times one equals one that means that two is of no value because one times itself has no effect.” On Rogan’s show, Howard’s “attempted to debunk the Pythagorean theorem, claimed he can kill gravity, said he does not believe in the number zero, and claimed he remembers the events of the day he was born,” according to Wikipedia. Tyson said, as a courtesy to his mom, he agreed to give Howard’s paper a professional peer review analysis and found it wanting in many respects. Tyson described that process in detail and outlined that a peer review requires the reviewer to be honest and unbiased in their evaluation. The peer review process Tyson described begins when an article’s sent to a scholarly journal and given a once-over by the editor to see if it’s appropriate for that publication. If it is, the editor selects experts in the same field as the article’s author, i.e. a professional “peer,” and the peers don’t know the authors and vice versa. The peer assesses the article’s quality, methodology, potential bias and ethical concerns, and forward their opinions and questions back to the editor.
Tyson also said Howard presented an excellent example of the Dunning-Kruger Effect. According to Psychology Today, “The Dunning-Kruger effect is a cognitive bias in which people wrongly overestimate their knowledge or ability in a specific area. This tends to occur because a lack of self-awareness prevents them from accurately assessing their own skills. The concept of the Dunning-Kruger effect is based on a 1999 paper by Cornell University psychologists David Dunning and Justin Kruger.” They tested participants’ “logic, grammar, and sense of humor, and found that those who performed in the bottom quartile rated their skills far above average … Confidence is so highly prized that many people would rather pretend to be smart or skilled than risk looking inadequate and losing face. Even smart people can be affected by the Dunning-Kruger effect because having intelligence isn’t the same thing as learning and developing a specific skill.” This Effect’s especially worrisome when it manifests in scientists. Thank goodness we have Data Colada (“Thinking about evidence, and vice versa”). This blog was established in 2013 by three top-flight scientists, Uri Simonsohn, Leif Nelson and Joe Simmons, whose 700-1000 word-long posts discuss potential acts of research-malpractice, particularly in the realm of social science. As the Wikipedia page notes, Data Colada “is known for its advocacy against problematic research practices such as p-hacking, and for publishing evidence of data manipulation and research misconduct in several prominent cases, including celebrity professors Dan Ariely and Francesca Gino.”
Gino and Ariely are both “superstar researchers” in social science who have played fast and loose with facts, stats, and findings in their major papers and popular books for years. They aren’t alone, and that’s why the Data Colada team “objected to the then widespread practice of cherry-picking data and attempts to make insignificant results appear statistically credible, especially an approach for which they coined the term p-hacking (defined by Wordspy.com as “Manipulating scientific data so that the results appear to be statistically significant.” Another word for p-hacking is data dredging, which Wiktionary says “is the misuse of data analysis to find patterns in data that can be presented as statistically significant, thus dramatically increasing and understating the risk of false positives. This is done by performing many statistical tests on the data and only reporting those that come back with significant results.” Data Colada was tipped off to Gino’s falsifying her research by other researchers who tried to replica her experiments and got very different results. So did the Data Colada team. Since Gino was the top-salaried professor at Harvard Business School, Data Colada informed the school of their findings and agreed to not publicize it until the school conducted their own independent analysis. They did that and put Gino on immediate unpaid administrative leave. Then another researcher at the University of Montreal, Erinn Acland, looked into Gino’s books and papers and found numerous instances of plagiarism. Science Magazine “confirmed Acland’s findings and identified at least 15 additional passages of borrowed text in Gino’s two books, including her “Rebel Talent: Why it Pays to Break the Rules at Work,” which should have been a tip off.
The world’s awash in intentional misinformation, but an individual can get informed by credible sources. One is Snopes.com (“The definitive Internet reference source for researching urban legends, folklore, myths, rumors, and misinformation”), and the more specific Factcheck.org (“a nonpartisan, nonprofit “consumer advocate” for voters that aims to reduce the level of deception and confusion in U.S. politics”). I recently used Snopes to verify that Mississippi State Senator Bradford Blackmon indeed introduced a “Contraception Begins at Erection Act.” Snopes said that’s true, but “it appears to be social commentary on reproductive rights more than anything” since Blackmon stated, “the vast majority of bills relating to contraception and/or abortion focus on the women’s role when men are fifty percent of the problem.” Factcheck offers a helpful checklist, “How to Spot Fake News”: consider the source, read beyond the headline, check the author and their supporting facts and the article’s date, determine if it’s a joke (a la the Borowitz Report), and check your own biases since “confirmation bias leads people to put more stock in information that confirms their beliefs and discount information that doesn’t.” As Robert McCloskey, author of the immortal “Make Way for Ducklings,” wrote “I know that you believe you understand what you think I said, but I’m not sure you realize that what you heard is not what I meant.”