Chapter 3 - The Mathematical function and its Graph
Section 3.2 - Nonsense Detection - The UnScientific Method
From "Piled Higher and Deeper" by Jorge Cham -
www.phdcomics.com
In a perfect world all scientific research could be trusted at face
value. Unfortunately, the world is complex and science is not black or
white. There is a lot of grey area where it is difficult to differentiate
fact from opinion. Read the words of the
Dalai Lama commenting on suicide bombings:
" You know, science or knowledge is just a method. The thing is how to
use that. Science itself is wonderful. Sometimes we use that knowledge for
destruction. It's not science's mistake, it's our mistake. Similarly, some
people manipulate religion in the wrong way. It's not the fault of
religion, it is the fault of people and politics. People talk about dirty
science. There is also dirty religion and dirty politics. "
The Dalai Lama is saying that science, just like anything else, can be
manipulated to prove any point. So how does one distinguish 'dirty
science' from 'good science'? Unfortunately, the answer is not a simple
one. There are shades of good and bad science in all research. It takes a
lot of experience to know how to filter out the nonsense. This article
will summarize a few categories most bad science typically falls under. If
you see any research that shares elements of these categories then you
should be skeptical of the results.
Many conflicting scientific theories explaining the same phenomena
If you see many conflicting theories attempting to explain a scenario, then that should
raise a red flag that most of these theories are wrong. Consider
Dr. Mehmet Oz, Director of the Columbia University Heart Institute, asking
eminent American science writer,
Gary Taubes, " Why is it that we can not
agree on dietary recommendations? "
" A couple major issues, first of all ... disagreements
come about because the science is surprisingly complicated,
the human body is incredibly complicated, all people are different, diseases
are divergent and different, people and their genetic elements and physiological
elements and lifestyle elements.
So when you actually get around to trying
to test it, you end up with this morass of confusing and conflicting data
out of which people can pick just the elements they want to support
their preconceived opinions.
And that sometimes make the particular researcher
look very sure that he knows the answer or she knows the answer, but
unfortunately that is not how you do good science. "
Lack of complexity implies unattractive conclusion which lack
glamor and perceived importance
Consider the excerpt from an article in
The New Yorker by eminent
surgeon Dr. Atul Gawande.
In this excerpt, Dr. Gawande describes how Dr.
Pronovost developed simple medical checklists that have proven to save
countless lives . Gawande writes, " If a new drug were as effective at
saving lives as Peter Pronovost's checklist, there would be a nationwide
marketing campaign urging doctors to use it. " As you will read below, his
checklists have fallen on deaf ears due to their lack of perceived
complexity!
We have the means to make some of the most complex and dangerous work
we do in surgery, emergency care, and I.C.U. medicine more effective
than we ever thought possible. But the prospect pushes against the
traditional culture of medicine, with its central belief that in
situations of high risk and complexity what you want is a kind of
expert audacity.the right stuff, again. Checklists and standard
operating procedures feel like exactly the opposite, and that's what
rankles many people.
The still limited response to Pronovost's work may be easy to explain,
but it is hard to justify. If someone found a new drug that could wipe
out infections with anything remotely like the effectiveness of
Pronovost's lists, there would be television ads with Robert Jarvik
extolling its virtues, detail men offering free lunches to get doctors
to make it part of their practice, government programs to research it,
and competitors jumping in to make a newer, better version. That's
what happened when manufacturers marketed central-line catheters
coated with silver or other antimicrobials; they cost a third more,
and reduced infections only slightly.and hospitals have spent tens of
millions of dollars on them. But, with the checklist, what we have is
Peter Pronovost trying to see if maybe, in the next year or two,
hospitals in Rhode Island and New Jersey will give his idea a try.
Pronovost remains, in a way, an odd bird in medical research. He does
not have the multimillion-dollar grants that his colleagues in bench
science have. He has no swarm of doctoral students and lab animals.
He's focused on work that is not normally considered a significant
contribution in academic medicine. As a result, few other researchers
are venturing to extend his achievements. Yet his work has already
saved more lives than that of any laboratory scientist in the past
decade.
Furthermore, often times new discoveries are based on very simple
incremental improvements to existing ideas. For this reason established
scientists find it very easy to ridicule them. They claim how can
something so simple replace decades of their research and experience?
Society agrees with them since they are conditioned into believing all
science is beyond their understanding. Since it is society that ultimately
funds scientist work, then without the support of society, the new ideas
will never take ground and flourish.
Cause and effect not established
The media often publish health studies that do not demonstrate cause and effect. The typical study goes something like this:
Researchers found that those who:
------ exercised regularly
--or--
------ ate fruits and vegetables throughout life were less likely to have contracted xyz disease than those who did not.
One can not make such type of conclusions without first proving there
is a relationship between the variables. The conclusions are as absurd as
saying "people who drink soda are more likely to contract xyz disease".
While the statement may be true for the sample population , it does not
establish there is any relationship between soda and xyz disease.
The problem is that those who drink a lot of soda
probably do or eat other things that are unhealthy, relative to the
other group, so any effect noted may be due to the other things.
Using the health study approach, one could argue that bridges made of steel are less likely to break
compared to those made of wood. Why? Because historically and statistically steel bridges have a better track record than
wooden bridges. However, any decent bridge engineer knows they can build a wooden bridge that is ten times stronger than a
steel bridge made of thin and undersized beam sections! This is because they understand the relationship between loads,
forces, stresses, and material properties. Detailed mathematics , physics and chemistry can explain what the
relationships are.
However, in medicine it seems the health studies often have not identified the association
between exercise and heart disease or smoking and lung cancer. They seem content with simply observing that some
association may exist which is just as primitive as saying "all bridges made of steel are less likely to break than
wooden bridges".
Any theory has at its foundation statements which are accepted as true
without being proven. One cannot build a theory starting with nothing.
Other sciences seem to be able to dig a lot deeper into the foundations than
medicine. One reason is because medicine is perhaps the most complex of
all the sciences as it merges elements of all of them. Combine that with billions of years of
evolution and you end up with an organism so complex that no simple theory
can explain it.
Inability to recognize variability in data
If you recall, the scientific method requires studying how interacting conditions define
a situation. The problem is there is variability with each condition. For example,
consider a scientist who is studying what causes trees to grow tall. One condition is
sunlight. However, sunlight is not a constant. Some years can have more sunlight than others depending
on the overall weather conditions. Therefore, sunlight has variability that can effect how much taller trees
grow one year relative to other years. It
has a tolerance, i.e +/- some amount every year.
This variability of conditions leads to uncertainty in the conclusions
of a scientific theory. When you add up all the variability from other
interacting conditions, the final result can have almost no meaning.
So you have to always question the variability of the conditions and ask
the scientist what cumulative effect they have on the uncertainty of their
results.
A good scientific experiment must be repeatable. The experiment has to
be able to be performed many times and always produce the same result. If
repeatability has not been established then you must scrutinize the
variability of each condition.
Scientific community seeks maintenance of status quo by rejecting new
ideas
There is a famous saying by former US President, Woodrow Wilson,
" If you want to make enemies, try to change something ". Human nature does
not like change, however, there are good and bad reasons for this.
Anytime you have an established convention, the one who comes along and
says that it can be done differently, better, faster; that person is seen as the renegade.
The scientific community, like any other community, has
an unwillingness to accept new ideas often for no other reason
than maintaining
the status quo.
There are often valid reasons for rejecting change and mantaining
status quo. It
is not the cost of the new technology, it is the cost of the process of
changing to it. Consider something like the complex and expensive black box on
an airplane. In today's digital age it would be easy and inexpensive to
add a backup system to transmit flight audio and cockpit
video via satellite phone and internet technology.
However, once something like the "black box" is in
use, it seldom changes unless there is a compelling need. To change it,
you have to have meetings, studies, vendor contracts, more studies, more
meetings, notices, hearings, training on replacement, etc.
It is the same reason why the on-board Space Shuttle computers are a couple
hundred times less powerful than the processors in an iPod: they work, and
there is no compelling reason to change them.
On the other hand, some scientists seek to mantain status quo
out of pure laziness.
Established scientists spend their lifetime
mastering a few core ideas. These ideas are their livelihood so
any challenge to these ideas directly impacts their livelihood and future.
Their job security depends on their ability to show society the value of
their work. If new ideas are discovered then society will no longer
perceive any value in their obsolete ideas.
Consequently, some scientists find
it easier to reject those with new discoveries rather than face the
humility of admitting the error in their obsolete ideas .
As you can see, there are both good and bad reasons for rejecting new ideas. Often
decisions to reject are based on combinations of reasons. A good scientist is one
who can prioritize the sensible from the nonsensical reasons!
Focusing on details while ignoring the bigger picture ( Phd's )
While one should have great respect for the work PhD's do,
nearly all of them suffer from the 'Ostrich mentality'. Their heads are stuck in the sand so
are unable to sense the train coming. Such behavior is fine in the research lab, but in the real world it creates unnecessary
red tape to get things done. PhD's are unable to see the big picture. Instead they focus on the tiny flaws of a system
to convince people that the entire system is flawed.
It is interesting to note the difference in perspective in the clinical scientist and the engineer. Scientists are
interested in 'exactness'; in the minutiae of detail. Engineers are more interested in systems and the dynamics of how
things actually work.
Clinicians will always argue they are more accurate while failing to understand the problems at hand in the real world.
In conclusion, the world is not only made of one color.
Stay open-minded and appreciate all the interacting
diversity surrounding you. The best way to avoid the trap of believing bad science is to read from multiple sources to
identify common elements and conclusions.
When numerous people you respect and trust say the same thing,
then you can be more sure you are reading mostly good science.
Keep in mind the real world is not black and white at all. Between pure science and
the incorrect interpretation of data, there is a huge gray area, and this is
where most research is done.
Unfortunately, as Thomas Edison said, " Opportunity is missed by most
people because it is dressed in overalls and looks like work ". This leads
to another famous saying, " The less one understands something, the
greater one's faith in its capabilities ". It is unfortunate that human
nature prefers ignorance and blind faith over hard-work and skepticism.
On the other hand, if you have something worthwhile, people will come. But you better be
prepared to show proof. If you have an idea for a machine that flies
around with no physical prototype, nobody will care, you can bet
on it. Talk is cheap. Demonstrate your machine and the world will beat a
path to your door, regardless of what the scientific community has to say.
To end on a humorous note, the following table translates commonly
used expressions found in research papers:
Published | Translation |
" It has long been known...."
| I didn't look up the original reference. |
" Of great theoretical and practical importance...." |
Interesting to me. |
" A definite trend is evident... |
The data seem practically meaningless |
" Three of the examples were chosen for detailed study.." |
The others made no sense. |
" Typical results are shown..
| The best results are shown. |
" It is believed that..
| I think |
" It is generally believed that....
| A couple of other guys think so too. |
" It is clear that much additional work will be required before a
complete understanding of these phenomena is possible..
| I don't understand it. |
" Correct within an order of magnitude.."
| Wrong |
" Statistically oriented projection of the findings..
| Wild guess. |
" Highly significant area for exploratory study.."
| A totally useless topic suggested by my committee. |
Source: Unknown
Next section ->
Section 3.3 - Dimensions