The title and the article is nothing new, we have known for some time that computer based neurocognitive testing is not perfect. Some have even concluded that these type tests are not worth the time and money. More and more I am hearing from professionals that feel that they HAVE to give this tests or others like it, even though it’s not precise enough in cases.
What is interesting to me is the website it appeared on, NFL.com, the one league that has propelled this product more than any other.
The timing of this article is also curious to me as well. Lately I have been getting many emails regarding computer based testing and the need for it and my overall thoughts on them. Perhaps there is a wave of skepticism regarding this tool, which is nothing new, or perhaps there have been more and more issues with these tests, now that they are being used by so many.
There is no doubt there has been an impetus upon baseline testing; the most time efficient is the computer based model. However if you look at the most recent Consensus Concussion Statement there it is rife with confusion about this as well;
Baseline NP testing was considered by the panel and was not felt to be required as a mandatory aspect of every assessment; however, it may be helpful to add useful information to the overall interpretation of these tests. It also provides an additional educative opportunity for the physician to discuss the significance of this injury with the athlete. At present, there is insufficient evidence to recommend the widespread routine use of baseline neuropsychological testing.
Before that paragraph the Statement notes the use of cognitive assessment as a part of the recovery constellation and how it should also be used to get a better picture of the recovery of the injured. Nothing in the Consensus Statement mentions the use of computer based, mass testing platforms to help with concussion assessment or recovery. So why are we using them?
That is an awful good question.
To me the best picture of recovery, in my opinion, can be gained by assessment of symptoms, cognitive, balance, visual and well-being of the injured person. That last piece is the most important, again in my humble opinion, if the kid/person is “not right” then they are not right.
I have seen plenty of concussed individuals that have passed the “needed” exams for all the other portions, but are still not themselves. Worse, I have had parents and kids a like struggle and counsel with me about why Jimmy/Sally is not doing well in school after the injury when the doctors and tests are saying they are fine. Worser (not a word) yet, is the personality and social changes I have seen with kids after a concussion that is often overlooked or not even mentioned by a physician when working on a recovery timetable.
The expectation of those taking the tests, computer based neurocognitive in particular, is that if they pass the test they are good to go. This has led to many concussed and caregivers of concussed to be frustrated with recovery. I know it has been told to us over and over that these are not clearance devices but most places have policies that require one to be back to baseline before being able to return to play. Heck if you just sprain an ankle your neurocognitive tests can show a deficit. For many others return to baseline is possible but for some it just cannot happen; hence the need for the variety of assessments.
My thought regarding baselines and effectiveness are simple; if we don’t have a “gold standard” quite yet then why spend all the money to get unreliable results? Moreover, if there is no correlation between scores and severity/recovery time then why use them at all? They are a good tool but, simple paper and pencil tests can provide the picture necessary to get a grasp on recovery.
The overriding theme of this post and approach on concussion recovery is to use all the tools one has, sometimes the best tools are those closest/know the person to make a clinical judgement. We have yet to produce a test that can give us definitive measures of both severity of injury and of full recovery. Because of this we need to be conservative with the youth and make the most informed decisions possible when someone is returning.
The Concussion Blog 
The NFL is simply unhappy the Lovell blabbed in “League of Denial” that NFL research was bogus.
The NFL still clears players using ImPact because NFL ImPact NPs can always interpret the data so a player can be returned to play. Coach needs the meat on the field.
While ImPact is a sorry instrument, NP tests, both computer and paper, are extremely useful when used correctly. Unfortunately, ImPact encouraged there use in less than an ethical manner.
Nothing can replace a clinician who knows the subject. Contact sports demand an ATC. If a subject is symptomatic, then no NP test is necessary. However, complex cases, many of which you see, usually demand NP screening and testing. Barth demonstrated the utility of baselines more than two decades ago. It is especially useful to have baselines when the subjects brain is changing rapidly, i.e., high school.
Blame Lovell, Collins, Maroon and UPMC; they have polluted a pool that should have been a tremendous resource to both clinical and research efforts.
Below is a post that I previously made on this site. The contents continues to be pertinent to the issues raised in today’s blog.
============================================
My graduate training and related applied clinical experiences emphasized that an adequate neuropsychological assessment requires the utilization of a comprehensive battery of reliable and valid tests.
Unfortunately when computerized screenings are discussed re brain injuries, many lay persons and professionals are not aware that:
1- The computerized test such as ImPact is a screening device and NOT a thorough neuropsychological assessment.
2- A review of professionally/peer reviewed and published literature ( by authors such as Dr. L. Mayers & Dr. T. Redick) have raised significant concerns re the reliability and validity of such computerized screenings. (see below citation & Abstract)
J Clin Exp Neuropsychol. 2012;34(3):235-42. doi: 10.1080/13803395.2011.630655. Epub 2011 Dec 13.
Clinical utility of ImPACT assessment for postconcussion return-to-play counseling: psychometric issues.
Mayers LB, Redick TS.
Source
Athletics Department, Division of Sports Medicine, Pace University, 861 Bedford Road, Pleasantville, NY 10570, USA. lmayers@pace.edu
Abstract
Computerized neuropsychological testing is commonly utilized in the management of sport-related concussion. In particular, the Immediate Postconcussion Assessment and Cognitive Testing 2.0 program (ImPACT) is widely used to assess the cognitive functioning of athletes before and after a concussion. We review the evidence for the clinical utility of this program in terms of validity, reliability, and use in return-to-play decisions. We conclude that the empirical evidence does not support the use of ImPACT testing for determining the time of postconcussion return to play
3- Computerized screenings are neither sensitive to all the subtleties of a brain injury/concussion nor are they comprehensive in nature…that is, the SCREENINGS are not thoroughly assessing the 3 major brain functions: cognitive, emotional and physical.
4- My PhD in the area of Clinical Psychology was completed during 2004. The below 4 quotes (i.e.; A, B, C, & D) are citations from my 2004 Dissertation entitled:
A Preliminary Investigation of Active and Retired NFL Players’ Knowledge of Concussions
A) “In an extensive review of research pertaining to mild brain injury, Reitan & Wolfson (2000) strongly cautioned that many researchers who have examined mild brain injury have typically not employed comprehensive neuropsychological test batteries in their methodology. The authors perceive this as a major flaw in evaluating the impact of mild brain injury, because they firmly believe that standard neuropsychological test batteries have frequently been shown to be sensitive to both focal and diffuse brain damage. They also pointed out that researchers typically have limited their assessment and focus due to a premature presumption that neuropsychological impairment is limited to a rather narrow range of deficits, and that a restricted range of tests is all that is required for adequate neuropsychological assessment. Our clinical experience suggests that mild brain injury produces diversified and even widespread neuropsychological losses in some patients (p.97).”
B) “The medical evaluation component of a brain injury assessment is also highly suspect since typical medical assessment methods (i.e., neurology exam, EEG, brain-imaging techniques) utilized in determining adverse effects of mild brain injury are rather gross and insensitive, and thus false conclusions may be derived which report no neurological deficits were sustained (p.43).
C) A similar perspective pertaining to false conclusions was shared by McClelland (1996) when he cautioned, “…the absence of evidence is not evidence of absence” (p. 566). This view was reiterated by Gronwall (1991), who stated, “…it is impossible to prove the null hypothesis. Failure to show a deficit does not prove that no deficit exists, and the neuropsychologist has a responsibility to make sure that appropriate tests for assessment are selected” (p. 257).
D) Damasio (1994), during a case presentation in which he discussed a particular patient, also noted problems with tests not being sensitive enough to adequately measure brain dysfunction/ impairment; he declared that “a problem here lies with the test, not with the patients. The tests simply do not address properly the particular functions compromised and thus fail to measure any neurocognitive decline” (p. 41). In the same vein, through the devising of more sensitive and precise tests designed to further scrutinize concussions and the postconcussion syndrome, it is highly probable that an organic basis will be uncovered (Robertson, 1988).
As I read and reflect upon perspectives shared…. I believe sincere individuals would benefit from becoming familiar with the contents of my 2004 Dissertation…which includes a snapshot of the evolution of concussion/brain injury knowledge that has existed in the professional literature for a significant period of time.
I agree that computerized testing is often inappropriately used and unfortunately relied upon as the “standard”, not just a tool.
However, I do use it in clinic. I’d love to have every kid evaluated by a neuropsychologist, but let’s face it, that’s impossible — I work with 4 of them that are very experienced in concussion. They can see 3-4 patients in a day for a truncated evaluation. If I saw that few, I would be out of business. So computerized testing can give insight to the population that those NPs cannot see. But I believe it can “catch” a teenager that is saying he is asymptomatic, parents who agree and has a normal physical exam — and a “big game” coming up in a week. I’ve had kids and parents lie – I believe it’s an attempt to get them back on the field – and the only way I have caught them is by computerized testing. So is it that bad?
Some thoughts:
1- If an individual is to participate in a sport… part of the participation costs factored in should be a thorough neuropsych exam…
if the exam can’t be afforded …then avoid the “game”…as concussions / brain injuries are a highly predictable occurrence in various sporting activities…
and the brain’s various physical, cognitive and emotional functions…need to be adequately protected…
2- Some parents and athletes of all ages will lie re their actual quality of physical health.
As Dr. House often exclaims… ” People lie ”
3- Screenings are screenings…not thorough exams.
At my yearly physical…numerous physical functions are examined…
I would not want a screening conducted on only a small portion of my bodily functions…and informed based on these narrow findings, that all my bodily functions are fine…as many fase assumptions are possible re the quality of health.
Reitan & Wolfson have advocated for a comprehensive neuropsychological assessment for an mTBI as ANY COMPONENT OF THE BRAIN and brain function may be injured. Again what constitutes a comprehensive eval may vary among professionals.
Neurocognitve screenings are more limited in scope & may only evaluate suspected areas of possible brain function/dysfunction.
In general, and based on my training and experience, screenings are not as comprehensive as an assessment.
4- As you have noted…Screenings may detect problems…these same screenings may also miss identifying / not discover other injured functions…and thus people may falsely believe / conclude that all is well…when no symptoms are identified.
In-other- words: Not detecting an injury during a screening does not mean that an injury has not occurred…
since ” the absence of evidence is not evidence of absence ”
As noted above:
A similar perspective pertaining to false conclusions was shared by McClelland (1996) when he cautioned, “…the absence of evidence is not evidence of absence” (p. 566).
This view was reiterated by Gronwall (1991), who stated, “…it is impossible to prove the null hypothesis. Failure to show a deficit does not prove that no deficit exists, and the neuropsychologist [or other evaluator] has a responsibility to make sure that appropriate tests for assessment are selected” (p. 257).
Damasio (1994), during a case presentation in which he discussed a particular patient, also noted problems with tests not being sensitive enough to adequately measure brain dysfunction/ impairment;
Damasio declared that “a problem here lies with the test, not with the patients. The tests simply do not address properly the particular functions compromised and thus fail to measure any neurocognitive decline” (p. 41)
=================================================
5- Below is an expanded , yet still partial, list of some possible concussion / brain injury symptoms.
Unless all the below potential areas of injury, and others not listed, are adequately assessed by trained individuals, it clearly becomes apparent there is a deliberate or inadvertent “rush to judgement ” occurring within a very limited concussion / brain injury screening process….
and thus false conclusions may easily be made re the functional / dyfunctional staus of a vulnerable human brain.
Headaches
Muscle contraction type
Migraine
Occipital neuralgia
Secondary to neck injury
Secondary to temporomandibular joint syndrome
Due to scalp lacerations or local trauma
Mixed
Dizziness
Vertigo
Tinnitus
Blurry vision
Diplopia
Memory dysfunction
Impaired concentration
Personality change
Anxiety
Depression
Sleep disturbance
Decreased libido
Irritability
Noise and light sensitivity
Fatigue
Slowed information processing
(R. W. Evans, 1987)
================================
Headaches
Feelings of dizziness
Nausea
Noise sensitivity
Sleep disturbance
Fatigue
Being irritable
Feeling depressed
Feeling frustrated
Forgetfulness
Poor concentration
Taking longer to think
Blurred vision
Light sensitivity
Double vision
Restlessness
(King et al., 1995)
PS:
Possible concussion insight — with tongue-in-cheek
Though the ” historical ” dates may not be accurate —
the essence of the below content appears to be on target when exploring concussion / brain injury history:
=================
The first testicular guard, the “Cup,” was used in Hockey in 1874 and the first helmet was used in Hockey during 1974.
That means it only took 100 years for men to realize that their brain is also important.
Peace. (-:
NP eval factored into the cost of sport is a nice, but really unrealistic idea. Given your prior post, I assume you mean that computerized neurocognitive testing is not enough, therefore a true NP evaluation is indicated.
There’s no way that there are enough NPs in the country to handle that volume when you consider that evaluation would need to occur for an huge amount of athletes playing youth and high school sports (Fr, Soph, JV, Varsity levels).
Love the thoughts on this subject… As a tool these type tests are warranted… But this is sorely misunderstood by the masses…
To: Kids SRC Doc
Some further thoughts:
1- Children and athletes are being needlessly being placed ” in harms way ” if adequate brain injury protection via thorough assessments is not made available for each individual particpating in a sport …
If athletes can’t be provided adequate protection via a thorough assessment then why expose the individual to potential brain injury ? Accepting limited / mini-cognitive screenings findings as being accurate serves to place the athlete’s brain health in danger…..
2- Neurocognitve screenings are more limited in scope & may only evaluate very limited suspected areas of possible brain function / dysfunction.
Thus why are some profesionals creating an illusion that it is ” safe to return to play ” when the brain has not been thoroughly and adequately evaluated for brain injury…
Within the sport culture there is an obvious existing false premise that a NARROW cognitive screening is sufficient to assess cognitive brain injury. A screening IS NOT an assessment.
As previously shared… concerns have been voiced re the validity and reliabilty of the screening instrument…and the related conflict of interests that have emerged.
Furthermore, not detecting an injury during a screening does not mean that an injury has not occurred…since
” the absence of evidence is not evidence of absence ”
.
3- A comprehensive concussion / brain injury assessment goes well beyond the assessment skills of a trained neuropsychologist or clinical psychologist.
I believe an interdisciplinary team of competent concussion / brain injury professionals should be involved in the assessment process… anything short of this creates and fosters an illusion of safety and that an adequate assessment has been conducted.
The need for a competent interdisciplinary team is based on the fact that any areas of the brain may become injured…brain areas that adversely impact cognition, emotions / social behaviors and physical fuctioning as noted in the various above listed symptoms found in my earlier post.
If you review and reflect upon some of the possible brain injury symptoms…a valid assessment must address ALL these possible injuries in order to rule out / rule in the presence of an injury.
Sadly…from what I have seen and heard…this typically does not occur for the suffering athlete.
There are no short cuts to a valid and reliable assessment.
More simply stated…
If an auto’s brakes are the only thing / component being evaluated for a purchase or at a yearly state inspection, and the entire auto is deemed to be safe to drive or pass inspection…how safe would you feel drivng this auto?