Thanks to an alert reader an article was sent to the inbox dealing with neuropsychological testing, specifically the ImPACT test. The author of the article published on Slate, Christine Aschwanden, provides a very well written reason as to why some are starting to take a much harder look on these types of tests;
On closer inspection, though, the whole thing begins to fall apart. “It’s a huge scam,” says physician Robert Sallis, past president of the American College of Sports Medicine. “They’ve done incredible marketing, and they’ve managed to establish this test as the standard of care with no evidence that it has any benefit.”
Dr. Sallis is not the only one, in fact we receive numerous questions about all the NP tests, and recently we have seen more and more research regarding the downfalls of this type of testing. Research like this one (sourced in the article) written by physical therapists and a physician are being done by independent parties and seem to reveal there is a problem. According to one of the developers of ImPACT some of the information is not viable if only because of the particular journal it appeared in;
All that aside, if you’re to trust the numbers, the ImPACT test would need to produce the same scores on a given kid each time he or she takes it in an unimpaired state. That doesn’t always happen. In one independent study, 118 healthy student volunteers took a baseline ImPACT test and then returned to retake the test twice more, 45 and 50 days later. In the follow-ups, more than one-third of the concussion-free participants showed up as false positives, which made it seem as if they really had the symptoms of a concussion and were maybe lying about the symptoms. Lovell points to the fact that this study was published in the second-tier Journal of Athletic Training, rather than a more respected neurology journal.
If the reporting is accurate, Lovell basically slams the people who are most often the ones having to deal with this instrument in the concussion “tool box”. Not only that, the elitist attitude that only “real research” is published in respected neurology journals. Sound research is repeatable, and all indications have shown that even the JoAT wouldn’t put trash in their journal, in fact a lot of the original research on concussions were found in JoAT. Wait there is more…
In 2008 research authored by none other than Lovell was published in Journal of Athletic Training. But wait there is more…
If you just simply “Google Scholar” Lovell+Journal of Athletic Training you will find that there are many hits, but this can be expected because of their target audience in the concussion battle, athletic trainers. What is more, there are MANY of his published articles that cite other research from the JoAT. Seems odd that he would slam the JoAT all the while using work from the journal and even publishing articles in it.
OK, my diatribe on the JoAT and Lovell’s apparent slight is over, back to the issue at hand.
ImPACT and others have an issue with testing reliability, it should be buyer beware. What really needs to happen is to have access to athletic trainers when there is problem surfacing about concussions. These properly trained individuals are the first line of defense, issuing a computer based test to “protect” you in case of problems will only go so far.
FWIW, I agree with the article about how ImPACT specifically has capitalized on the great marketing of their product. I also believe that tests like this are important to the whole picture and as they and other NP test advertise they should not be used solely for return to play decisions. However if money is an issue, the free SCAT2 is always an option for a baseline and follow-up tests.
The Concussion Blog 
Dustin:
You bring up some good points. Regarding the Journal of Athletic Training and it’s quality: People sometimes think of journals in tiers but it’s an antiquated and biased method of evaluating a journal. As you point out, if you want your research to end up in the hands of the clinicians most likely to use it in the United States then you want to publish in the Journal of Athletic Training. These tiers are often based on impact factors, which can be biased by the number of times articles in a journal gets cited. The problem is that when you get into smaller disciplines there’s less people available to cite the work.
I also agree with you that in regards to the performance of neuropsychological testing we should remember that the NATA Position Statement for concussions suggests that NP testing should only be one aspect of monitoring concussions: “In addition to a thorough clinical evaluation, formal cognitive and postural-stability testing is recommended to assist in objectively determining injury severity and readiness to return to play (RTP). No one test should be used solely to determine recovery or RTP, as concussion presents in many different way.” There’s a lot of potential issues to NP testing but if we are aware of them we can keep them in mind while using the data and can try to further improve testing methods.
Ryan Tierney and Jane McDevitt, two concussion researchers at Temple University, have posted about several articles that get into the performance of NP testing and the importance of performing the testing certain ways (available at sportsmedres.org).
I’ve been questioning this setup for a couple of years now. Good job getting it out there, Dustin. It’s a matter of conflict of interest. I don’t thing it’s totally wrong for researchers to have sources of income or business on the side, but in this case the great demand for a quick and efficient baseline/post-injury test may have rushed its clinical use.
It is possible that the explosion in popularity of ImPACT has clouded the judgement of its producers. Again, I don’t see any issue with producing a product from your research, but it must be properly vetted (in terms of validity) before you can put it out there.
Dustin:
The Slate article is a hatchet job perpetrated largely by ONE neuropsychologist with a bad case of green-eyed monster. ImPACT was developed as a clinical tool by Lovell and his colleagues for them to use with the teams/athletes they were serving — something that would be as accurate but far less time-consuming than paper and pencil testing. Only after they validated it did they realize — and pursue — its commercial possibilites.
Remember, ImPACT is a tool, not an absolute arbiter. Anybody who uses it as the latter is a fool. Nowhere, in any of the literature that they’ve published, do Lovell and colleagues advocate using ImPACT exclusively in return-to-play decisions. In fact, they specifically caution against doing so.
I’m not sure what Dr. Lovell said about the JAT. There is no direct quote in the Slate article; he is only paraphrased. Anything that comes out of Slate is severely Slanted. So consider the source. (more to follow)
Anybody who questions the validity/reliability of ImPACT ignores the recent work of Leverenz and colleagues at Purdue who found — much to their surprise — that ImPACT results correlated perfectly with their own studies done with fMRI.
The inference in the Slate article, that SCAT 2 is somehow an inexpensive alternative to a validated computerized test, is laughable. SCAT 2 is an excellent sideline tool but is not practical for a single high school athletic trainer to use as a baseline test for multiple athletes.
John,
Was Leverenz study conducted in the manner consistent with the manner in which Impact is normally conducted or in a lab setting? What was the severity of the injury examined. What was the research question asked, n-size, power, effect-size and p-value of the results. I find it odd given Lovell et al. received 2MM from NIH (2002-2007) to concurrently validate ImPact with fMRI and failed to publish results.
Second, well conducted studies have clearly demonstrated that ImPact is not a very robust instrument. While superior to ANAM , it is inferior to both Axon/Cogsport and Headminder.
Certainly Headminder has maintained a consistent stance on how NP instruments should be used and its utility with in responsible program to elevate concussion.
Impact has suffered from well documented issues of overselling itself including but not limited to you can baseline at home, if you clear ImPact you are good to go, you don’t need a baseline.
Lovell has a lot to answer for.
There is certainly a place for the NP testing here. However, brain function on a electrical and chemical level must be appreciated. The initial trauma causes changes in neuronal function that becomes further compromised by the inflammatory cascade that follows. Depending on many physiological factors pre-injury, the sequence of events in the brain can take many different roads. Cognition is the highest of brain functions. To make final judgements of brain functionality after injury solely on NP testing of cognition is oversimplification and potentially dangerous.
Optokinetic Testing (eye movements) has been shown to be a sensitive marker to assess potential injury post-concussion. In my opinion, Videonystagmography (a non-invasive computerized assessment of eye movements) should be used as a baseline test on all athletes and any head injury during a game should mean that the athlete should not play until their VNG test was repeated to look for problems.
We use this technology as an objective’ way to safeguard our athletes. It also can guide our rehab plans!
– Dr. Mike
I concur with Dr. G that concussion management using Videonystagmography is a beneficial and objective component to measuring the effects of concussion on the brain and particularly the vestibular system. We see less deficits with VNG and more abnormalities with dynamic vision (postural stability of the eyes) when doing computerized Dynamic Visual Acuity (DVA) and Gaze Stabilization Test (GST) designed by the NeuroCom group throught their InVision Programs
The Lovell comment about the JoAT aside (if it was truly his comment), I think we need to be careful with how we react to articles such as these.
Anyone familiar with current concussion research knows that no one method of evaluation should be solely used when evaluating the injury. Does that make ImPACT invalid? Does it make it any more invalid when compared to SCAT2?
Looking at the Slate article, one quote about “Two more studies, published last month examined the usefulness of ImPACT and concluded that it has very little practical value.” The little practical value links you to the research you link to in your post. The study also addresses the PCSS (Post-Concussion Symptom Scale) and the BESS. It reaches the same conclusion about all three of those tests (By the way the study even lists ImPACT as being stable and reliable). Should we stop using those as well? It also appears that none of the individuals in that study had baselines in any of those test pre-concussion.
Another “quote” in the Slate article about ImPACT reliability being “unacceptably low” no longer has a valid hyper link to even see which researcher made that comment.
The issue here should be that we are taking as many steps necessary to protect people from further injury. That means we should be using as many “tools” as possible to do so.
Dustin, the way you highlight the article above is written in a way that makes it seem people are better off NOT using ImPACT at all. I agree that ImPACT should not be the sole determination of return to play, but I have not seen the research say it is significantly less effective than other similar NP tests (SCAT2) for example. The quote you have from the above article even seems to show it’s overly sensitive meaning it might keep people out longer rather than sooner. (Never mind that study resulted from a 100 cohort group with no baselines, no inclusion of anyone with ADD, learning dissabilities, depression or other ailments, and included individuals who might not have returned for testing to actually be returned to play.)
Having a large portion of athlete’s baselined (properly) on ImPACT is far better than what it was in the past, which was a completely reactionary environment with no idea if someone was even close to “normal”.
I ask you this question. Would it be more beneficial to have a recent ImPACT baseline test for use following a concussion or no baseline test and simply relying on a post injury SCAT2?
That’s how you’re post above is written. If that’s not how you meant it then you should clarify because your words are becoming popular and you don’t want to give people the wrong impression.
Bryan & John, well everyone…
First great comments by all, it is a very worth while discussion…
Second, if you have read me enough you will know I use ImPACT however the meaning of the test has changed in my opinion.
Third, the intention was not to dissuade anyone from choosing any instrument, rather promote thought and discussion, there are positives and it is capitalized on by the company (rightfully so), but there is always the other side of the coin and the “buyer beware” we must be cognizant of…
Fourth, I have a ranking of NP tests that I prefer I have shared them before, but they are only opinions…
Fifth, there is a better way, IMO; to make it clear that a trained medical professional (see athletic trainer or other properly trained individual) should be present to make these decisions, not a coach or AD that looks at results. It happens, and really the results even sent to a MD with some knowledge has no idea how to evaluate the results, really that falls on neuropsych’s.
Sixth, I feel one of the most important parts of the SCAT2 is the balance assessment, if a true baseline has been used… The BESS is very good, quick, but it is very hard to “fake/lie” during that portion of the test.
Seventh, I probably would say that we are advancing in the right direction, I merely placed this article, like others in to cyberspace to inform, and yes throw my opinion in on it…
Again thanks for reading and commenting, it is truly what is needed!!!
Dr. Mike, VNG might be great, but how many athletes a year do you think we can baseline with it so it can be used post injury?
Dustin,
I want to first say I enjoy the blog as there is some very useful info that addresses all facets of the concussion issue. I stop and read here every day.
I first want to start by posing the simple question, “What is the alternative?” By this I mean what is the alternative to concussion management, specifically at the high school level, these days. I have had experiences as a certified athletic trainer at the professional, collegiate, and now high school levels. I can specifically remember a time in my early career that this injury scared the ever living hell out of me because I was being asked to make a recommendation with nothing more than subjective information from the athlete and a potential injury that I could lot physically put my hands on to evaluate. This is my personal experience and it forced me to take a proactive approach to educating myself to the infinite degree so that I may make the best possible informed decision regarding concussion for all parties involved, including myself. I have had experience with Headminder and ImPACT over the last 10 years. If we go back to the basics, these tools were developed to simply help aid in “taking out the guesswork” when dealing with sport-related concussions. It has never been the end-all to concussion management. When it comes to evaluating cognitive human behavior (which is what these tests are doing) there will always be a grey area. This is where our fundamental responsibility and common sense as human “evaluators” must prevail.
It seems as thought there are many contributors who are hammering ImPACT and Mark Lovell in particular because of marketing (financial) strategies. That is simply their opinions and I respect that. I personally think that I wish I would have thought of it first (ImPACT that is). I believe there are other “products” that have hit the market recently that are by far less beneficial to the management of this injury and can be contributed to a “market gain” (financially) more than ImPACT can. For example, the impact indicator. I want to go on record here saying that I have no conflict of interest to any concussion product or service. This is simply, like others, my opinion.
As I continue to further educate myself and stay on top of the concussion managment issue going on today, I will say that without ImPACT (or any other NP test for that matter) we are right back to square one in “hoping that we make the right guess and no one dies or becomes disabled” because of a wrong decision. ImPACT personally has helped me, and subsequently my athletes (along with the multi-disciplined approach of course), make better decisions regarding return to play and to the classroom. I have had some resistance to the test (mainly because by human nature we are resistant to change and lack of education on the topic), but the overwhelming majority of coaches and parents are thankful that there is “something” out there to help guide us all in the right direction after a sport-related concussion.
I close again with the simple question, “What is the alternative?”
Thanks,
Tommy
Tommy,
Thanks for visiting, I appreciate it… I hope that I truly provide some insight…
I have said many times I think there is a place for NP tests (look at comment above), however if handled incorrectly and misused it is more dangerous than “making a guess”, using the false hope that a computer test has cleared someone (hey MD’s do it to)…
The alternative is simple in my mind, having an AT or other medical professional that knows the athletes, coaches and parents enough to know when something is just not right… I am willing to bet that you have made decisions based upon clinical evaluation only… Heck that even comes back to when do you re-ImPACT someone (another day)…
I am just merely pointing out that tools like these are handled well by well educated and informed medical professionals like you, but there is danger in just “getting” the program for a school and running with it…
I hope that makes sense…
I appreciate the comment!
Tommy,
I am hammering Mark Lovell and not solely for his financial strategies. ImPact was promoted for years as a green to go solution. It was utter nonesense. Test at home, who needs a baseline blah blah just make the sale – he is medical professional and was a university medical professor. Lovell is/was part of NFL concussion denial complex for 15 years which is unforgivable. Many professional athletes were badly misled and damaged becuase of the promotion of non-scientific pronouncemets regarding that filtered down to youth athletes (former pros have been forced to resort to litigation becuase no one would take responsibility for misleading these guys – the complaints are fascinating). The loss to hockey of Pronger, Crosby were under his care. Does he or Collins and Maroon take responsibility for putting them back on the ice? The patient shouldn’t be pulling themselves off the ice but that is what happended. He is either an idiot or unethical; he is not an idiot.
I invite you to review his NIH submissions that clearly evidence problems at the base of the product. He knew it and sold it as more than it was.
At some point, Drs Lovell, Maroon and Collins need to be held to account. Stating we are learning so much, well the boys seem to skipped the medical library from 1990 to 2009. Jordan, Cantu, Erlanger, Barth and Relkin all suggested that long-term consequences of multiple head injuries were demostratrable. In fact, Lovell, Maroon and Bailes suggested the same facts in an obscure 1998 they authored. It just was not convenient to discuss it. Cantu and Jordan suggested more than a decade ago subcussive injuries may be the real bear in the room. Nonetheless, UPMCs marketing machine spins tails.
As far as what to do, use SCAT2 on the sideline. Use a NP test, properly. Use one that is built on science not marketing. Execute a protocol consistent with the lastest concensus.
On other hand, it should make one nervous when you buy a product from someone who sense one thing and knows another.
My son has failed the impact test 3 times. He has no concussion symptoms at all , He will not be able to play in a game now where scouts and college coaches were coming to watch him play . The word is at school that kids now want to “throw” their baseline they take in 9th grade, This test is being used solely to evaluate my sons return to play. I am thankful for some test but would like to see it in combination with other tests , This has become very frustrating
Joe:
NOT a fan of home testing. Will get you details of Leverenz work when time allows.
John Doherty
John,
Is the the article Functionally-Detected Cognitive Impairment in High School Football Players Without Clinically-Diagnosed Concussion?
It only shows an n size of 23, ImPact seems that its scoring index was corrected to cope with test-retest reliability issues in two domains. There are numerous other deificincies in the study.
These data clearly do not establish the validity or concurrent validity of ImPact. In fact, I accessed Irv Muchnick site and read through the UPMCs NIH submissions on ImPact abd Lovell et al. clearly admit the product has severe deficencies.
As I stated in the early thread, it is odd after spending more than $2MM in taxpayer money claiming to cross validate and instrument with imaging that no publication or data was ever presented. Furthermore, it is unclear whether Lovell et al. disclosured their ownership of ImPact or fudged the fact to fleece the taxpayer, you and me.
Dustin,
I agree 100% with you and think we are certainly on the same page. My colleague and I (another certified athletic trainer) had a vision many years ago to implement a program in our area. If you can guess, the timing for such a program was not yet appropriate because our community was not yet ready for such a drastic change in concussion management all at once. We sat back for quite a few years and gave folks “enough rope” to “hang themselves”, proverbially speaking. This was almost 8 years ago. As of June 28, 2011, Louisiana has passed legislation and the panic button was pushed. But have no fear, we were ready. Instead of being reactive, we simply chose to be proactive on this issue. When physicians and even other athletic trainers in the area were “stepping back” from this issue, we went into it full steam ahead. We were even successful in proving to the school systems how much liability were now put on them because of this law. Again, we went through the law with a fine tooth comb and put together a pretty comprehensive concussion management program (key word being management) that was in the best interest of the athlete, the school system, the parents, and everyone else involved. This program included ImPACT testing for ahtletes across 30 high schools in our area AND management of the injury by the certified athletic trainers that cover each of those high schools. This program also included consultation and referral from one of only TWO credentialed impact consultants in the state of Louisiana. ONLY TWO. We have set the bar high and were met with minimal resistance to this program. My job now as a certified athletic trainer is as the lead consultant for our program with direct communication to our CIC physician. I personally have been involved in every educational training program that ImPACT offers in regards to their software (workshops, webinars, and direct access to Lovell and Collins amongst other should any questions arise).
I believe the problem, to be quite honest, in our area is the “old school” mentality has yet to go away with some of our physicians. There are many physicians in our area that kids will try to sneak off to for clearance that have said, “No way! I’m not treating concussed athletes in my clinic. You will have to go somewhere else.” Then there are still those who take the subjective only approach and sign off on a kid they have seen only once. That’s were it gets pretty scary in my opinion.
In regards to the other software out there, I am not awrare that they even offer any sort of “training” for their product. Nor have I seen any research done on the validity, etc. of Headminder, ANAM, Axon, or the others. If you have any I would love to see it. This is what bothers me about all of the ImPACT comments. I have stacks of peer reviewed articles by others than Mark Lovell and Micky Collins (some authors that I know personally) on ImPACT and it’s validity and other topics. Is there some grey area to NP testing? Yes. But like I commented earlier, we are testing human behavior. There will ALWAYS be a grey area. The key is truly the multi-disciplinary approach to management of the injury along with common sense.
Should coaches and parents purchase a ImPACT software package, instruct the kids to read the directions and take the test at home? HELL NO!! That’s like taking a drug test at home without a chaperon. The big problem with that is (1) they are certainly not qualified or trained to administer the test and (2) once they print out that report and see all those numbers they will have no clue as to what they mean! At that point you do not have a true “management” program in place. This is where I agree with most of what the other contributors have wrote. I disagree simply because some of the comments are superficial. I am always careful of my sources and Slate magazine in my opinion is not a reliable source.
Are there flaws in the ImPACT software? Without a doubt. But it’s the same with any product that you buy or services you subscribe to these days. Look at your cell phone. They promise you a nice fancy smart phone and when you get it there’s a battery life problem or a software glitch, etc. But over time the bugs are worked out and researched and your needs are met. ImPACT is the same. There was a problem and they had the resources and vision to provide a solution to the problem. Love it or hate it, that’s what it is.
Again, Dustin I commend you on your tenacious dedication to this blog and this topic. I hope to keep the dialogue going.
Thanks,
Tommy
Tommy,
Great comments… We need to touch base, find out how to get more of “you” out there…
Joe,
I am not condoning what Lovell may or may not be doing. I don’t know because I don’t work in the same office or live in the same house. All I have to go off of is third party information like the rest of us. I would say that it seems that even if he was part of this “NFL denial complex” then every certified athletic trainer, physician, coach, player, and fan were part of this as well. We drive cars every day that are probably undeniably not as safe as the manufacturer claims them to be, right? But we still like the look of a shiny new car and depend on that vehicle to get us from point A to point B on a daily basis. I don’t see anyone refusing to drive because the car company’s “may be part of a sinister plan to make a whole lot of money and kill everyone.” The same could be said for oil companies. But we continue to put gas in our cars.
I guess my ultimate point is we don’t truly know what’s going on because of the reasons mentioned above. All I know is that I agree that if used and interpreted correctly, NP testing (specifically ImPACT) can give us an objective piece of information to make a better informed decision. I think the ultimate answer to my question yesterday is simply this. If we insist on our youth athletes playing contact/collision sports after a concussion then these test measures are the solution to today’s problem. The alternative is to simply discourage any youth athlete of continuing that sport, regardless of number of concussions or such, and quit cold turkey. I think society has some priorities mixed up and we will simply not be at that point anytime soon.
We still have to remember that it is indeed hard to manage something you “cannot see” with your own eyes. We rely on MRI’s to see inside a knee when we have a potential ACL injury. I don’t hear anybody complaining about the MRI machine or the manufacturer. There are potentially unnecessary diagnostics done every to the tune of millions of dollars a year. What’s the difference? It’s still an objective look before going under the knife.
Again, I hope we can continue this dialogue because I am certainly open to looking at both sides of the coin.
Tommy
Dustin and all;
I know I am a little late to the game, but I had to make some comments to this blog. First, let me say thanks to Dustin for hosting this site. I have been following it for a while, but this is my first response. All of the information on the site is great.
I have been involved with concussions for a while. First as an athlete, then as a coach, then an athletic trainer, and now as an MD. I can remember when the only “test” for a concussion was LOC. Basically if you were awake and could walk, you were good to go.
With our current knowledge of the short and long term dangers of concussion, it is becoming more and more important to identify, for lack of a better term, “sub-clinical” concussions before and athlete returns to play. I have been using NC testing for a while now (specifically Impact), and feel it is a vital tool in the assessment of concussion. However, it is only one part of our management and RTP criteria. I tell our trainers that I do not want to know the Impact results till an athlete is cleared clinically to RTP. We then run the NC test to see if any “sub-clinical” deficits remain. Unfortunately sometimes players are tested several times before they are clinically ready. It is then that we start to see “test fatigue”, or varying results and the test looses its value.
In addition to picking up the “sub-clinical” deficits, I believe that NC testing is invaluable to the medical staff when they are trying to explain to the athlete/coach/parent why an athlete is not ready to play. Having an objective number to show them goes a long way to support your decision.
A couple more comments and then I will be done. First, with regard to the Slate article, it is clearly slanted, and I am not sure why they chose that angle. What is clear is that she is not on the front line and has no idea about the nuances of this injury. Second, the Broglio article in the JAT is a very well done study from my perspective. I am not a statistician, but when I look at the raw data, the tests seem to have very close retest reliability. I may have to crunch the numbers (or maybe someone more skilled can do it for us) but I do not see how they got an ICC down in the .40 range.
Last, as a disclaimer, I have developed a new NC test (I will be looking for beta testers soon to help validate the test if anyone is interested, or still believes in NC testing). I hope that does not invalidate all that I have just said.
First, thanks for ollowing and the positive comments about te blog
Second, your experience warrants more commenting
Third, your points are all valid
Fourth, I am willing to try ou about anything
Dr. Benford,
Actually the Broglio article was seriously flawed. It was beyond Broglio’s math or his statisticians. There was a strong response made by a very serious biostastician that journal failed to publish. I suspect it was beyond the editorial committee’s math (Dustin you might want to ask them about it). From what I understood the article’s statistician did not disagree with the corrections. Nonetheless, after adjusting results with proper mathematics Impact has serious test/retest reliability although not nearly as bad as ANAM which was not part of the article. Cogsport(Axon) and Headminder performed much better with Headminder performing best. Cogsport and Headminder were obviously designed by serious scientists and put through proper validity studies which ImPact failed to execute. One should review the comments in the unpublished NHI fMRI/Impact concurrent validity studies (can be found on Irv Muchnick site) that demostrates that several domains have problems and one seems so poorly designed that it stands contrary to pencil and paper instruments.
I believe some of the very low reliable scores attributed to ImPact can be traced to some Navy studies attempting to identify blast injuries with a sports product. These are distinct injuries and one will generate numerous false positives with an instrument designed for sports injuries.
Finally my concern is with any NP product is how it is baselined and used in follow-up. ImPact liked to spin tales that had nothing to do with a sound protocol. While your center seems to be following a rigorous process many places do not for many reasons ranging from it is too timing consuming to people don’t believe sports concussion is serious or repeated head traumas (subconcussive as well – we just lack the instrumentality to measure that as this point) can have long-term psychiatric and neurological consquences (Thanks mostly to NFL propoganda). Concussion and subconcussion produce real brain damage, it just may not result in measurable or observable declines for decades. Resolution simply means the brain is functioning at normal or near normal levels because it is plastic and drawing upon cognitive reserves.
Good luck with the new instrument, I hope you took the time to test it rigorously.
Joe:
Leverenz’s original publication, which you accurately cite, reported on their first year of study. They are now 3 years into it, have expanded to a second high school — and girls soccer — and according to Leverenz and his colleagues, the numbers haven’t changed. More to be published soon, apparently. Will be happy to share with you the outline Leverenz has been distributing at his presentations. It is compelling.
Joe:
Thanks for the clairity on the Broglio study. I still like the methodolgy and concept, but I agree the stats are wrong. Do you have numbers for the Cogsport and Headminder results? I would also like to know how you know about all the “behind the scenes” info on this study.
John:
Do you have any prelim numbers for the follow up on the Leverenz study?
John,
I am curious how it is comparing populations as females often have different on average more severe responses and slower resolutions than males. See Broshek (2004?). I look forward to seeing the reliability because in the published study, it mentions, a correction undertaken by the clinicians. Therefore it would imply that, ImPact, plain vanilla, is inaccurate in at least two domains. So they are achieving stability through some sort of adjustment.
Dr. Benford,
I work principally as modeler (stats knowledgable but not a biostatistician) of complex medical and pharmaceutical research questions for private industry. How does one ask the right question, measure it and and statistically validate it? I detest sh*t research and further detest marketers and amateurs (much of sports medicine research) making a mess of scientific research.
Jake & Joe;
My email, published at the end of each of my weekly columns in The Times (Munster, IN, http://www.nwitimes.com) is ptatcsport@sbcglobal.net. Send me your email address and I will forward you a copy of the Leverenz outline.
So as everyone seems to keep going back and forth on studies and statistics and their validity, what should we do in regards to implementation?
Everyone agrees having a baseline for an athlete prior to a concussion (what ever method used) is almost a necessity now for management of return to play. What are Athletic Trainers at High Schools, where there is usually only 1 for several hundred athletes, supposed to do?
SCAT2 is the cheapest, but logistically for 1 person to do several hundred baselines themselves over the course of a school year is almost laughable. Even adding a few helpers the fact that it’s a 1 on 1 test is tough. Add in the fact it’s hard just to get athletes to show up for some sort of group computer baseline testing and the logistics are improbable.
Axon and Headminder might have better statistics, but cost difference is huge. Axon is 2-3 times more expensive and while Headminder is closer to ImPACT at the 300 test level, higher than that it becomes almost laughable as well.
With school budgets vanishing that leaves most with ImPACT as almost the only choice. Is ImPACT’s validity that bad? Most of the issues I’ve seen people bring up about it seem to be that repeated testing (as is the case post injury) can produce lower scores over time. From an injury management standpoint this would lead to athletes being held longer than needed (which in my opinion isn’t really a bad thing). There has been a whirlwind of ImPACT discussion lately so I could be wrong with that impression.
Yes a BMW(Axon) or Mercedes(Headminder) will get you somewhere pretty comfortably and with quite a few safety features. Unless you can tell me that ImPACT is the equivalent of the old Ford Pinto (i.e. the one who had a propensity to explode in a crash), then I’m going to go with the choice that is affordable rather than none at all.
Thoughts?
Brian I like where you are headed, I get it… But if you are going to spend ANY money on help with concussions then spend it on an athletic trainer. I think throwing money at the problem with NP tests when not administered correctly only creates problems.
I think you are overplaying the SCAT2 a bit there are ways to implement that and get 50-70 kids done is one day if you went hard at it… Also you could do this over a 2-3week period… If you make it manditory, part of the plan/policy then everyone will show… It might be more leg work, but it costs nothing but time to get it done… If you happen to have an iPad or iPhone or iPod Touch you can get the SCAT2 app and have it all stored for you, or email all the baselines to your MD…
Think about the yearly savings and getting the SAME exact results as the other NP tests, plus you add the balance assessment…
$300/year over 10 years is $3,000 (that is only if you baseline once, which should be done yearly)
more like $600/year you would be saving… Which in cash-strapped communities is a lot…
Just sayn’
Dustin and Bryan:
I agree with you both. I think the money should be spent on mandating Certified AT’s in the schools. I can also say there are some AT’s out there who still are riding the old school train around town and may not be the best solution to the problem. But, with that being said, I still thin that is the first step in getting all this going in the right direction.
I also agree that unless you know how to interpret the test you essentially have a useless $500 report. But regardless if you’re using Headminder, ImPACT, or any other platform of NP testing you are trying to quantify human behavior using algorithms and numbers. I think it’s impossible to get the EXACT same results every time no matter what you do. That’s were the INTERPRETATION becomes important. I think as human behaviors we all want immediate definitive answers and that’s just not always realistic. Especially when it comes to deciphering the human brain and it’s behavior patterns. We can get a lot damn closer these days than we could in the past with these tools in place.
Where I disagree somewhat is on the SCAT2 discussion. A few years a go I tried to SAC baseline test athletes at 2 of our 30 high schools and it was a complete nightmare. Maybe 250 athletes total. This year, myself and roughly 15 athletic trainers were able to achieve ImPACT testing roughly 3000 athletes in about a 6 week period, roughly 4 hours a day (give or take, and not on consecutive days). Although scheduling was a challenge and computer problems were inevitable, it was certainly a lot less of a hassle than trying to SCAT or SAC that many athletes. Plus, I find the SCAT2 (quite frankly) a bit difficult to understand and administer correctly. I (like I know some have) actually created my own version of the SCAT2 for use within our program. It of course it compiled of all tried and true components of the SCAT2 including the BESS and vestibular screening. I just made it more user friendly for myself and my staff.
Tommy
As an ATC I always think there should be an Athletic Trainer covering high school sports. That being said I don’t really follow your logic since $1,000 a year spent on NP tests wouldn’t buy ATC coverage instead. Even if that $1,000 was able to tip the scale to providing the coverage it would probably be limited enough that they wouldn’t have the time or logistics to implement the SCAT2.
I agree with Tommy that implementing SCAT2 comprehensively as a baseline for all contact sports at a larger school is downright insane. 1 ATC (which is what most high schools have) and 200 fall athletes is 30 hours of testing. How accurate are those baselines going to be after having done 50-70 in one day? To reference Joe’s comment below, SCAT2 is great for the sidelines, but if you haven’t baselined them with it then the score is hollow and has no context.
Currently the SCAT2 as a baseline is logistically prohibitive to complete for large numbers so the choice really comes down to either no baseline or ImPACT. As I said earlier, ImPACT may not be perfect, but until someone shows that it puts athletes at risk (i.e. indicates they are healthy when they are not) I’ll choose that over nothing.
I agree with Dustin. I stated it in a previous post, money should first be spent on a ATC trained in concussion management. If the schools don’t have the money for an ATC, it needs to think about dropping certain sports not limited to football (I am not an ATC so I have no dog in this fight). Next SCAT2, which is free, should be used both to screen athletes and cheerleaders and on the sideline. It is a blunt instrument but strong on symptoms. While it has not been validated, a quick review would indicate it has elements that are sound. RTP is more complicated especially in complex cases. It would serve most districts well to adopt a conservative policy if it does not have NP testing that is well organized and processed. While research conducted by Barth and Hovda suggest 3 to 10 days is normal time to resolution (in most cases [IABA would say 75 days – Omalu 90 days – based on what is unclear]), we are dealing with young brains who have little chance at playing at the professional level. Certainly all players should dump the code nonsense (they are not special forces operators) and pseudo toughness crap (They are Not NFL professionals [who still need education] but they are paid a great deal of money and top athletes with medical care) and tell the truth. Any symptoms, the kid is done until resolution.
If the school is banked or a college, it should spend the time and money to use a good NP test, under the correct conditions and execute proper protocols (more refined judgments can be made). A lot of excuses can be made that it takes time and money but it costs as much as a McDonald’s happy meal per athlete. On the other hand, it is time consuming to do it correctly.
Athletes will get hurt some will be permanently impaired and some will regrettably die. We can’t baby proof the world but we can be is responsible. As an ATC you protect the health, safety and well-being of athletes so just keep doing the job.
Joe,
Quick comment. I agree with your first sentence, “…. money should first be spent on a ATC trained in concussion management.” However, I think the common misconception by the general public is that AT’s are just now getting around to being “trained” in concussion management. I know you are not an AT, but I can tell you this is nothing “brand new” to the AT profession. We are, and have been, at the forefront of this injury for a very long time. Although there has not been a lot of experience with the NP testing until recently (I’d say realistically the last 15 years) AT’s are the “foot soldiers” of concussion management by default. I can tell you real life stories from the “trenches”. Everything from dealing with the athlete to the coach to the irate parent who thinks you (meaning the AT) are a quack for such suggestions regarding evaluation and management. We aren’t always the most popular people on campus and certainly (at least I) don’t strive to be. I’d rather be able to lay my head on the pillow at night knowing I’ve done what’s in the best interest of that athlete and everyone involved, regardless of public criticism.
I also agree that the testing is in fact the cost of a happy meal (plus some more depending on post injury follow up testing) and is by far less than the cell phone bill that these parents pay for their kids on a yearly basis. It’s all about PRIORITY! Let me repeat. IT’S ALL ABOUT PRIORITY!!! The money is not the issue. Don’t let anyone fool you.
The baseline time consumption (as well as post injury follow up) is definitely a challenge, but certainly doable.
Tommy
Tommy,
No question there have been ATs who have/are completely in the trenches and on the frontline for 15 years. Some who/do take it more seriously than parents, players and in the between 1994 and 2004 most docs.
There are also some holdovers and amateurs who just go along to get along. Don’t want offend a parent or a coach.
I am aware of ATs who have been threatened with lawsuits not so long ago for retiring kids or kepping a kid off the field. It is crazy. Some kid who weighs 5’8″, 160lbs and slow thinking he will go D1 so he has to play.
Fianlly, if SCAT2 is such a burden, I respect you have already taken a step by a first cut redesign, perhaps Justin could solicit comments so the package could be redesigned to become more usable. Unfortunately, the committee who designed failed to speask with the end user (ATCs). I am sure items could be jetisoned without losing its utility. In fact, by making it more firendly, it would have utility.
I agree Joe. Would love to share what I have with Dustin and get his opinion. I also agree with Bryan because the SCAT2 in my opinion is useless without a baseline. But perceptually it gives the “looker-on” a sense that the situation is being well handled on the sideline. Makes sense? Again my intent of recreating our version was to achieve the same thing perceptually over a few less minutes and still get some concrete evidence as to what is going on, IF there is any grey area in a sideline diagnosis.
Hope to continue the dialog with everyone.
Tommy,
I went back and downloaded a recent from of SCAT2 from a Canadian site.
Other than someone’s name and general details, I don’t understand the clinical need for a baseline. It is for use on the sideline. Elements like Symptom-scale, SAC, Glasgow, Maddox, BESS etc. are only useful post injury. The cognitive elements have extremely low ceilings – most people should be perfect. There are no cutoffs so change cannot be assessed at this point (RTP).
If is exhibiting symptoms or balance issues they should not be playing in the first place.
Am I missing something?
Joe,
An honest question. How many high school athletes have you evaluated on a sideline for concussion?
The reason I ask is that I know for a fact that there are definitely kids out there that won’t get a perfect on the SCAT2 the first time round, even with no history of concussion. I tried it out with several off-season baseball players who had just finished a lifting workout (figured it would simulate just having come out of a game). Had 6 kids go through the SCAT2 and none of them scored perfect. Some didn’t know the correct day or date (hell I’m not even sure of the date myself half the time). 1 out of 6 had a perfect BESS score, with most having between 5-8 errors. (Most were with the single leg balance portion as being baseball players it’s not a skill often performed in their sport). 1 out of 6 got the entire last string of numbers in reverse correct and 4 out of 6 got the months correct.
This was in a controlled setting away from people and with out a possible head injury. So I’m supposed to then administer this on a sideline with nothing to compare it to? While I’m sure many athletes might score a perfect 100, there will be others out there as well that would score lower than these 6.
If we’re only supposed to compare to a perfect score (as opposed to individual baseline), then the test really isn’t sensitive enough to deal with a wide range of population.
Bryan,
I only was speaking about the cognitive pieces should score perfectly. I am surprised normals would not know the day of the week or what half they are playing or who scored last (maybe these athletes have other issues such as an LD such as ADHD). It goes the same for repeating simple word lists. My experience is simple word lists have near perfect responses. Symptoms should be clear.
Bess has a threshold of five errors – so not 100.
Perhaps Zurich needs to do more work for a broader high school populations.
Bryan,
An added thought, if a child could not answer at a very high and consistent level (uninjured or any type) the questions presented in SCAT2, there is no way someone should show a valid score on the ImPact memory domain. It would effectively be a more than 4 sigma event that indicate sand bagging or some organic limitation such as ADHD, LD, or other limitation.
I respond to both previous comments because I think there are some valid comments. However, I can’t speak intelligently on some things you are saying Joe. I, like many other athletic trainers im sure are more of a “field researcher” rather than clinical like it very much seems you are. I do know this much though. I can speak on what I’m seeing as far as trends and such from the ground level.
I revised our sideline evaluation based on what works for our area. I added things that i thought were important and deleted (or modified) things I felt were not relevant to a sideline evaluation. This document is one page, front and back. I’m not saying it is clinically tried and true, but it works for us. It basically achieves what the SCAT does just condensed.
With that being said I truly believe that there are psychological factors that affect all of these test. I said it before. We are testing human cognitive behavior and we can’t expect it to be exact time after time. We are unpredictable at best. As evaluators we have to take all of these procedures for what they are worth and common sense again must prevail. These kids today don’t make much of a case for themselves because of all the ADD/ADHD and such. There are so many variables that don’t make managing this injury any easier as clinicians/AT’s.
I think there is plenty of evidence showing the validity of ImPACT vs. the alternative which is self reported symptoms predicating RTP.
I think we’re all on the same page, just maybe different sides of the page.
Joe,
Simple word lists were fine, the number sequences in reverse were a bit harder (mostly the longest string). Day of the week and Date are two totally different questions. I said some of them didn’t get the DATE right. You could ask 100 people on the street what the Date is and I’m guessing not 100 would get it right. I never said they didn’t get most things right, they just weren’t perfect.
The BESS portion of SCAT2 is 3 segments with 10 possible error points for each segment (30 points). Each deviation is 1 point. Where is it listed that BESS has a threshold of five errors? (I can’t even find that in the Zurich paper in the sideline evaluation section).
Your comment of “Symptoms should be clear” shows you have not asked today’s high school students to rate their symptoms. The “Headache”, “Drowsiness”, and “Anxious/Nervous” categories alone would be most likely answered at a 1 or a 2 even when perfectly healthy.
If you want me to just tell every athlete I evaluate for a concussion that they need to score perfect on the SCAT2 before they can play again, then fine. I’ll probably be keeping many who don’t have a concussion out of activity (which is fine with me because I don’t mind being overly cautious). With that being said please leave me your real name and number so that some parents may call you and yell at you directly.
Most sideline evaluations already have most components of the SCAT2, but with out a comparison what is it really telling you. Yes you look for “abnormalities”, which is what the scores in the SCAT2 indicate, but for the most part subtle deviations can’t be distinguished.
I think most tests in general need to broaden there work to look at younger populations.
Tommy and Bryan,
I am from the clinical world. For approaching a decade, I have been involved in dealing with complex cases. Cases where professional athletes are permanently impaired and need to retire he risks a catastrophic event. I know it is not easy listening to people or parents demand to injure themselves or children but parents and adults need to know. I have the dubious privilege of reading cases years after the last snap or high school game when a person can’t remember anything (forget a playbook), can’t hold a job, exhibit very negative social behaviors and may have become substance abusers. They wonder what happened. How did this bright former student athlete, this vital soul can’t operate in society.
The range of cases of former NFL, NHL and NCAA players will paint a very sad picture of individuals who were once heroes that are now invalids.
Rigorous protocols may be inconvenient but they save minds. Many years from now as research that is being conducted becomes common knowledge one’s conservatism will serve your conscience well.
It is not worth arguing over the science of mild-traumatic brain injury, CTE, or PCS. It will be dealt with in the scientific world as will the issues of NP instruments and sideline protocols. Data is simply data and it is becoming clear to the public, what the scientific community has largely known for approaching 20 years, that concussion, can have serious consequences, especially when it is poorly treated.
Great discussion going on here.
I think we all need to remember that any NC test is an adjunctive medical test. If there is not a medical profession available, who has evaluated and made a clinical impression, to interpret the test, it makes no sense to get the test. There is no NC test available that can replace a medical professional and their judgment.
I also think that if you are a single coverage AT at one or multiple schools, and you are spending an undue amount of time trying to get NC baseline tests on everyone, your valuable time might be better spent in the multiple other aspects of the job. If you have the resources and manpower, then use a NC test as an adjunct to your clinical impression and clinical RTP criteria. If you do not have an NC test, then just use a very through clinical RTP protocol.
Unfortunately we can not ask the athletes or coaches for much more help then we are getting. For them, this is often the most important thing in their lives. They have no perspective on the dangers or long term effects. They will do everything in their power to get back on the field/mat to participate again. That is why, at least at the high school level, every team needs a dedicated AT. With all the publicity that head injuries are getting, those state that do not have laws regarding AT’s for all games should be pushing to get those through.
Joe,
Maybe I’m missing the boat then. Again I pose my original question, “What is the alternative?” If we don’t want these kids to become the NFL players who are now “damaged goods” then how do we approach this issue at the high school level? If NP testing is not the answer, and the SCAT2 isn’t the answer, then what is? Please enlighten me because it seems all I’m hearing is “data” this and “data” that. What exactly do you call thousands of baseline test and hundreds (yes hundreds) of post injury test that we’ve done on our athletes since August of 2011? Where do you think your “clinical world” data and statistics come from? What I’m hearing is that we need AT’s who are knowledgable of this injury in high schools but that we can’t have any tools to fight this fight at the ground level which would make sense as to why we are seeing all of these complex cases in the “clinical world”. I see complex cases every day also. They’re disguised as teenagers who don’t have a grip on the world around them yet and parents and coaches who live vicariously through them and put there lives at a potential risk every time they buckle their chin straps. I hear every excuse their is as to why their kid still has a headache or why the test is so hard (it’s shapes and colors for Christ sake, not quantum physics!) and that there kid would never pass that test on a good day. Meanwhile, they have a baseline (comparing themselves to non other than themselves mind you) that they did excellent on!!
So again I may be confused, but what is the alternative to the current management approach and tools used?
Tommy
Tommy,
First I think SCAT2 has it place subject to better defining its use and making it more practical for trainers.
NP testing is a very useful tool especially on difficult and when athletes are not forthcoming.
Nonetheless, it starts with the process and protocol. With 3000 kids I can only imagine the strain you are under especially in the face of coaches, parents and athletes that don’t see concussion is a problem. The old shake it off which probably both of us experience in our youth (I was put back into a game after 5 min LOC event and we had the best doctors available at the time). Hopefully, CNN, OTL and other outlets will continue to educate the public to make your life easier.
What is coming. Biomarkers which may detect the presence of certain proteins associated with brain injury. Both Banyan and Cleveland Clinic are in trials. Banyan’s most current report seems to indicate many false positives but let’s see happens. Genetic testing to identify athletes most likely to suffer injury. NOCSAE has been supporting an effort and I am confident DoD is looking into it. It would remove high risk parties from contact sports. The Navy has produced a portable and relatively inexpensive instrument to detect hematomas.
Interventions are a mixed bag and require a lot of work. David Hovda at UCLA is doing very important work to understand the event process so we can figure out how to mitigate and eliminate long term damage. Drugs are being identified to mitigate memory loss (not a cure). A lot of effort is being put into the discovery of compounds enhance brain efficiency and neural repair.
We are at a difficult moment as we have enough knowledge to know there is a problem but insufficient knowledge to know how to fix it. Mountains of data are being analyzed to understand the mechanics of the injury and disease process by both academia and industry. I believe you will see rule changes, proved equipment and modifications in coaching that will reduce how many kids you will treat. Unfortunately, the medical side will take time (only a few really qualified Docs spent time looking at these matters like Barry Jordan and Bob Cantu) but more and better people are getting into the game.
Finally, moving this ball forward will take the engagement of ATs as you have ground level knowledge of the issues both medical and social that can confound the best laid plans.
Joe, could you tell me what company you are working for? I may need to consult when I start setting up my study.
Thanks
Why is 30 hours of SCAT2 baseline testing seen as so outrageous? We work at least 40 hours a week and if we are in a high school, many of us have summers off. Take a week before football starts and get the baselines done. 30 hours of work that lasts a year, maybe two. Suck it up and get it done.
agree