A week ago I returned from spending a few days in the USA attending the International Meeting for Autism Research – an annual academic conference, which took place in Baltimore this year. IMFAR, as it is known, is one the highlights of my year. An opportunity to get together with researchers from all disciplines who share a common interest in understanding autism and helping people who have a diagnosis, and their families. Last year I blogged about the focus which I feel some of this ‘help’ should have, inspired by conversations and experiences at IMFAR, and I thought I’d do a similar post this year.
This post will centre on the specific topic of biomarkers. To get us started, let’s have a think about what a biomarker is. This article attempts to provide a clear definition and opens by saying:
The term “biomarker”, a portmanteau of “biological marker”, refers to a broad subcategory of medical signs – that is, objective indications of medical state observed from outside the patient – which can be measured accurately and reproducibly. Medical signs stand in contrast to medical symptoms, which are limited to those indications of health or illness perceived by patients themselves.
As a quick translation for anyone put-off by the jargon there (the paper was not written for a general audience), this basically means biomarkers are signs linked to a specific condition which are can be measured by a doctor or scientist with little chance of making a mistake. Another feature is that measuring the same biomarker over and over should give pretty much the same result each time. These are distinct from ‘symptoms’ which rely on patient report – e.g. “my knee is hurting” – or professional interpretation, as in an IQ test. As the name implies, biomarkers tend to rely on biological samples. For example, for a genetic condition such as Fragile X syndrome, diagnosis is via a DNA blood test which is objective (not open to interpretation) and, if repeated, the result will come back the same each time.
What are biomarkers for?
Biomarkers provide reliable, concrete diagnosis which doesn’t rely on ‘shakier’ methods like clinical opinion (see below for more on this). In autism, the quest to find a biomarker largely focuses on a desire to be able to categorically state whether an individual has autism or not. This aim has a few sub-goals as well. First, if we had a biomarker for autism based on, for example, a blood test, this would make it easier to identify autism early in life which would allow young children and their families to get useful support right from the start (but see this post on ‘useful support’ and early diagnosis). Second, this method of diagnosis might be a lot cheaper, quicker and less stressful for families than the current array of clinical assessments. In particular, we might be able to reduce the number of people (especially women) who reach adulthood before discovering that undiagnosed autism explains many of the challenges they have been experiencing in their lives. Third, a collection of biomarkers might help us to define useful sub-types of autism. One of the few things I think almost everyone connected to autism can agree on is the wide variety among people with a diagnosis. This ‘heterogeneity’ within a single diagnostic label makes it hard to talk about autism, hard to provide useful support for autistic people and their families, and hard to do research. Being able to identify valid sub-types within the autism spectrum might help us gain better understanding, and provide targeted supports which match to specific needs.
So far, so good – the idea of finding an accurate, quick, cheap diagnosis, which can identify meaningful sub-types of autism, sounds pretty good. And sure enough, at IMFAR 2016 it felt ike 9 out of 10 talks I attended ended with “and this could be used to develop a biomarker for autism“. Biomarkers are clearly all the rage. So…
What’s the problem with biomarker research in autism?
Sensitivity / specificity : Developing a biomarker is really really really hard. It is also quite hard to explain why without taking ages about it, but I will do my best. People interested in picking apart one particularly egregious example of poor biomarker science ought to read this briliant post by Jon Brock. But for the time being, let’s imagine I have developed a new Biomarker for Autism Detector Test (or BAD Test) which uses the length of your little finger of your left hand as a measure. This test is based on the hypothesis that autistic adults all have left little fingers longer than 6cm*.
Senstivity is a measure of how well a specific test detects people who do have a particular condition. If I test 100 autistic people with my BAD Test will all 100 of them have long little fingers? Specificity is a measure of how well a specific test can rule out people without a condition. If I measure the little fingers of 100 people who don’t have autism, will every single one measure below the 6cm cut-off point? The answer to both of these questions is almost certainly No. Some autistic people will score below the 6cm cut-off, some non-autistic people will have long little fingers. So now we have to work out how many of these so-called false positives and false-negatives (people mis-diagnosed by the test in one way or another) are acceptable. 1%? 5%? 10%? What if I run my test on a group of 100 people with autism and only ‘miss’ seven of them – that’s pretty good, right? But then if I run it again on 1000 people, and miss 70 of them… That doesn’t sound so good. So I try another 1000 people and this time I miss 106 of them. Oh no! Did something go wrong with my test? No, this is probably because any one estimate of sensitivity (or specificity) is in itself unreliable. As always in research, running the same study repeatedly will yield slightly different results each time.
As I noted above, I would need a lot more time and word count to go into this issue in detail (and you can read more on the sensitivity and specificity aspect here). The broad point I’m trying to make is that collecting the data required to check the sensitivity and specificity of my BAD Test will take a while. It requires large groups of people, robust methods (e.g. a way of measuring finger length which is identical every time and can’t be biased by researchers seeing what they expect to see), complex statistics, and a strong a priori hypothesis. This last one means a clear statement of what we expect to happen, stated before we start the research study, and not just an interpretation of what we found which was made to fit the data after we collected and anlaysed it. Most of the so-called biomarker research I’ve seen at IMFAR and in journals clearly fails to meet these criteria.
Differental diagnosis: Another key feature of a biomarker is that it should be able to distinguish not just between people with autism and neurotypical people, but also between people with autism and (for example) people with social anxiety. Or people with attention deficit hyperactivty disorder (ADHD). Or people with developmental coordination disorder (DCD). One of the major flaws of current biomarker claims is that the evidence nearly always relies on comparison of groups of autistic people with neurotypical peers. In reality, young children (or adults) presenting at clinical services for a diagnostic assessment do so because their parents (or they themselves) are worried about something. In some way, they are different from whatever is considered ‘normal’. When evaluated by an expert to see whether an autism diagnosis is appropriate, they might be diagnosed with autism or they might get another diagnosis. For a biomarker to be truly useful it needs to be applicable to this situation of “differential diagnosis”. The question being asked in a clinic is not “Is this person autistic or neurotypical?” but “Which of the many available diagnostic categories best fits this person’s profile and needs?“. In nearly every case I’ve seen lately proposing a new biomarker for autism, the biomarker’s relevance to the second question is at best limited, but usually non-existent.
Better than a clinical opinion? We’ve already seen that one of the goals of biomarker research might be to streamline the diagnostic process – making it faster, more precise and easier for families. The idea of simply having a blood test for autism must sound appealing to those who have experienced the long waiting times and uncertainty of current diagnostic practice. However in adopting a streamlined process based on a simple biomarker, we risk losing much of the benefit of the current clinical pathway system. When someone with suspected autism meets over time with a range of clinical practitioners (e.g. paedatricians, speech & language therapists, occupational therapists) we find out a lot more than just their diagnostic status. The clinical team get to know the individual and their familiy. They may have some insight into how a child is learning and changing over time. They will be aware of a young person’s life stage and context – what kind of school are they in? what do they like to do in their spare time? what are their goals for the future? To be clear, I haven’t heard any biomarker proponents recommend that a simple test completely replace a comprehensive clinical assessment. But I am concerned about some aspects of the current drive for streamlining and efficiency in diagnosis. Autism is just not a simple thing and our diagnostic practice, and support services, must embrace that fact, rather than wish it away.
What about the ‘bio’ part? The term biomarker is a contraction of biological marker. The strong implication here is that biomarkers shoud rely on biological samples – like DNA. Instead in the world of autism research, increasingly people are assigning tentative biomarker status to responses to behavioural tasks, such as patterns of eye-movements. The major problem I have with this is that, in many cases, it would therefore be possible to fix the test. An autistic person who doesn’t want to be labelled as such could do their homework and just choose to look at the part of the screen they know autistic people don’t normally look at. And vice versa – a family in need of extra support might be able to acquire an autism diagnosis for their child by instructing them to look at a particular part of the display. If these examples seem a little extreme, consider instead the perhaps more likely scenario that a particular individual repeatedly experiences the same test and their responses therefore adjust over time – as they begin to be more confident with content they intially found off-putting, or get bored of the features that initially captured their attention. A biomarker which is subject to this kind of intentional or inadvertent manipulation by the individual would be clinically meaningless.
Translational laziness: This is my biggest bugbear with the repeated “this could be a biomarker” refrain at IMFAR 2016. At the moment, quite rightly, there is a strong emphasis on so-called translational research. Developing findings from basic science (i.e. information collected in a controlled laboratory environment) into practice – whether that is diagnosis, intervention, education or social services. One fundamental and common approach in psychology, and other disciplines, is the two-group comparison. We take a bunch of people with autism and a bunch of non-autistic people and we compare their responses to an experimental task, interview questions, online survey etc. When a difference between the groups is found, increasingly the temptation, desperate to prove translational applications of the science, is to say “this could be a biomarker for autism“. No. No it couldn’t. A mathematically significant difference between the average score of two groups (e.g. one group’s average is a good bit higher than the other group’s average) has none of the statistical or methodological features required for biomarker development or evaluation. In the vast majority of cases the finding is unlikely even adeqately to justify someone going on to do the further data collection and analysis necessary to robustly assess a candidate biomarker. No funders would pay for research built on such a tenuous foundation, no scientists would give it a favourable peer review. Not to mention the question of how the supposed ‘test for autism’ would operate in clinical practice ^. But still the same point appears again and again: “This could be a biomarker for autism”
To conclude…
The biomarker obsession among autism researchers at the moment is motivated by a desire to do something useful and practical with their research. To create a better diagnostic experience for people on the spectrum and their families, to provide a foundation for more personalised support and accurate science. This would be super. However researchers need to practice far more responsibility in their attempts to locate and report on candidate biological markers. As well as meeting the very high standards required scientifically, they should consider more carefully the clinical and social context in which their proposed biomarker might operate. As it stands, the current restricted and repetitive interest in biomarkers among autism researchers looks like it requires an intervention.
*in case it isn’t completely clear from the context, this test is entirely made-up and bears no resemblance to any autism biomarker research that I am aware of in the real world. It is ridiculous for a whole host of reasons.
^ for example, a large number of proposed biomarkers for autism rely on recording and analysing patterns of eye-movements but it isn’t clear to me how feasible it is to integrate the specialist equipment and the skills necessary to operate it into clinical services.
