Earlier this month at IMFAR 2013 – the International Meeting for Autism Research – I presented a conference poster reporting on the Click-East project. Sadly we haven’t completed the randomised controlled trial (RCT) yet, so I couldn’t share the final outcomes – that will have to wait for IMFAR 2014 – but I’m taking this opportunity to reflect on what I’ve learnt over the past 12 months of running an RCT as the process begins to draw to a close. Here, in no particular order, are the top ten lessons I’ve learnt so far:
1. Aim for a recruitment range, not an absolute
It’s traditional in designing an RCT to use power calculations to help you decide how many people you need to recruit for a study. For anyone not familiar with these, essentially you look at previous research and take into account not just how big their samples were but also how dramatic their findings were. For example, if a study recruited 100 people per group and had a hugely significant effect, then to replicate it you might not need quite so many people in your study. In real life, this process is complicated by the fact that we’re rarely directly replicating someone else’s work. So for Click-East we had a look at published studies doing joint attention type interventions, but some of them were parent training and some of them were therapist-led. Some of them lasted two weeks, some of them lasted six months. And the outcome measures they were using were all different too. You can add to that the fact that, unlike my example above, normally what we’re doing is saying “well they didn’t quite get a significant effect and they had 20 kids in each group so do you think if we have 25 that would be enough?” So it is a bit of a guessing game to say the least. In doing our power calculations, we decided a sample of 30 children in each group would be good, and that would “allow for dropout” too – so if some families pulled out of the project along the way then we’d still have a good chance of finding effects (if those effects existed). But what I wish we’d done was set a range for ourselves, or rather a lower limit. Because now I’ve recruited 54 families and 3 have dropped out and I have no idea whether the final sample of 51 is going to be enough or not. We shall see…
2. Plan your recruitment carefully around holidays
This covers two kinds of holidays. The first one is my holidays. I tried to see as many participants as I could just before going on holiday last summer, and did nothing else in the fortnight before I went away. The problem was that then, when I got back, I had an empty calendar. I had to send out recruitment letters, wait for responses, book people in and it was almost a month before I started getting data in the bank. Add to that my fortnight holiday and the whole thing ground to a halt for six weeks! I should instead have sent out all those initial enquiries before going away (with a note to apologise in advance if they had to wait a bit before I replied) and then hit the ground running when i got back with a whole load of families rareing to go.
The second issue is around national holidays – particularly Christmas in the UK. We aimed to finish recruiment at the end of the year in December but underestimated not just the effect of people being unavailable for the last ten days in the year, but also their reluctance to commit to new things in the run up to Christmas. We managed to extend recruitment in to January but I would say in the future that it’s wise to write off December as an opportunity for recruitment, perhaps especially when seeing families with young kids.
3. Be careful about referrals from unknown recruiters
Our project got picked up by a clinical team who were enthusiastic and kind and shared our recruitment call with families in their case load. Brilliant. But then I found that a large proportion of the families coming in weren’t eligible for the study – children were over the upper age limit or, worse, they came in for their first assessment and then didn’t meet ADOS(autism diagnostic observation schedule) criteria. The problem was that the recruiters took a broad view of eligibility for the study but we had some tight entry rules and hadn’t communicated these well to the recruiters. I did fix this before it continued, but I wish I’d done it earlier. I was so grateful for their support I just let them press on, but I should have asked them not to approach families and suggested a meeting or at least a phone conversation to check the details first.
4. Budget for birthday cards!
The families enrolled in the Click-East trial are incredibly precious to me. I’m constantly amazed by their commitment to the project and to the concept of research and I hope to continue to work with them in the future. But sometimes it feels like all my correspondence with them is just begging letters from me: can you send back that questionnaire please, here’s a reminder about your next appointment, will you fill in this online survey. So it is great occasionally to send something in the post which is a bit less demanding. I send out birthday cards for all the children on the project, and they all got Christmas cards last year too. I only wish I’d put a bit of extra cash for stamps in the budget!
5. Use standard operating procedures and posters
I was quite dubious about the need to document the project processes in detail when I started out. I thought it was obvious what I would be doing and it didn’t occur to me that I hadn’t really worked out the details until I was about to start seeing my first children. Suddenly the standard operating procedures for recruitment, consent, assessment and so on were invaluable, and I turned them into big posters all over my office to remind me what to do. For example, your research proposal might say something like “families will be recruited via the website” but what does this mean. If someone emails you to say they want to know more, you reply and send info. But what if you don’t hear back again? Should you nudge them? Once? Twice? If their mobile number was in the email, can you call it to see what they think? There’s a fine line between helping a busy family out and pestering a reluctant parent to join in. What about when somone calls to express an interest? How much detail should you request before you give them a chance to read your full information sheet? What should you do with that info if you don’t hear from them again? Standardised procedures gave me the confidence to push people a little, safe in the knowledge that I knew when to stop, as well as clearly delineating all my other responsibilities in a step-by-step format. I’m hapy to share copies of these with anyone who’s curious to know what they look like.
6. Take lots of notes on your assessments, especially if you have hours of film
At the end of the RCT you’re going to have a whole bucket of data, in my case, hundreds or hours of film footage of assessments which I don’t have time to review. For our study, we asked parents to consider giving separate consent for this footage to be used in things like conference presentations, lectures and for training purposes. Loads of them gave their permission, understanding that in order to help educate people about autism, what you really need is to show people directly. So if I see a child do an absolutely perfect hand flap, or lining up a load of toys, or echoing my speech perfectly, I make a note so I can go back and extract this example to illustrate a slide in the future.
7. Start with the fun stuff
We’ve been seeing children aged under 6 years old with autism and they take part in a range of assessments. For a good while, I was attempting to do the Mullen Scales of Early Learning with them when they first arrived. It was a disaster. The Mullen can be engaging to some children but it is not super fun and doesn’t involve a lot of toys and games. The theory was that I should do this first to give the children the best chance of doing well while they were rested. But instead I think it meant their first impression of this (already unnerving) new person and new place was boring. I switched to doing the ADOS first which meant the children walked in to a room full of toys and anyone reluctant to get stuck in could normally be persauded by the appearance of the bubble gun. Much better.
8. Think about the papers you want to write, right from the start
Obviously you can’t create the results you want but you can think about what you want to say about your findings right at the start. So, whether you get a significant group effect or no group differences , you might want to explore the background of the participants – do variables like maternal education or socio-economic status modify your main findings? If you have a main effect, you might want to be able to say this is an approach which works for anyone and if you don’t have a main effect, you might in fact find a sub-group who DO benefit. What’s key here is to think about the level of detail you ask for and how you organise it. We asked parents how many hours per week their children went to nursery, and what supports they received there, in quite a lot of detail. In fact, I think all we will extract from this is a categorical variable that sorts children into: no nursery; mainstream nursery; mainstream nursery with support; specialist nursery. So I wonder really what all those detailed questions were for.
9. Have a steering group
By far the most valuable part of this RCT for me personally has been as a training opportunity. And that training has been delivered by an incredible steering group who have guided the whole process from right when we started creating the app to now. The group includes a clinical psychologist, a paediatrician, and a human-computer interaction specialist, all of whom have a specific interest in pre-school autism. We meet every three months and preparing the agenda for each meeting provides me with a really valuable chance to sit and reflect on the progress so far, the questions that have arisen (and my responses to them) and to think about what happens next. Likewise, the minutes from those meetings guide my progress over the following quarter year.
10. Blog about it!
I’ve never been a technophobe but I’m not an early-adopter either and when my husband suggested I blog about my research I was skeptical. I’m still not sure many people read these posts or whether anyone at all finds them useful! But I am so glad he convinced me to do it because I find the process immensely therapeutic. I like the notion of the blog as an archive of the evolution of a research project – or perhaps of a research career, as other, new projects have begun to feature on this blog more recently. I am also glad of the incentive to look through my research projects and other academic duties and find something of relevance to the community around me. I’ve always disliked the ivory tower notion of academia – here we are footling away on projects that no-one understands or cares about and that will never reach the outside world. No thanks. This blog allows me to make the links between my research life and the real world explicit and if once in a while someone looks back through that window and sees something interesting then I have done my job.
