I’ve been working with SenseMaker® now in development projects for a few years and I was recently reflecting on how far we’ve come in what we do. Lots of lessons – often learned the hard way in the early stages, but never disastrously. It’s time to share some of those lessons, for anyone who’s interested.
This one is critical. The biggest mistake – one that is thankfully rare in the development world – is expecting things to work first time out of the box perfectly. There are always hiccups, for all sorts of reasons, and resolving problems is rarely helped by a blame-hunt. The most important thing is to spot the problems early and then work around them. That starts with testing.
- Test the concepts in the field – are they understood by the people on the ground, do they cover enough of the possible options?
- Test the language – is the translation good enough? what nuances have crept in that were unintended? is the language at the right level for the literacy levels of the population we’re dealing with? do our interlocutors (if we’re using them) understand the concepts enough to explain them?
- I’ve learned to introduce a deliberate typo into translated material. If your proof-readers/checkers don’t pick up your typo, it hasn’t been checked
- I also add a note to concepts to explain what they’re not – to help clarify the concepts. I’ve noticed that we spend lots of time trying to define what a concept is, when it can be easier to look at what it might be confused with and define it partly by what it is not.
- Test the tools – does the website work the way we think it does? have we checked the data? have we checked the formatting for non-Roman alphabets?
- Test the actual environment and system that will be in use – just because I’ve tested the website on my nice iMac with reasonable UK-quality broadband in the latest versions of browsers, doesn’t mean that someone in Ethiopia (or the EU!) can access it from their company-official laptop that’s still running Windows XP and Internet Explorer 6 via the connectivity treasure that is Ethiopian internet…
- Test the data – have we entered every type of data that we’re aiming to collect and have we checked that it’s come through as we expected?
An example triad
I’ll be honest – I saw this first done by Anne McMurray in Northern Ireland, but I’ve stolen and adapted it now for most projects. Up front, include a triad that’s locally-sourced but irrelevant to the project. It needs to feature three things that you have to blend – so tea/milk/sugar is one we’ve used in the past, as is “how do you like your dhal?” – lentils/onions/water. Then label three points in the triad – firstly one that’s only about one corner, then another that’s a blend of two corners, then a third that has elements of all three. Label them clearly – but then, if you’re feeling daring, include an actual triad with those points to it. By making them actually do it, rather than just read about it, you have a better chance of people blending three filters on each triad rather than treating it as three options of a multiple choice question.
Multiple collection routes
(This depends a little on the nature of the project – if you have to bring academic rigour to your research, some of the following options may be restricted.) Depending on a single, formal collection route in a complex or chaotic environment is to put a lot of costly eggs into a single fragile basket. Better, in my recent experience, to build a framework and then use it opportunistically – collect material from multiple routes. Plan for five or six. The redundancy means less panic when a couple of collection routes fall flat. Routes I’ve used in the past include:
- Adult education institutions
- Local managers’ regular rural visits
- Meeting places
- Social media
- Replacing standard admin elements
On those occasions when we use people to collect material in the field*, we now coach and mentor, take them to the field, get them to share practice and come up with their own list of hints and tips for the next tranche of collectors. In most places, everyone has heard something, everything has been heard by someone, but noone has heard everything – so getting them to work/talk together really improves the learning. Much better than berating/teaching from the front of the room.
This one parallels the “Test everything” up above. People forget things – how to “save” an example; technology systems change – upgrades can often result in downgrades as any long-term user of Windows or Microsoft Office can tell you; complex environments will continue to evolve. So check the data every so often – is it still coming in? is it still legible? are the collectors consistently skipping certain elements? Planning will only get you so far.
Produce results interactively
This depends again on the type of project. If it’s pure research, then a report may be the perfect output. I’m not a fan of reports – I’m far more interested in helping people get to grips with the new information, understand the consequences in their own terms and work to use the information to improve their projects. And I don’t believe that ever happens effectively through simply telling/writing/presenting conclusions – especially if they’re long and detailed.
- Workshops with respondents – present them with patterns, ask for their interpretations, dig deeper
- Workshops with stakeholders – same as above, but for subject matter experts ask them first to mark where they expect to see the data before you reveal it. Their assumptions will often be good, but if they’re off, they’ll tend to ignore any learning if they haven’t had to state their assumption up front.
- Teach the Explorer software – it is unbelievably powerful once you start to play with it
- Use the Cognitive Edge exercises and techniques from the training courses – ritual dissent, Cynefin, action forms and safe-to-fail probes
Stakeholders share an example before seeing results
This one I’ve learned over time – if you’re showing patterns of triad/dyad/MCQ/Stones data to stakeholders, make them fill in the full framework beforehand. On paper preferably at the start of the session where you’re presenting results. This forces them to think about what the triads are – and reduces the number of “But what is that telling me?” questions.
I'm also extremely grateful to all the clients, associates and partners with whom I’ve worked in all the various projects, with particular nods of the head to Milica Begovic at UNDP, Irene Guijt, Kecia Bertermann and Rebecca Smith at GirlHub and Dave Snowden. Their brains, spirit and patience has helped immensely in getting through the various learning moments to the point that we can now call it “experience”!