Archive for the ‘Discovery’ Category

Big Data, the Machines and You

Friday, September 14th, 2012

Ah, Big Data, the old IT bandwagon rides again eh?  Who’s with me?  Yeah, you and every other IT consultancy in town.

The thing is, beyond the hyperbole (and of course the ridiculous notion that data can be big or small or even mid-sized) incredible things are beginning to happen with data that affect the products and services we use, how we innovate and even how we understand the world around us.

More and more we are using big data services to help us in our personal lives, they recommend our purchases, answer our search queries, even translate our languages and every day, through the beauty and wonder that is machine learning, they get _better_.

Having access to more and more data, combined with technological advances like the cloud which provide seemingly limitless storage and compute power we are able, finally to start to harness the incredible power and potential it offers not just society at a broad level or just to huge organisations like Microsoft, Google and Facebook but we also start to get to a point where that power becomes accessible to every individual and every single business.

As with all such major advancements, we’ll face our fair share of challenges too; some will be technical – we’re still looking for the needle, but now it’s in a billion haystacks, some will be cultural – how do you ensure that data is accessible and of sufficient quality? And some will be just plain hard – like in a world of data and machine learning, what happens when the algorithms take over?

As it turns out, bandwagon or no, big data is crucial to our respective success.  Don’t believe me?  Well, why not waste 30 minutes of your life listening to me trying to convince you.  This is a presentation I gave at this year’s Turing Festival trying to make exactly that point.  (You can also download the slides here).

Like it or not, the world of big data is here – it is now up to us to figure out how to make best use of it.

(n.b. Thanks and appropriate respect go to both GetAmbition and Interactive Scotland for both organising the event and creating video and supporting collateral).

The You Centric Web (Personalisation 2.0)

Wednesday, February 1st, 2012

clip_image001We live in a world that is increasingly connected, with services that offer a degree of “personalisation” based on assumptions of our needs created on an extrapolation of our previous actions.  As our digital footprints become larger and more connected, we are offered the potential to move to a new level of services that place us, rather than the services we consume firmly at the centre, a web that is ultimately formed around the individual rather than the other way around.

Today, personalisation is an incredibly blunt tool. The services we use, the content we consume, the adverts we see are all provided based on the basis of some prediction as to what we might be trying to achieve based on our past behaviours.  However, it is not just that this prediction is currently extremely basic, it also does not yet fully take into account the rich, broad context that accompanies us wherever we are, whatever we are doing, a context that holds the key to creating services and experiences that offer us not just a precise reflection of the services we need but also positions us to make discoveries of incredible new content, products and truly human experiences.

As individuals, our actions are guided by a complex range of signals which we intuitively use both explicitly and implicitly.  These signals form the basis for the services and content we choose to consume and influence the decisions and discoveries we make in everything we do.  The You Centric Web is a place that is not just aware of this context, but brings it together and makes sense of it, delivering an overall experience that is truly reflective of us as unique individuals.

This context exists across four key dimensions that are real time, living states that change and shift as we go about our everyday lives.  The four dimensions that influence our behaviour are:

Emotional – my emotional state.  My current emotional level will influence the decisions I make and the services I’m looking for. For example the music I select when I’m happy may be different from that which I select when I’m less so.

Social – who is with me physically and my virtual social connections.  Watching a sports game or a movie with friends is a different experience to watching it alone or connected to others remotely for a shared viewing experience.  Equally, the context of what my friends do is also a powerful signal that provides a trusted source of influence that may be incorporated to help me make my decision.

Environmental – where I am, the device I’m using, the time of day, the temperature, my location, my direction of travel, the current weather and so on.  Each of these factors plays a role in influencing both my decision and equally inferring the intent of my actions.  Searching for “sushi” on my mobile device at lunchtime while walking down a street in the centre of town will likely be for a different purpose than if I were to search for it sat at home in the evening on my main home computer.

External – This represents a broad range of external factors that offer further contextual signals that may influence my actions and decisions.  For example, a significant societal issue (like the recession) or a nationwide campaign on childhood poverty (or global warming etc) may make me consider different choices about the activity, actions and content I pursue.

These four domains are joined together and under-pinned by a rich pool of historical evidence about previous actions which can serve to highlight a likely (but not certain) outcome for any given decision or choice.

The connection of these different dimensions of context has been impossible in the past as not only did we lack the ability to accurately capture and interpret our current state in real time, we also lacked the ability to join them all together and analyse them as a collective.  In a world of socially connected experiences, big data, cloud computing and natural user interfaces this really starts to change.

Using natural interface technologies like Kinect, we are finally able to start to capture and use much richer information about the emotional, social and (some of) the environmental factors that will influence my activities. As they continue to evolve, devices like Kinect will help understand our emotional state, the environment we’re sitting in and who is with us. Early examples of this have already been shown, but we know it is still early days and the technology still has much further to go before this is pervasive and usable across a variety of services.

clip_image002In addition to the technological developments described above, the increasing capability to connect and analyse vast, disparate data sources starts to provide the opportunity to take a broader “systemic” view and a deeper level of insight that can be used to infer further elements of the context surrounding an individual.  In this area, the brave new world of “Big Data” and the cloud becomes an immensely powerful capability that offers the potential to provide incredible new context and insight that can be used to shape experiences even further. My favourite example of this was some recent research done in the US that analysed cellphone usage data from 50,000 individuals and was able to accurately predict the _future_ location of any given individual with 93% confidence. (And there I was thinking I was in control. )

By joining up and effectively understanding this broad, rich context, it becomes clear to see how basic today’s world of personalised services is and just how far we’ve yet to go.  Being told that  “people who have bought product A also bought product b” is no longer going to feel useful or even relevant.

However, beyond the further technological innovation that will be required to make this a reality, there are also several advances in how we as a modern society think about, use and trust the services that will be required if we are to get to a point where we can really maximise the potential of this world that is moulded around the individual.  In particular, there is a growing trend of a fear of “over personalisation”, a world full of filters and “popular content” which is devoid of discovery and one in which the power (and importance of) serendipity becomes increasingly hard to come by.

The common mistake being made that drives this fear is to think that personalisation by default excludes discovery, or that perfect personalisation means ultimate precision.  This is a world where I am precluded from finding new things which I am unaware of or from uncovering new items which may not be considered relevant to my interests.  In many ways, the You Centric Web must represent the slightly random, uncontrolled nature of our human world, injecting random and unrelated content in order to broaden and extend the overall experience and introduce new direction and insight.  In some ways, this is no different to the role a good news editor plays in ensuring that the audience receives a broad range of content that is of interest to the majority along with an essential range of content that is intended to interrupt and disrupt your established areas of interest – a process that not only broadens the mind, but equally expands the potential for new discovery and conclusions (and the adjacent possible).

clip_image003In a world drowning in data and information, personalisation provides the only way for an individual to not just find, but most importantly, to trust the information and services being provided.  Trust is key here, not only does the consumer need to trust the services, they increasingly need to be able to trust the service provider, to know that their data and information about themselves is respected, kept private unless the consumer has indicated otherwise.

We need to recognise that our society continues along a long established journey about privacy that is not new in the information age, it has in fact been an issue for discussion and debate for hundreds of years.  The information age accelerates the pace of change, but the basic principles remain the same for the individual.  I need control, I need transparency and increasingly, I need a tangible value proposition (i.e. what do I get in return).

These principles, offer us a way forward.  By putting the individual in control of their data, being transparent about how the data is being used and crucially being really explicit about the value that will be provided we can turn around some of the “trust issues” that we read about so frequently today. 

The You Centric Web offers an inversion of today’s web, placing the user at the centre and in full control of the overall experience. It promotes discovery, celebrates serendipity and offers a personalised path through the oceans of data, content and experiences that the modern digital society has to offer. The technology required to deliver the You Centric Web is beginning to appear now, but we have further to travel before it can reach its full potential.    As a society, we will have to shift our expectations of how the digital world can augment the physical world and service and content providers will need to plan for and deliver on the potential of this connected, intensely personalised world as well as work hard to win the trust of consumers by placing them at the centre and protecting and respecting their rights.   The You Centric Web is an inevitable part of our technological evolution; it is now down to all of us to ensure we are able to take up the promise of everything it has to offer.

Exploring the Adjacent Possible

Wednesday, September 28th, 2011

Over the summer, a friend recommended Steve Johnson’s book “Where Good Ideas Come From – the natural history of innovation”, not only is it a great book, it contains some really important themes that I think are worth calling out for future exploration. Here’s a few that resonated with me (and why) but you can expect to hear more about these in the coming weeks. (Anyone paying attention will also note that Johnson dedicates a whole chapter in his book to “serendipity” – I have excluded it from this post not because I don’t find it relevant but because I want to focus on it much more deeply in the near future.)

1) The Adjacent Possible
This is one of those “obvious now that you mention it” kind of things where simply, when you are looking for the best ideas, or what makes ideas work, you have to find the ideas that are adjacent to existing concepts, two steps away is too far and will never happen, one step away and you’re in. This is where the natural history meme comes in, if you think about evolution, you can’t start at the big bang and immediately make humans or sunflowers or puppies or chicken tikka massala (although I’m thinking there would have been enough heat there for a pretty efficient tandoor oven so you could have had a shot at that last one), the point is you have to incrementally change all those atoms and molecules over a period of time until the evolutionary process provides you with the opportunity to move that final strand of DNA or molecule to make that complex outcome or even Graham Norton.

hovercarThis principle is exactly the same for our ideas, the reason I can’t have my hovercar is because the concept is not adjacent to our existing ideas or technological capabilities – we simply lack the building blocks to put it together (although, after hearing about that Swiss bloke practising fusion in his kitchen, you’ve got to think that somewhere out there, is a Doc Brown waiting to happen).

In the book, Johnson uses the analogy of a house with infinite rooms, all interconnected. You walk into the first room, it has 4 doors each connected to further rooms with further doors. You can’t go to any room in the house without first traversing a series of other, connected rooms. This is where some of the inherent beauty of the adjacent possible lies, it’s potential grows exponentially with every new stage of discovery.

It’s also a great tool to try and use when you’re trying to validate the strength of any given idea – does it represent the adjacent possible or the distant impossible?

2) Liquid Networks & Exaptation
Forgive me, while I get all Hybrid Organisation on your collective asses, but we’ve been talking for a couple of years now, about the importance of workplace design and its relationship to the collective innovation of a team or organisation. Johnson provides some nice analogies that help to push that story a little further. So, close your eyes, put yourself back a few years and imagine yourself back in the 5th form at school, in double Physics to be precise, just before your mock O’level. Using the concept of solids, liquids and gases, Johnson describes the inherent ability for individuals (acting like molecules) to oscillate those individuals around them (stop sniggering at the back Coplin, I said “oscillate”). If you constrain the extent to which the individuals can move, i.e. restrict their office layout with offices and cubicles, it’s like the molecules in a solid, they exist but have little effect on each other. If you go to the other extreme and remove all constraints (i.e. no office at all) you end up with chaos (the collaborative equivalent of a gaseous state) where molecules dissipate into the environment and have little effect on each other.

What you really want, he proposes, (and the Hybrid Organisation agrees) is a liquid state, where the molecules are free to move around but remain within the overall context of the organisation. Molecules can bounce around inside the liquid, resulting in an agile collective can change shape and move as needs dictate. This is essentially creating the perfect environment for collaboration, innovation and exaptation. Exaptation? That’s when an organism develops a trait for a specific use, then that trait gets hijacked for a completely different purpose. Feathers are the classic example of this, initially evolved to provide temperature regulation for birds, the birds discovered that they’re also pretty good for flying too, and then evolution takes over and you end up with streamlined aerofoils rather than fluffy, downy warmth.

Exaptation of ideas is a huge part of innovation and is also a crucial part of serendipity. Like serendipity, it also requires a free, liquid network that enables disparate ideas, individuals or concepts to come together to create innovation.

3) The Slow Hunch – Why Eureka! Takes Time
EurekaThere’s not much to this important concept, other than it dispels the myth that most of our great innovations come from single individuals who experience a “light bulb” moment and have the big idea. Although that explanation may work for Hollywood because it’s a better and easier story to tell, the reality is that many of the important discoveries, innovations and ideas of our entire history are actually the result of lifetimes of work and study often bordering onto obsession. We know this inherently, but often choose to ignore it. Our best ideas, are often the ones that we’ve left to fester at the back of our minds, nurtured and tended by, a long shower or a long walk, to help them mature to their full potential. <Assumes crap French accent> “Ideas are like fine wines, they need time to mature, percolate and infuse” .

The reality of the slow hunch is actually a cumulative factor that builds on liquid networks, exaptation and serendipity – all elements that make our view on _­discovery_ so crucial to our overall objective in search. What is key then, is to be able to feed, nurture and evolve an individual concept over a long period of time, incredibly though it seems that in order to make the breakthrough, the concept needs to be at the background of your cognitive activity not the foreground, slowly percolating away to completion.

You may not see the link in all of this, but every time I walk into the office I do. These days, I know I’m pretty lucky, I have a job that makes it easy for me to explore the adjacent possible, to “exapt” ideas by forming a liquid network of influencers and thought leaders inside and outside of Microsoft – but I also can see the constraints, especially as we get busier, we don’t always have time for the slow hunch to mature, and we find it hard to leave the safe confines of our team, office or sector. My encouragement to you is to make time, make time to cultivate new ideas, make time to speak to people outside of your team, make time to explore the adjacent possible. It won’t help you today, but I guarantee it will set the stage for your greatness tomorrow.

Back to the Forbidden Planet – Exploring in the Digital World

Friday, May 27th, 2011

Back in our youth, when we had more time, less experience and big dreams, I suspect many of us will have spent plenty of time exploring.  No, not the pith helmet “Dr Livingstone I presume” kind of exploring or the gap year partying that passes for the “travel broadening the mind” kind of exploring that goes on these days.  No, I mean time well spent exploring music, literature or some other similar love.

FPFor me it was comics (yes, I know, just how much more of the geek stereotype can I fulfil? Just the sandals and Star Trek insignia tattoo now, and I am complete).  I started reading them when I was 7, and they changed my life.  I fell in love with a particular title (2000AD) which, still running today, represents the absolute peak of British writing and comic artistry in this genre but 2000AD alone wasn’t enough, I wanted more.  Remember, these were the days before graphic novels, VHS/DVD, the internet and more than 3 channels on the telly and there were few places in the country you could go to explore this world further.  In fact, there was just one oasis, aptly titled “The Forbidden Planet” a comic shop in London, some 150 miles from my home town.

The Forbidden Planet became a mecca to me, I would make an annual pilgrimage down to London (or semi annual if my pocket money could stand it) just to spend hours in that store on Denmark Street, exploring the incredible new world that I had discovered.  I still remember the place (it’s in a new location and much flasher now), I remember the layout, the euphoria of so much content in one place, I remember the smell of the floorboards and old paper and the excited apprehension that comes from being the country hick in the city.

The hours I spent in there lead me to all sorts of extraordinary places, new comics, books (even proper grown up ones that normal people read), all things I would never have found if I hadn’t had the opportunity for a tactile, tangible experience of what was effectively curated content.  I’ll go out on a limb and make a guess that all of you will have had a similar experience (or maybe still do), it won’t necessarily have been comics, but I’ll bet you spent a lot of time in book stores, record shops, music shops, motorbike dealers whatever, doing exactly the same thing – exploring.

So apart from a little misty eyed nostalgia on my part, what’s the point of all this? Well, the point is that providing the exploratory part of discovery in a digital world is _bloody hard_.  I’m not arguing the semantics here about the value of holding the album cover of your favourite artists new release poring over every detail vs looking at a Jpeg of the same, as I believe we will adapt to getting that experience digitally (and in many ways it can be richer), no this is about coming to a place filled with similar (but not the same) kind of content where you are free to explore your interests. 

As the curators of content, we can do so much, we can provide the path to explore new worlds (both accurately and, as we get smarter, randomly yet with relevance) but we have to work hard to provide the “environmental” experience which becomes so important to us as individuals.  The key actually comes back to the same old thing – “content is king” but in reality what this means (when it comes to exploration) is establishing a broad set of meta-data about individual elements and more importantly, being able to surface this meta-data as well as the specific item as part of the curated content (or search result).  This alone won’t replace those dusty bookstores of our youth, but it will in some way help to form the bridge between the digital and analogue worlds.

In many ways, this is part of the sentiment behind Stefan’s recent allegations that traditional search is failing,  this can easily be passed off as jingoistic hyperbole (as Danny Sullivan tried to postulate on Twitter) but in reality it’s a really important reminder that the web and more broadly the internet is no longer powered by links alone. This is about providing a digital service that is reflective of the analogue equivalent, serving each and every query with a broad result that includes a rich spectrum of responses and associated content, moving us waaaaaaay beyond those 10 blue links once and for all.

Curation, Serendipity and Rastabilly Skank

Thursday, May 12th, 2011

Although typically talked about in the context of museums and artefacts, increasingly curation is becoming adopted in a digital society as the concept of having others select a collection of content for you.  It’s an incredibly powerful concept and as we move forward with search it becomes one of extreme importance as we seek to get the right combination of both relevancy _and_ trust in our results.

rastabillyCuration of content is nothing new, and I’d forgotten how pervasive it was in how we consume content (and has always been) until I started randomly thinking about how one of this year’s big new trends will be cloud based audio (cue Google, Amazon and Apple announcements) and if I get all my music from the cloud, probably curated through some mechanical process like Lastfm or Pandora then is radio dead? Well, of course it’s not. Radio works because (mostly) they have proper people “curating” collections of tracks for specific audiences.  Humans (especially good ones) know that there’s a difference between the Clash and the Sex Pistols and that “Punk rock” is an attitude not a music style, right kids? (But I digress).

This random thought then joined up with something I’d heard earlier in the week from those lovely folks at the Guardian’s TechWeekly podcast – (sure, I’m sucking up to them, but ignore that, they still represent the _only_ place in this country you can go to for proper analysis on the societal impact of technology).  Last week they interviewed the folks at Artfinder (brilliant concept by the way) and whilst they talked about their innovative service, they stopped off to talk about the importance of curation and our old friend serendipity.  Then Chris Thorpe (founder of Artfinder) said “John Peel was probably the ultimate serendipity engine” – talk about the penny dropping (and at the wrong speed too).  For those of you viewing at home in black and white, John Peel was one of the most influential DJ’s in the UK, his tastes were, let’s just say eclectic, he knew no musical boundaries, and his playlists provided the soundtrack to the youth of millions of kids in the UK.  What made Peel brilliant was he knew his audience, knew his music and had the confidence to introduce new material (new to the audience, or new to the world, it didn’t matter.)  Ironically for this anecdote, it was Andy Kershaw or Mark Lamarr that played this role for me and given that unlike John, they haven’t yet shuffled off this mortal coil, their continued absence from the airwaves remains a national disgrace – can you fix it Jem? – (BTW – You need to follow Jem if you want the best curated experience of all that BBC radio has to offer).

So back to the point, why is all this radio nostalgia important?  Well think back to what I just said – curation works best when it is done with:

  1. good knowledge of the audience,
  2. good knowledge of the subject and
  3. the courage to introduce something new.

These are the very essence of discovery in the digital world and yet another signal about why, even with the best machine learning systems and algorithms, you still need the human/social signal to get it right. It’s easy to generate a list of “likes” of seemingly connected content, and it’s easy to play to the “herd”. What’s hard is to make it properly personal in a way that will resonate with the individual and take that concept of personalisation to the next level. 

This, my friends, is our challenge, if we are to truly get beyond relevancy, introduce trust and become the ultimate mechanism allowing the curation of the web for individuals, we have to figure out how to make search the enabler (note not owner) in how this happens.

Besides without this or Peel’s incredible talent, how am I ever going to find the next Rastabilly Skank

The other side of social search

Wednesday, May 4th, 2011

listenI have a friend, (hard to believe I know, but it’s true) who is the only person I have ever met that can accurately predict whether or not I will like a particular band, album or piece of music. He sends me links and information on bands I’ve never heard of and would likely never discover and makes accurate predictions on the extent to which I will like them even though he knows I’ve never heard of them, never mind listened to them.

It’s a great service for me and saves me no end of time (but equally costs me a lot of money, because when he’s right, he’s absolutely right and I end up buying the entire back catalogue) but what’s really interesting about this, is that it represents a principle of a very different type of social search to the one we’ve been discussing recently around the aggregation of “sentiment” (Facebook “likes” in our case) around a given topic.

The importance of this new approach is that it deals with the assumption that “your friends are like you” that is implicit in the current method of introducing the social signal to search. Although at a high level, this assumption may be broadly right, at a more granular level it’s often completely wrong. My social network is made up of friends, family, colleagues (old and new) and a few other random acquaintances – to make the assumption that all these people are “like” me is, generically, probably true, but at a more specific level it is hideously wrong – for example, one of my brothers supports West Ham United and listens to the Smiths. Not following WHU is probably self-explanatory, but like Mitch, I must confess I never went through a Smiths phase. But I digress, my point is that just because someone in my network likes these things, does that mean I do? Of course not, generically you might infer that our connection may imply I like football and 80’s indie music but to be explicit about it would just be silly.

A new piece of research from our friends in MSR Cambridge is focusing on this principle, using a technique called “prediction extraction” to solicit opinions from friends as to whether they think the individual in question would like the item in question before they have even experienced it.

This approach is based on the observation that “although your friends are not you and may not have the same tastes as you, they are likely the people who actually know you best”.

You can read the detail of the approach here but essentially it offers a number of advantages, primarily around the accuracy, quality and coverage that the harnessing of this tacit knowledge brings, the real trick however is how to extract this information in a way that is easy and rewarding for the contributor and seamless for the consumer.

Predicting your friends opinions is nothing new (Mum _always_ knew best, right?, and it augments rather than replaces the “wisdom of the crowds”, but it does offer a new way of providing accurate, insightful predictions around the relevancy of a given topic or item to the individual. Going forward we’re going to need a range of these techniques if we are to truly humanise the way search provides us the answers we’re looking for.

My friend and music sensei doesn’t like the same music as me (can you believe he only has _one_ Men They Couldn’t Hang album?) but he does know me well, and he loves music – this combination alone could save HMV’s fortunes (and likely bankrupt me!).