Beneath the surface of things


Whatever you think of the Microsoft Surface device, it’s been fascinating to watch the first steps of development of apps for the new style of interface.

It’s a radically different world now from when we started out web development in the last decade, but it will be just as hard to develop the new ways of thinking about how we interact with machines in this new setting.

When the web was in its infancy, we saw lots of ideas being transplanted from the world of print publishing, and later from the world of advertising. That resulted in some pretty horrible sites being built for many years.

Remember all those years of non-scrolling pages, or the multitudes of ‘single idea’ sites where the navigation was in the style of a cockroach, or a pizza, or a brain, or whatever? That was people taking what they knew about another subject and misappropriating it for a new and much more interactive medium.

But the fundamental design restriction of the early days wasn’t lack of knowledge or thinking about the interaction of computers and people. It was technology itself.

The interface technology – adapted painfully from academic routes – was difficult to learn. And the skills needed to build the functioning parts of these applications were very hard to locate at all. At first, it all had to be written from scratch which few could do, and even if they could, even fewer could bridge the gap between the teams who would build the applications and the teams who could work out what they should do.

With hindsight, we’ve come a very long way in a very short space of time with web deisgn. The world is now full of very talented interface designers, user experience specialists, technologists, development platforms and reasonably coherent ways to get them all to work towards a common goal.

So, the challenge of developing in a fundamentally new paradigm for Surface is quite different. The technology itself is no longer the killer issue. The platforms (WPF, XNA and .net) are well understood, as are the architectures for information flow and manipulation.

Instead, in the user-experience space, we’re faced with a need to deeply rethink pretty much everything we thought we knew.

Here’s an app which allows multiple surface users to search through and display news and entertainment content from MSN: Pulling all the feeds together and building the app are relatively straight-forward but working out how the users best get to that information is a another matter.

In this new setting, we must avoid the non-direct metaphors which characterise GUI, so almost all the controls that we’ve spent the last 10 years standardising have to go. The concepts of buttons, windows and scroll bars should all be vanquished. Instead, we need to find direct metaphors about interacting with objects. We need to find ways to avoid having specific orientation. We need to think about how multiple users can work together with the content. And we need to do all this in way which is seductive and enticing.

Does the news reader do this? Well the feeds themselves may not be a problem, but they are still hardwired to sources that have been created for online applications. Are we avoiding orientation? The now somewhat ubiquitous use of a large circle seems simultaneously too obvious and problematic. Where is the natural metaphor for this? A bowl of sweets perhaps but inside our ‘bowl’ we have a swarm of what can only be described as ‘windows’. And what a rotatable device solves in terms of orientation, it screws up immediately in terms of multiple users, as it faces in one direction at a time.

It’s certainly not easy. And, it’s wrong to pick harshly at early attempts to solve these problems, but we’ll necessarily need to work a lot harder, take more risks and be more experimental if we’re to get more than a large, horizonal web styling.

Looking back to the web, we’re in a position now when so many elements have become engrained in our thinking. We have a great deal to rethink

  • Single source of consistent navigation
  • The concept of a home page
  • The concept of flow being within containing elements
  • The idea that the user opens and closes things (still very prevalent)
  • The concept of linear stand-alone ‘states’ (already eroded by ajax interfaces)
  • Ideas of what it means to be recoverable
  • Single user tasks (even if we only have one active user, we can have multiple concurrent task- or explore-elements alive concurrently)
  • The concept of private browsing – another user’s actions can (and on Surface normally do) interrupt and influence each other
  • The method of ‘what’s next’
  • The entire concept of intermediary metaphors
  • What we mean by scanability for copy and images
  • The role of content
  • The sorts of assets (copy, images and video) which make sense in the medium

The list goes on.

And owing to the nature of the device and its development, progress is likely to be slow. This is accentuated by the polarisation of companies who are developing apps right now – between extreme usability and user experience experts who look at these problems very academically, but also often in a great hurry,  and development companies who are keener to show technology ideas than think in too much depth about what goes on in the mind of the user.

Since Surface is a reasonably long way from being a consumer product, the audience for Surface applications is essentially the same groups – those excited about the hardware and those who’ve spent years learning how you do digital design for the web.

Few apps currently bring the promise of natural user interface, collaborative computing or object recognition. Before we can do that we still have a lot to learn and even more to unlearn.

On the Surface

We’re lucky enough to have a Microsoft Surface where I work (Conchango). The reaction to it amongst staff, clients and – it has to be said – the cleaners has been most interesting. 

For those who’ve not seen it, the form factor of the new device is a horizontal, apparently touch-sensitive screen mounted on top of a rectangular plinth that keeps it about 1/2 a meter off the ground. The only way to control the device (when running in proper mode) is using the screen itself. To worry the cleaners, the whole thing sort of glows in standby mode or features a rippling water pool. Once in fully-on mode, the device has a number of applications (still mostly at demo stage) which allow the user to interact with music, photos and other graphically-produced objects. 

Microsoft Surface

Of course, the Apple zealots mutter, ‘it’s a large iPhone’. Others are more concerned with its unmistakable tableness, although I’m pleased to say they’ve stopped calling it ‘the table’ (team email today: ‘Big pile of cakes. For Anni’s birthday. On the surface.’). Others concentrate on it’s computerness (the ‘a computer in a table’ brigade). It runs on Vista you know… and so on.

I think the truth is a little more exciting.

This is Richard’s daughter, Bel trying it out:

(and the blog entry)

Bel is four years old, and is already a fan of the Cbeebies site but find it difficult to navigate with a mouse. And what did she think of the Surface?Well, of course, she didn’t think it was a large iPhone. She also didn’t think of it as a computer. She just thought of it as a collection of objects to interact with. In one of those supercute things that three and four year olds do automatically, she even wiped he finger on her trousers when changing colour in the drawing application.

On iPhone, Windows 7 and the million other devices that will arrive on the shelves of Amazon over the coming year, multi-touch is an additional input style to complement (or replace) other input methods and devices.

For Surface, the opportunity to try something new has a few important additional elements:

The first sounds like nothing but is actually quite important. The fact that the device is horizontal forces us to reconsider much within the design space. In particular, it breaks down the concept of orientation, and it makes it much more relaxing to operate it with your hands directly (rather than via a second device like keyboard or mouse). Operating vertical touch screens by touch is tiring, and there is a limited range of interaction.

The second is not well explored yet in demo applications but is potentially the biggest shift – and that is that the device can (indeed: should) be operated by more than one user simultaneously. Traditional computers discourage this by their nature (one mouse, one keyboard, vertical screen best viewed head on). Yet how often do we collaborate on one or more machines? How often have you turned your monitored to share with someone else in the office? 

And yes, it’s quite big. And that helps it to attract people in to the device.

Thirdly, because the interface is driven by cameras (not capacitive as with the iPhone), it can recognise certain objects that are placed upon it. This functionality is limited at the moment, but still it pushes our imagination. External, physical objects introduced to a virtual, computer generated space can create natural reaction, and open up possibilities for those objects that didn’t exist when they were in real life. We’ve used this to extend the properties of a physical wine bottle (to add tasting notes). We’ve seen the ability to turn a coffee cup into an air hockey paddle.

Of course, Surface points to a future of ubiquitous computing – a sort of dystopian blade-runner fantasy full of shouting billboards and walls that you type on. But actually we’re already starting to to be able to use the device to breakdown this thinking. Rather than superimpose computer-type experiences into our physical spaces, the opportunity is to extend our physical spaces using computing power. And it can be by small increments. Whatever the best application for Surface in 2009 looks like, I can tell you one thing: it won’t look like a Windows or Apple application. It will be seemless and natural, it will confuse and blend physical and virtual. It will encourage multiple people to use it at once. And you’ll be able to put a big pile of cakes on it.

And you can’t say that of your iPhone, your computer or your coffee table.

(Let me know if you’d like to come and see it at our offices).