11. The Problem
• Make sense of Z and
RGB data
• Where are the hands?
• What are they doing?
10
12. The Problem
• Make sense of Z and • What is the user trying to do?
RGB data • Select?
• Where are the hands? • Grab?
• What are they doing? • Drop?
• Scroll?
• Flick?
• Do nothing? 10
13. The Architecture
Mouse/KB Javascript
Signals Events
JS
CV Web JIRA
Process Browser Plugin
REST
11
Hi guys, my name’s Shihab and I’m the team lead on the integration team at Atlassian.\n- I'm not sure if you recognise me from this morning without the Darth Vader outfit\n- But I’m here to talk to you about the cool things we’re doing with wallboards\n\n\n- In the next 10 minutes or so I’d like to focus on some of the motivations behind this project, what it’s like under the hood and where we can take it to.\n
If you wind back the clock one year, we had a big schpeil on information radiators at summit. \n\nHow many of you guys have an information radiator in your office? -> Sweet\n\nThese radiators help you effectively communicate the status of your project both within and outside your team.\n
Things like build breakages, cumulative flow and even the number of days left in your iteration could be up for everyone to see.\n\nWe have these wallboards up all around the office at Atlassian, to the extant that even our product manager now comes up to me and complains about the state of our red builds - like it's his problem!\n
Back in November, we ran a competition to discover which team around the world had the most ultimate wallboard. \n\nThere were a few categories for prizes last year, including the "best old school" wallboard and the community favourite\n\nBut the one that caught my eye was the ultimate wallboard winner: the Vodafone Web Team from Copenhagen.\n\n\n
If you haven’t seen this before, I definitely recommend you check out their video.\n\nThese guys move physical RFID cards on their card wall, which causes issue transitions in JIRA and the resultant burndown chart is projected onto the screen.\n\nThis blew my mind: _interactive_ information radiators using cool bits of technology. \n\n
Around the same time last year, a new piece of tech was opened up thanks to a hacker who happened to be analysing USB traffic between an XBox and a Kinect.\n\nHow many of you guys have played with a Kinect? You know those motion based-games like dancing and fighting?\n\nYou probably think the Kinect is an advanced piece of technology:\n- You've essentially got a webcam, which is no more powerful than the ones you have on your phones today\n- but then you have this interesting pair of components: an IR transmitter and a receiver - which calculates the relative depth of pixels in the Kinect’s field of view.\n
If you think about a photo, you know it’s composed of many pixels and each pixel is essentially a colour value. What Kinect gives you is the relative distance each pixel is from the Kinect.\n\nNow this depth detection is what sets the Kinect apart. It makes it significantly easier from a computer vision perspective carve out the image into blobs and perform detection and tracking.\n
All this tech immediately spawned a dream: wouldn’t it be cool if you could interact with a wallboard, just using your hands?\n\nThe back-of-a-napkin design for this was simple: a Kinect would be hooked up to the wallboard and a user would be able to enter a “hot zone” to start playing with it.\n\nNo need to print QR codes or mess with RFID tags - just walk up to the wallboard and do your thing.\n
If you think a little bit deeper, you can imagine what it’s going to be like at the low level: we need to make sense of the Kinect’s output to determine where the hands are in the stream of zero’s and ones.\n\nOnce we’ve got that, at a slightly higher-level, we want to try and work out what the user is trying to do. Are they trying to select, grab, flick, or even do nothing?\n
If you think a little bit deeper, you can imagine what it’s going to be like at the low level: we need to make sense of the Kinect’s output to determine where the hands are in the stream of zero’s and ones.\n\nOnce we’ve got that, at a slightly higher-level, we want to try and work out what the user is trying to do. Are they trying to select, grab, flick, or even do nothing?\n
This diagram shows you the rough architecture of our system.\n\nWe have a computer vision process (C/C++ code) that chews through the raw Kinect output and determines what’s going on: has someone approached the wallboard, where are their hands, have they walked away?\n\nThis information is passed on as mouse movements and keyboard signals via the operating system. So basically, we’re emulating OS input. The web browser forwards these on as Javascript events to our JIRA Plugin - essentially some web resources and REST resources - which then interprets these events and updates Greenhopper and JIRA accordingly.\n\nEnough talk, let’s see the Kinect in action!\n
So let me just switch over to show you what the computer vision process can see.\n\nThere's four quadrants under the hood, and I'll go through each of them.\n\nFirst up, in the top left quadrant, you have the colour image from the Kinect's webcam. We don't really use this information for computer vision, but it's useful to visualise what's gong on.\n\nNow in the bottom left quadrant, here's where it starts to interesting. How many of you guys have seen the movie predator? Well, it's nothing like that. This is a representation of depth information from the Kinect. I’ve encoded near pixels with warm colours and far pixels with cooler shades.\n\nSo what can we do with this depth information?\n\nIf you look at the bottom right quadrant, you can see that I've set all the pixels to be off. Now watch what happens as I walk towards the Kinect. There's this imaginary plane over here - if you are nearer than this plane to the Kinect, the pixel lights up. We can use this to detect if the user wants to interact with the Wallboard, or is just passing by.\n\nUser Enter. User Exit. The CV process passes on the state changes via keypress events to the UI.\n\nSo that's detection of the user - but what about the hands? Well, it turns out, we can apply the same approach and define a hand plane to work out if a hand present. We can go one step further, and approximate where the user is pointing by looking at the centre of this hand blob. On the top right colour image, the centre is shown as a green dot. We can easily pass on this information as mouse co-ordinates.\n\nThe final piece of the computer vision at play is detecting the whether the user's palm is opened or closed. So as you can see here, my palm is wide open - pause. If we approximate the polygon around my palm you can see it's a kind of a star shape, with pointy bits (my fingers) and inverted pointy bits (the gaps between my fingers).\n\nIt turns out that it's possible to reliably identify an open palm by looking for these inverted pointy bits. So as you can see on the top left colour image, the computer vision process is drawing blue dots over the gaps between my fingers. Open palm, closed palm - dots are gone.\n\nNow let’s flick around to the UI to see the computer vision in action. The passive wallboard cycles through wallframes every 30 seconds. However, if I step up to the wallboard, the CV process detects I’m there and invites me to interact. \n\nNow, I'm in control. The UI is set up to listen for fast mouse moves. I can swipe at the wallboard to cycle through the wallframes at my leisure to navigate to the information I'm interested in.\n\nWe needed to come up a way to assist users when they start interacting - so, we implemented visual cues to let them know their hand is off the edge of the screen, or that their hand cannot be found because it's not near enough to the Kinect. We've tried these cues out in practice and we've found it to work pretty well.\n\nMost traditional wallboards are a wall of cards. If you download Greenhopper today, you can view your taskboard as a wallboard, just like this - but you can't really do anything with it.\n\nThere’s a psychological sense of achievement when you physically progress an issue that you’re responsible for, from Todo to Done - and that’s something we looked at trying to replicate.\n\nIn terms of UI, we played around with the idea of “push to click” but found it was unreliable, so we implemented a “hover to click” for clickable elements.\n\nNow we let the system know who we are, and then find the issue we want to transition. Say we've had a very productive week, and we've finished all our issues. Once we stop playing with the taskboard, we can flick over to the burndown chart, and see that it's been updated live.\n\nAnd there you have it - an interactive information radiator - just by waving your hands :)\n
This isn’t the end - this is just the beginning. Go home, take the Kinect away from the kids and plug it into your wallboard. We’re making the source available for anyone to download, right now.\n\nThe code is in a tech demo state, so there’s no double-click installer. Once Microsoft releases their official SDK, we’ll look at wrapping it up into an installer or browser plugin. We’re also keen for the next level of innovation - perhaps expose for wallboards, ability to assign issues to avatars, to do adhoc estimation with your hands, or even face recognition for authentication!\n\nWe invite you to join us on playing with awesome tech, developing cool plugins and having loads of fun!\n
