Driving Fun in Unity with Augmented Reality

Over the past couple of months I’ve had the chance to get in a bit of Unity research and development time, and one of the areas I have looked at is Augmented Reality. By far the most entertaining experiment so far has been my “AR Driving” demo, which I tried out at the Skive London office recently. It could be said that this particular demo isn’t strictly AR (because it doesn’t overlay graphics on the video stream) but still, it uses AR technology to map the orientation of an AR marker to the steering, acceleration and braking of a car. I recently demoed it in our London office, result: Fun!

The control works by reading the orientation of a single AR marker printed on an A4 sheet. The sheet is folder so that the marker is visible to the camera on one side, and a steering wheel image is visible on the other side – so the user knows which way up to hold the paper!

The rotation around the Z axis (as in, turning the steering wheel) is mapped to the car steering, although not entirely linearly – in order to make the controls easy to use it required a few tweaks, such as smoothing out the sometimes jittery raw data, and giving bias to a ‘not-quite-dead zone’ around the zero-degrees rotation area to make it easier to drive the car in a straight line.

The acceleration and braking controls are implemented by reading the “tilt” of the marker around the X axis. Holding the marker straight-on to the camera gives the equivalent of a steady gentle press on the accelerator. Tilting the top edge towards the camera pushes down the accelerator, while tilting it away releases the accelerator completely and applies the brakes.

The final result is a fairly intuitive driving control, and most people who had a try in the office seemed to find the car responsive and easy to control on their first try, as the video shows!

At the present time (Unity 2.6), implementing AR isn’t entirely straightforward, not least because Unity doesn’t have any built-in way of reading video streams coming in from the webcam, so the webcam stream in this demo is being read via a custom DLL, and the AR markers are picked up from the video stream using the beta version of “UnityAR” (also a custom DLL, which provides an interface to ARToolkit) from Weltenbauer who have kindly made their beta version free. An alternative to this technique is to set up a socket connection from a flash movie, and send either the video data, or the AR marker data via the socket to Unity. Neither of these methods are particularly ideal!

To cut a long story short, because of these custom DLLs the demo is a Windows-only EXE at the moment, and can’t work as a webplayer at all. Now, if UT would add support for webcam input in Unity (in a similar manner to flash) it would open the doors for browser-based AR, which is something that Unity could really excel at.

Unity does have a User Feedback Forum, where you can vote up feature requests for future versions – so if you’re interested in this, vote up Webcam input!


11 Responses to “Driving Fun in Unity with Augmented Reality”

  1. I’ve been meaning to investigate the AR->unity combo….thanks for the post;- very cool

    • khassib Says:

      I have a project to capture images through a webcam, so I use Unity and I did not find a way to capture images after the camera for further processing.
      my aim is to detect the movement of some control points in order to move my objects in the 3D scene unity

      Please help me, please

  2. Mark Daggett Says:

    Wow Nen this looks great. You need an orgami airbag now!

  3. Excellent! I still haven’t gotten into the AR stuff yet but I’m enjoying reading about it!

  4. […] had Augmented reality from robotDuck Stop by and SUBSCRIBE robotDuckVideo YouTube Channel and also stop by robotDuck’s blog and drop some […]

  5. Hello mate… I would really like to know how I could use this custom DLL to make the webcam’s streaming possible with Unity3D!

    Can you help me with that ?

    thanks man!!

  6. absolutely cool stuff!!!
    definitely it would be great to have those dlls running on osx and in webplayers too.
    As a flasher I’m really happy for flash to be indispensable in order to have a web or a crossplatform deployment… however I voted for that feature, unity rocks! šŸ˜€

  7. […] by and Subscribe to robotDuckVideo his YouTube Channel and also stop by robotDuck’s blog and drop some […]

  8. […] more about this augmented reality ride onĀ robotduck’s blog:https://robotduck.wordpress.com/2009/11/30/driving-fun-in-unity-with-augmented-reality/ and Stop by andĀ subscribe to robotDuck’s YouTube […]

  9. […] by and Subscribe to robotDuckVideo his YouTube Channel and also stop by robotDuckā€™s blog and drop some […]

  10. kagura Says:

    Great! Very awesome
    Can you share your game to everyone? Everyone’d like to play this great game.
    Thank you very much.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: