How to build a holodeck, Week 32

I’ve been somewhat distracted this week, but I managed to spend a couple of days getting a point cloud feed working over the network. The point cloud feed is very heavily throttled (something like 1K points a second currently) which means the results take a long time to look decent - but it’s working. Apologies for the “lots of words” update, I’ll have some cool videos next week.

Network performance

Up to this point, most of my prototype networking has been about simple data transfer between multiple clients (including voice) without the need for a specific server. Photon has been a great solution for this, as it makes sending to multiple clients over the public internet trivial.

For chucking data feeds around the local network, with one machine acting as a feed aggregator - not so much. There appear to be limits on how much data I can send per serialised object (something on the order of 1.2K if I’m reading the docs correctly, and this matches the error messages I’m seeing serialising about 250 vector3s plus some additional setup data) which is a tiny portion of a pointcloud dataset, let alone a full depth image or similar. This isn’t really a flaw with Photon, as chucking huge amounts of data around over the internet shouldn’t really be a goal for any multiplayer game (you want to send as little data as possible, ideally) but it’s put a bit of a spanner in my current prototype which I’ll not easily be able to work around. Photon’s setup also means that all data is sent from each client to all other clients, which is exactly what I’ve wanted for previous networking prototypes, but not something I need for this particular test.

Likely, the best solution here is to go back to basics, set up some server code on the feed aggregator machine, set up some client code on the other feed machines (Kinect machine no 2 and the Tangos) and (given that I’ve not got any data or latency constraints on my local network) just throw as much data around per frame as I can, from each client to the server. That’ll be at least a few days work.

Camera pose calibration

I’ve got my two Kinect cameras now pointing into the garage space, each giving a different view into the scene. The Tango pose is already “calibrated”, in the sense that it knows where it thinks it is, and where it’s pointing, in the world space. The calibration here is done when the initial Tango Area Definition File is created, which defines the origin and orientation of the tracked space.

For the Kinect cameras, I’m effectively manually guessing their position and orientation. I’ve written a helper app on the Tango so I can line the Tango up with the Kinect, tap a button and see where the Tango thinks it is, and then use this as the Kinect’s pose position/orientation, and that looks “good enough” right now. This breaks if I move the cameras, or the camera gets bumped, and it’s very susceptible to mis-calibration (even being off by a degree in one axis means an error of 7cm at 4 metres distance, close to the far plane of the depth capture on Kinect). And my manual calibration error is much larger than that.

I’m going to need some solution to properly calibrate camera poses, either by locking on to a known marker in the scene, or some kind of clever math once I’m roughly in the right place and start getting a data feed coming in. This stuff is unknown to me at this point so - lots more research here.

Once the camera poses are calibrated to a decent degree, the end results of the aggregation process should be pretty cool.

Next week …

Fingers crossed, back to it. If I can fix the network performance and calibration issues, the next step after that is going to be putting the texture feed on top of the depth / pointcloud feeds.

Written on October 2, 2016