ARPool (Augmented Reality Pool) was a project I was always very interested in getting involved with so when my supervisor mentioned he need a few of us to get the system ready for a demo I jumped on the opportunity. We were going to be taking ARPool to the Ontario Centers of Excellence Discovery Show – it is kind of a mix between a trade show and a conference poster session. The Queen’s marketing guys were also super excited because they felt Queen’s was out done the previous year and ARPool was our show stealer.

The first thing we had to do was update the travelling version of ARPool, what we needed was an enclosure to keep it dark ish around the system – if it is too bright then the augmented reality projections onto the table are tough to see. We also needed to build a new mount for the projector and camera, our supervisor wanted to try straight projection rather than off of a mirror so it needed to be higher up. We experimented with a few ideas and set up the system in Beamish Monroe Hall and ran a successful demo for Queen’s Admitted Students Day.

At this point I hadn’t had too much exposure to the code behind ARPool but it was already clear that the system was not what I would call rock solid. Near the end of the demonstration the ball detection algorithm was not finding all the balls on the table – fixing the ball detection algorithm would be my first contribution to ARPool. Having learned that testing new algorithms on a live system lacks the quantitative feedback that is necessary to get the best results I took a handful of test images and headed back upstairs to my lab to tackle the issue. A quick look at the old detection algorithm and it was easy to see how the system was suffering from having multiple inexperienced developers. The algorithm went through 4 iterations of finding contours, then drawing the contours to a new image albeit with some filtering each time (area, shape etc.) and then running find contours on this image. I would sincerely hope that the find contours algorithm can find the contours you just drew…

I rewrote the ball detection routine, tested it on my data set and then when I was satisfied brought it back downstairs, ball detection back online. Next I tackled the Background subtraction code – it was actually pretty good but I was able to speed up save and load times dramatically by switching to cv::FileStorage from plain text. We added a simple smoothing filter to the cue tracking just to make the system appear less jumpy to users. The system was working pretty well although there were a few design choices I didn’t care for I thought we were in good shape for OCE.

We packed it all into a single cube van which was seriously pushing it space wise and took off for Toronto. We spent most of the first day schlepping everything from the parking lot to the conference hall and then setting up the structure and pool table etc. We started into the calibration process and while it wasn’t smooth it wasn’t much worse than usual. Calibration finished enter the big problems… We load up the version of the code we had finished with at Queen’s and it isn’t working – wait what this code just worked!? Things are happening out of sync and we can’t figure out why, this is where my frustration with the way the system is set up begins. We eventually load up the code from before we fixed all the various algorithms and it is at least running in the right order… so we slowly add back one feature at a time until the system is working as we left it. I don’t like this – we never figure out the problem and this seems unstable. Oh and by the way it is now the next day at 1:00 the event is starting we got it running just in time. The system runs great for 4 hours straight we’re all a little surprised frankly. We demo to a lot of interested people mostly other demonstrators at the event (I’m still not totally sure I really understand the point of this event). We had back to the hotel after the reception which by the way was pretty fun oh and did I mention the Queen’s put us up at the Royal York (it’s pretty swanky). After dinner we went to a Jay’s game to relax for the evening.

The next morning (and this is the more important day of the show) we arrive and boot up ARPool – aaaannnd it’s not working today seriously this system is moody we ran it 4 hours straight no issues yesterday and now it’s deciding not to work again. I don’t remember exactly how but I do remember that I was the one who fixed it – I added or changed some piece of code that was completely trivial and then the system started working again – hmmm whatever I just want to get through this day now.

Got the system up and running and the day was a success – pics or it didn’t happen!

So what was the issue? Well ARPool is an event driven program written in Qt using signals, slots and the works including several timer events. Our best guess is that sometimes they would get our of sync. We had worked really hard on improving the algorithms but what really got us in trouble was the system layout and how tough it was to see and control the order code was executed. The higher level stuff was really poorly organized and documented for example there were 2 functions that were very key to the operation one called “run” the other “run_again” uh yeah that is not how we do things.

All in all a successful but certainly not stress free trip. I vowed to fix the system layout of AR Pool before our next demo because I did not want a repeat of OCE - that and I’m an engineer and I have the problem solving bug and this was a problem screaming to be solved and I already had an idea how…