👋 Howdy folks!

I’m Digital Dave back with another issue of Bocce Labs. If you aren’t yet subscribed, you can do so here:

Welcome to the 1 new subscriber this week!

Last week, I shared my thoughts on how I can accelerate the development of the various bocce projects that I have going. I made a listing of all my current projects, their statuses, and prioritized them according to a plan I set two weeks ago. You can read last week’s issue here:

Maybe you’re thinking to yourself, Tech project acceleration is great, but I just wanna accelerate the development of my bocce skills. Well in fact, you can. Hire Andy Zimmerman from American Bocce Co to teach his ABCD (American Bocce Core Development) class at your club.

I like to think that I’m one of Andy’s best students since I’ve taken the course 3-4 times for reinforcement. Each time Andy drops something that helps me even more. Or maybe my brain just makes some new connections each time. Or maybe I’m just bad at sports 🤷‍♂️. Andy will get you rolling straight no matter what: [email protected].

This week I’m going to tell you about a challenging project that I intentionally pulled the e-brake on — Synthetic AI Data for Bocce.

The Problem

When I first embarked upon working with cameras for bocce scoring and stats, I knew I needed data.

I knew I needed data from many different scenes if I was going to be able to train AI models that generalize well.

Consider all the courts you’ve ever played on:

• Some are inside

• Some are outside (in various weather, cloud coverage)

• Some have shadows (from trees, poles, people) or sunrays (shining through windows or between leaves on a tree). The angle of the sun makes quite a difference

• Some have hard surfaces

• Some have smooth surfaces

• Some have lines and markings on the court like that damned patterned carpet like at the Las Vegas Nationals

And think more about what’s in the scene:

• Multicolor patterned balls

• Solid balls

• Bocce balls

• Pallino ball

• People

• Referees

• Beer cans

• Ball bags

Now imagine that we want to train a visual AI classifier, object detector, or segmentation model that can do its job and not get confused if encounters any and all of the above or even something else.

AI models rely on lots of training data and then sending that data through the neural network in batches again and again in order to tune the neuron weights.

I’d be collecting data for years if I wanted to account for all of the above. Plus collecting data of bocce scenes is a challenge when it comes to mounting cameras, etc. Like I’ve said before, I’d rather be playing bocce, and honestly I’d rather that someone else builds a bocce stats system than me. Collecting data isn’t something that’s especially fun (well minus the opportunity to play while the cameras roll).

While that seems straight forward, now imagine that you need to annotate and tag all of that data. Yep, if you’re training a ball object detector, you need to mark/annotate all the balls in the image. And more than that, you might also need to intentionally mark things that are not balls (maybe round drink cups imaged from above that are approximately the same diameter) In all 1000s of the images from the scenes you collected.

Of course we can cheat a little with data augmentation (i.e., random shifts, zooms, shears, crops, salt/pepper noise, and other mutations) of the data. That will help us make a larger data set — one that a model will generalize to better upon training.

But that’s going to take a long time.

The companies that are doing well in the AI world either:

1. Collect lots of data and annotate it

2. Borrow lots of data by crawling the internet and annotate it (sometimes there are legality concerns here, but hey, I haven’t seen OpenAI/Microsoft/ChatGPT get in trouble yet)

3. Or create synthetic data

Synthetic Data for Bocce

When you go back and imagine all the combinations and permutations of the bocce scenes and items in the scenes, you might wonder what if we could generate synthetic data using 3D modeling and software.

Almost like a video game, many companies these days rely on synthetic data for training their visual AI models.

Creating scenes can be similar to creating video games. Then you programmatically change the configuration and objects with some level of randomization around all while they obey the laws of physics that you enforce in the simulation.

And remember annotating/tagging the data? Well since the computer generated each object in the scene it knows where/what it is and it can annotate and tag everything for you!

You can simulate sunlight, moonlight, artificial light. You can simulate clouds, rain. You can simulate anything.

And you can place cameras anywhere in the scene. You can place virtual cameras overhead or on a pole on the side of the court at an angle. Both are good for mapping coordinate systems.

In fact, there are many cases where synthetic data has been used to train AI models and ultimately the results were better than with real data.

Working with Rendered AI

I went down the path of such a project aided by a Seattle-based startup called Rendered.ai. They’re mostly known for creating military/government synthetic data. The government pays a pretty penny for it too. It is extremely valuable.

Luckily for me, they worked with me as a beta user and they didn’t charge me. Under the agreement, I gained access to their codebase and would have eventual access to their cloud system for generating the data using computing clusters.

It was a fun project. I learned a lot about Blender 3D — an open source tool for 3D animation/video games/scene creation. Blender has a Python API.

A huge thanks to the following people at Rendered AI who made this project fun and a great learning experience:

• Nathan Kundtz - Chief Executive Officer

• Tom Freeman - Head of Customer Innovation

• ____________ - The Helpful Engineer that I worked with weekly to which I owe many more thank-yous for answering my novice Blender questions (misplaced his name)

If you need synthetic data for your computer vision/AI project, definitely track down Tom Freeman on LinkedIn.

Challenges as a beta user and beginner 3D designer

The main problem with this project was that it was beyond my expertise. I’m not a 3D designer. I found operating Blender to be challenging and time consuming. As you can see above — my renderings were okay, but there was a lot of work left to do — a lot of code to write and a lot of things to learn.

Furthermore, on my computer it took about 8 minutes to generate a single 2D image from a 3D scene. I’d make a code tweak and have to wait 8 minutes each time. It was tough.

Creating the scenes and images in the cloud is much faster, but before you create the dataset with that large expensive compute, you usually work locally.

I think I latched onto Rendered.ai a bit before they were ready for folks to generate their own data. If I approached them today, it would require less coding and more pointing and clicking with a mouse on their webapp.

Pulling the plug

Anyways, I pulled the plug on this project and came up with other solutions and/or kicked the can down the road on some of my projects.

Needless to say, I have trained a few successful AI models for this project and even accomplished it without utilizing synthetic data. I’ll certainly share them on this blog.

See you on the courts

Thanks for reading!

See you on the courts soon.

~ Digital Dave