Testing that it Works
While researching and developing our film inspector and Waypoint system, we also had to research, design, and develop how we would test that it works.
There’s a scientific part and a psychological part to that. The psychological part may be the most difficult. What does it mean when we say that it works? What do you think it means? What does somebody else think it means?
I can’t settle that question in a blog post, and maybe it doesn’t need to be settled at all. Maybe it needs to go on and be debated endlessly. Think of how much more we know about film today than we knew 50 years ago. Think about how much better we have gotten at discussing what we know, sharing what we know, and learning from each other.
The scientific part is a little more straightforward. Film is plastic and it has a shape. It had an intended shape when it was manufactured. There are standards that were written by moving image engineers and scientists. There are ways to measure plastic and numbers to compare it to.
This blog post isn’t the place to explain all of the measurements that our system makes, we’ll get to that elsewhere. But I can explain how we test, and how we decide that it works.
There are two families of measurements and two main criteria of success. Measure damage and shrinkage. The criteria for success are don’t miss statistically significant damage or shrinkage, and don’t invent them where they don’t exist. The latter is called false positives.
FI-16 takes the photographs and Waypoint makes the measurements. Both are important, and Waypoint is counting on the FI-16 to do its part right. FI-16 does its part by taking the best pictures it can of the film in high illumination, consistent exposure, and consistent white balance.
Waypoint then uses computer vision, much like the computer vision that guides autonomous vehicles and thousands of other autonomous devices that need to understand what they can see around them. Except instead of trying to recognize moving vehicles and pedestrians in crosswalks, Waypoint’s computer vision looks for what shouldn’t be there.
So we took a reel of 16mm white leader film and measured it by hand. We counted every sprocket and every frame on a synchronizer. We took a hole punch and punched out a half circle on the edge in between sprockets at every foot, that’s once every 40 frames. Then we counted everything again.
Then we inspected that leader film hundreds of times. Waypoint should find every hole punch, and not invent any false positives. Waypoint exported its results to a database. We gave the database to a PhD in statistics and asked him to give us statistical analyses of that database. It works.
Then we took an old and wavy film of the maximum length that FI-16 can hold, 2,000 ft. We inspected that film by hand with a synchronizer, magnifying glass, bright lights, you know the drill. We noted every damage and we took photographs.
We’re analyzing that film as of this writing. It’ll take a while. It’ll be interesting to see if anything changes after we run it a couple hundred times. FI-16 is very gentle on the film and won’t hurt it, but it will be flexed on and off its reel. Does that do anything? Stay tuned, we’ll let you know.
We’ll keep devising new ways to test that it works. We’ll keep expanding on what it means to say that it works. This never ends.
Measurement, science, statistical analyses of large databases. That’s how we do it.