What’s in your Vault?

What’s in your vault?

Obviously that’s a play on the funny television commercials from a bank that asks “What’s in your wallet?”

If you’re joining us on our mission to make better and faster film inspections and condition reports, then we know what’s in your vault. Films. Lots and lots of films.

You have a cataloging system so you know a lot about those films. But what do you really know? Do you know what condition they’re in? Every foot and frame of them? Do you know what they might look like on that computer in your pocket?

When those films were put into cans and stored in your vaults, there was paper money in your wallet. Times have changed. Now there’s an electronic credit card in your wallet, and maybe not even that. Maybe you’re like me and you use the computer in your pocket.

What’s in your vault? Let’s find out together. With all the powers the computers offer us today.

Object and Facial Recognition for Film and Images

No, we don’t do that. But we know that someone will.

We’re living in a time when computer vision tools are exploding in power, popularity, and use. Sometimes we know it, sometimes it’s in the background and invisible to us. That computer in your pocket already does it.

We made a list of computer vision object recognition facial recognition softwares, just the highlights that we think are most relevant to film archives. Just ask and we’ll give you a copy. We made it for our own education and we’re happy to share what we learn.

Our film inspection products, FI-16 and Waypoint, take photographs of your films so our computer vision tools can analyze their condition and produce condition reports for you. Our products also make video workprints.  This means that if you choose to send to recognition software or online services, you have video and still images to choose from. Or send them both, or send them to different softwares and services. Fun, eh?

There isn’t one best software or service and they’re probably never will be. It’s a Pandora’s box and that’s a good thing. Try them and see how you like them. Keep using the ones you like best.

We can tell you this about some of the key differences that you’ll encounter.

Some of them are optimized for video. Knowing that the source is video let’s the recognition software infer important things about the images in sequence. If a face is recognized in one frame and repeated for many more frames and then begins to turn around showing the back of that person’s face, video optimized software can figure out that it’s still the same person. If you feed thousands of still images to a recognition software and it finds a face, it won’t know that the back of that person’s head is the same person.

Video optimized recognition software can also use the soundtrack. Take the interview format for example. The camera may be on the interviewer when they ask the question but when the shot cuts or pans to the interviewee, the interviewer is still there. Humans know that intuitively. Video optimized recognition software is figuring that out too.

There can also be benefits to recognition software analyzing each frame separately. For example they could take a quick first pass through a long sequence of images and take note of which frames it can analyze the best. This will depend on computer vision factors like edges, exposure, and remember that some edges are contours of colors, not black or white, not contours of contrast.

After identifying the faces or objects in those key frames, the image optimized recognition software can then use those as a base for getting better results from all the other individual photographs.

So if all of that makes sense to you, there’s a third approach. A hybrid approach. Start by analyzing the video, but first break it out into thousands of individual still images. Then use all of the first two approaches, and compare the results to learn even more.

If it sounds repetitive, complicated, or even mind numbingly boring, please remember that that’s what computers are good at. That’s why we invented them. To do that work for us. Preferably while we are doing something else more interesting to us, things that computers simply cannot do. For example, computers don’t have social skills or empathy. That’s what humans are good at.

And that’s why we employ video everywhere in our products. Humans are incredibly sensitive to body language, motion, anything that moves or relates to human behaviors. During a film inspection the machine presents the photographs on the user’s computer screen and it looks just like video. It’s really still images going by so quickly that your brain will interpret it as video. Then your FI-16 can make a video work print, compressed so you can watch it on your computer in your pocket or any computer as far away as you’d like to send it. Waypoint overlays computer vision measured analysis in graphic form on copies of the still photographs and in video workprints that it makes from them. Again with the computer in your pocket and the computers as far away as you’d like to send them.

Object and facial recognition are wonderful tools and humans will always discover things they don’t. Work together on this. The important thing is to inspect your films, know what condition they’re in, use your judgment to apply your resources to preserve them and make them accessible. Let the computers do the boring parts.

Storage Wars

I wonder why there’s no storage device or cloud storage named Pandora’s box? Moving image archivists study storage of all types. They need more of it than most people. They need more kinds of it than most people. Film in cans in cool vaults. Digital storage of every conceivable type and price point. And they’re not just thinking about storage for convenient retrieval, some of them are thinking about storage for centuries and beyond.

There’s no single right answer and even if there was, it would change by the time this blog post reaches your eyes. But I can tell you a few things about what we are throwing into the mix.

FI-16, our film inspection device, has some storage inside of it. Not a lot, just enough to capture the photographs it takes of your films. That is stored in non-volatile RAM, about a terabyte of it. After a few hours of film inspection you’ll want to move those photographs and the video workprints that FI-16 makes from them onto their next storage.

Waypoint, our film inspection computer vision and condition reporting online tool set, is hosted on Amazon Web Services. Waypoint only needs the photographs long enough to analyze them with computer vision. So that leaves a lot of options to the user.

You could connect your FI-16 to the internet and upload the photographs and video workprints directly to Waypoint. Waypoint makes more video workprints to illustrate the film condition on a frame-by-frame basis. We’ll keep those for you on Waypoint along with the inspection data that Waypoint makes for you.

You can download all of that to your own storage. You can copy it to some other cloud storage if you like. Just do that thing you do.

Or you could connect your FI-16 to your local area network and move the photographs and video workprints there first. We have an uploader program that you can install on your own server to copy the photographs to Waypoint over the internet.

After Waypoint has analyzed the photographs you could pay us or Amazon to store them very cheaply in Amazon Glacier. Or not, that’s up to you.

How many copies do you need? You could take the belts and suspenders approach and keep all the copies ever made. You could take the efficiency and preservation approach and keep the photographs and video workprints where you keep all your other digital files and backups.

Sorry for opening Pandora’s box but moving image activists are used to it by now. All we can do is make it easy, reliable, auditable, efficient, and as fast and cheap as possible without giving up those values.

And we will keep staring into Pandora’s box for as long as you do. Film inspection and moving image preservation and access is a long game. In fact I would say it’s an infinite game, the kind that gets better the longer you play.

Printers Propellers and Platters Oh My!

When we decided to start R&D on a new way to inspect films, we gave ourselves a blank slate. All we knew for certain was that we would take advantage of the latest technology including computer vision, computer controlled motors and transport, and the latest in materials science.

One of our pivotal choices was whether or not to use 3D printing 🖨️. There are pros and cons. Traditional methods for making metal and plastic parts are very precise. Precision is a good thing. 3D printers have a head that moves in four dimensions, melted plastic that comes out of it, and a platter to hold that plastic as it hardens, and that platter slowly moves as the plastic grows. It can’t be as precise as a laser cutter, as you might well imagine.

But speed matters too. Mostly in experimentation and trying out new ideas to make something work just a little bit better. So we chose 3D printing for some of the elements on the outside surface of the FI-16, things that we could continually improve and mail out to our customers.

Surprise!

If a film inspection station holds the film horizontal or nearly horizontal – FI-16 is about 15° off of horizontal and that’s enough to give a small amount of vertical force – then that raises the question of whether or not the film needs a platter on both sides. Gravity holds it down to one platter. So do you need a top platter?

Well the answer turns out to be – sometimes. Archival films do what they want. They don’t obey orders. So, does the top platter need to be the same as the bottom platter? The answer turns out to be – Yes or no. Sometimes.

So we ship an FI-16 with three bottom platters because the third one can be a top platter on the take-up reel if you need it. You rarely do but when you do, it’s just the thing. Then we kept experimenting with different kinds of top platters using our 3D printer.

Some of them are just like the bottom platter but made of plastic and therefore lighter. Easier to handle. Cheap to replace if you break it. Some of them have a gradual curve on the side that faces the film because the take-up reel film may want gentle nudging in the beginning and then not need it anymore after the film gets going. We call that one the UFO!

Then came the propeller.

Sometimes you want to nudge or wind the film by hand. You want to rotate the film on its platter and you want to see it while you do. So we started cutting away more and more of the UFO until it was just two straight bars in either direction. Then one of our film inspection operators said it was finicky to place her fingers on the thin plastic. Rather than make a thick chunk of plastic, we curved one edge of the bar upwards. Then we curved it upwards more. Now she can get a really good grip and she can see all of the film. And it still does its job as a top platter to keep the film winding nicely.

When we took it off the FI-16 and looked at it by itself, I swear it looks like – a propeller. With the spindle hole in the center it looks like you could stick it on your model airplane and send it off in the sky.

Printers, propellers, and platters, oh my! Hey, have a little fun while you work.

Testing that it Works

While researching and developing our film inspector and Waypoint system, we also had to research, design, and develop how we would test that it works.

There’s a scientific part and a psychological part to that. The psychological part may be the most difficult. What does it mean when we say that it works? What do you think it means? What does somebody else think it means?

I can’t settle that question in a blog post, and maybe it doesn’t need to be settled at all. Maybe it needs to go on and be debated endlessly. Think of how much more we know about film today than we knew 50 years ago. Think about how much better we have gotten at discussing what we know, sharing what we know, and learning from each other.

The scientific part is a little more straightforward. Film is plastic and it has a shape. It had an intended shape when it was manufactured. There are standards that were written by moving image engineers and scientists. There are ways to measure plastic and numbers to compare it to.

This blog post isn’t the place to explain all of the measurements that our system makes, we’ll get to that elsewhere. But I can explain how we test, and how we decide that it works.

There are two families of measurements and two main criteria of success. Measure damage and shrinkage. The criteria for success are don’t miss statistically significant damage or shrinkage, and don’t invent them where they don’t exist. The latter is called false positives.

FI-16 takes the photographs and Waypoint makes the measurements. Both are important, and Waypoint is counting on the FI-16 to do its part right. FI-16 does its part by taking the best pictures it can of the film in high illumination, consistent exposure, and consistent white balance.

Waypoint then uses computer vision, much like the computer vision that guides autonomous vehicles and thousands of other autonomous devices that need to understand what they can see around them. Except instead of trying to recognize moving vehicles and pedestrians in crosswalks, Waypoint’s computer vision looks for what shouldn’t be there.

So we took a reel of 16mm white leader film and measured it by hand. We counted every sprocket and every frame on a synchronizer. We took a hole punch and punched out a half circle on the edge in between sprockets at every foot, that’s once every 40 frames. Then we counted everything again.

Then we inspected that leader film hundreds of times. Waypoint should find every hole punch, and not invent any false positives. Waypoint exported its results to a database. We gave the database to a PhD in statistics and asked him to give us statistical analyses of that database. It works.

Then we took an old and wavy film of the maximum length that FI-16 can hold, 2,000 ft. We inspected that film by hand with a synchronizer, magnifying glass, bright lights, you know the drill. We noted every damage and we took photographs.

We’re analyzing that film as of this writing. It’ll take a while. It’ll be interesting to see if anything changes after we run it a couple hundred times. FI-16 is very gentle on the film and won’t hurt it, but it will be flexed on and off its reel. Does that do anything? Stay tuned, we’ll let you know.

We’ll keep devising new ways to test that it works. We’ll keep expanding on what it means to say that it works. This never ends.

Measurement, science, statistical analyses of large databases. That’s how we do it.

Copyrights in the Digital Age

Did you know that if you make a video workprint of a film, you own that video?

I don’t think that was the intention when the US Congress wrote the Digital Millennium Copyright Act. They were thinking about video games and word processors. Then downloading happened.

Film archives don’t want to infringe on the copyrights of others. I don’t recommend it to anyone in any context. But that video workprint? Who has the rights to that?

Consult a copyright lawyer if you like but the answer will probably be a long one. It’s complicated. It’s getting more complicated all the time. A couple years ago I spoke on a panel addressing an assembly of copyright lawyers and government ambassadors from around the world. I stuck to what I know but the real surprise for me was how little anybody knows about what’s next. They’re working very hard and diligently to figure it out. One core problem is that the internet is global but copyrights are local, WIPO notwithstanding.

I don’t know what you want to do with this information. Copyrights are a fascinating subject. But that video workprint you made with your FI-16 isn’t a copy. It’s a new work. It’s video assembled from still photography aimed at a film. If you make a film copy of a film, copyrights cover that pretty clearly. If you digitize a film to make a digital copy of its content, decades of law practice cover that pretty clearly. But a video workprint made on a FI-16 isn’t the copyrighted movie. It’s photography of the plastic. It’s meant for the archivist, to help them do their work well.

What is that?

Real Doesn’t Always Equal Pretty

When we run a film inspection on FI-16 we’re not looking for pretty. We’re looking for real.

The photographs are meant to capture the reality of what that film looks like if you held it in your hands. In really good light. So the photography software has an automatic exposure and an automatic white balance. It’s not as pretty as what you expect to see from an iPhone.

Video editors know that you can turn blacks into grays but you have a lot of trouble turning grays into black. You can dim the brightest whites but you have a lot of trouble making the brightest white out of gray. Why? Because the video editing computer doesn’t know which parts of the image are meant to be gray so they get turned to black and white also. The same is true for colors. Instead of contrast we call it saturation. But the limitations are similar.

When you’re inspecting a film on the FI-16 you see video on the computer screen. That is the actual photography shown to you. Moving at whatever speed you set the film to run. It’s not meant to be pretty, it’s meant to be accurate. So you can see what the film really looks like, what it really is.

Here’s a fun experiment. Next time you’re sitting around a table with some friends ask everyone to pull out their phones 📱. Laptops and tablets too if they have them. Then everybody play the same video on all the screens at the same time. See how different they all look!

We live in the digital age and it’s not going to unwind itself, so to speak. Manufacturers of smartphones, tablets and laptops aren’t trying to make all the videos look the same on all the screens. They’re trying to please you. They’re trying to make their screens prettier than everyone else’s. That’s not reality in video, that’s the reality of commerce.

When you’re inspecting your film, photograph its reality. Pretty comes later.

Editorial Edits & Craft Edits

When a film archive sends films out for digitization they don’t see the difference between editorial edits and craft edits. They just see the final results.

Now that film archives can make their own video workprints alongside their automated film condition reports, they might learn more about the difference between editorial edits and craft edits.

Editorial edits are what they sound like. They’re edits made for content. They don’t contain every foot of the film. Leader, blank or unexposed segments are omitted, etc. Maybe there are extras that were included with the film but not meant for normal viewing. Maybe there are trims that should be edited together, and maybe they should be their own video. Or maybe they belong somewhere in the main video. These are editorial decisions.

Craft edits are artistic in a different way. Maybe it’s a black and white film that has turned brown and needs to be seen black and white again. How black and white? How much contrast? Should it look as if it was developed yesterday or should it look like it was developed half a century ago?

Maybe it’s a color film that is faded and needs to be color corrected. How much? How saturated or vivid should it look? Should it look as if it was developed yesterday or should it look like it was developed half a century ago?

Maybe the film has shrunk or has problems that vary all along its length in varying amounts in different segments. It can be stabilized. How much? Rock solid, or with a vintage look? Modern storytellers sometimes artificially induce a vintage look for artistic reasons to a film or video that they shot this week.

Editorial edits and craft edits are art. There is no right or wrong, there are only choices. Not everyone has to agree on every choice. Fortunately with video choices can be made again and again. Unlike a film print, it’s not cast forever in plastic.

But there is the original photography of the film. That can be done again but the film will be older. The original photography is digital and doesn’t age like film.

We think the original photography of the film should be realistic. Just take a good color photograph of every frame, edge to edge and some of the frames next to it. Let the editorial edits and craft edits turn those photographs into art. While the film rests undisturbed in its vault.

No-prep Workflow

Traditional archival film handling is a workflow. There’s a lot of preparation involved. What if there wasn’t?

  1. First you set up a space with winders and splicers and synchronizers and a light table and a magnifying glass and gloves.
  2. Then you open a film can and load it onto a reel, and another real onto the other winder.
  3. Then you wind the film slowly and take a lot of notes. Footage, frame numbers, visible damage, measures of shrinkage perhaps, edge codes, soundtrack type or none, credits, etc.
  4. Then you wind the film to a specific heads out orientation and put the film back in the can.
  5. Then you carry it to the scanner to make a video of it. Take the film out of the can again, thread it again except this time the rollers and film path are complex and finicky.
  6. Then you turn on the scanner and set the controls (expertise required).
  7. Then you put the film back in the can and take it back to the inspection station and wind it for archival storage again.

What if…

  1. You open the film can and set the film on a platter. Thread it around a couple of big and airy rollers to the other platter.
  2. Type or barcode scan a reel ID and click your mouse to start the photography. Punch a button to start the film rolling.
  3. Watch the video on a computer screen and your mind is free to think about what you see. Maybe take a note or two if you see something remarkable but otherwise just let it run. It doesn’t matter if it was heads out or tail out because the video can be edited later.
  4. When the film is wound to the other platter, simply lift it back into its can, still on an archival hub. Close the can and put it away.

Done. No prep. Open the film can one time, spool it one time, put it away.

Let computer vision measure everything about the film and produce an automatic condition report. At your leisure, or someone else’s leisure, watch the video workprint. Watch it in the comfort of your own desk or on your own phone.

Sound like a good idea?

Tiny Desk Concert

When I started in the music recording business  half a century ago, nearly everything we did involved logistics and heavy stuff. The instruments were big and bulky. The recording studios were insanely expensive and had to be purpose built. They even had special doors and ramps.

Musicians today assume they can do their work anywhere. They’ve got computers in their pockets and in their backpacks. Recording studios are still nice, but not required. They just find a desk and go to work 🎶.

Some musicians can give an entire concert from a tiny desk. What can you do at your desk?

You can inspect films at any desk, and you can choose the one most convenient to the films. Bring your FI-16, pull up a chair and start the show.

We normally imagine the film inspection to be a solo endeavor and the output will be video workprints and condition reports. To be viewed later, like a music recording. Or…

It could be like a tiny desk concert! Everybody is online these days. You could share your screen with viewers near and far at the same time that you’re inspecting. It could be a community event. Or colleagues could join you from their own desk wherever they may be.

Sounds like fun to me.