Neural Networks for Generating HDRIs

Deep Learning is the new wizard tech that turns previously unsolvable problems into merely tricky ones. It gives computers the ability to come up with a plausible guess - a feast that previously required human intuition and experience.

One problem, that requires VFX artists to guess all the time, is to match real-world lighting just by looking at the filmed footage (a.k.a. the plate). You're probably saying "Wait - that's exactly what on-set HDRIs are for!" and you would be correct. But the real world is messy. Maybe the director never got the memo that he should invite a VFX supervisor to the set, maybe said supervisor was too distracted chatting up the costume designer instead of shooting HDR images, or maybe the director is building his entire commercial clip just out of stock footage. Maybe the HDRIs were even shot on set, but somehow got lost in the complicated jungle of client-vendor communication protocols, involving multiple levels of middle men. The sad fact of life is that a scary amount of CG artists have to match lighting by hand, still to this day.

Now what if... a neural netwok could automatically generate a plausible HDR image directly from the plate?

NeuralVFX



A rogue VFX supervisor under the pseudonym NeuralVFX figured this was worth a shot. He managed to train a neural network by creating a huge pile of randomized HDRIs and a corresponding render for each. Based on those example pairs, the magic black box learns to predict an HDR image when you show it just a render by itself.


Pretty remarkable, huh? It really picked up on the overall shape of the lighting. The hues are wildly off —- his network seems to prefer neon green and pink, like a true 80's retro hipster. Well, that might just be a juvenile phase, possibly caused by feeding the algorithm too many psychedelic example HDRIs during training. But the principle itself looks promising.

The third step is the real deal: After training and validation, now let it guess the lighting of a plate! And this is where Mr. NeuralVFX really caught my attention:


You can see a much better explanation of the process, along with a few more example images, on NeuralVFX's original blog post:

http://neuralvfx.com/lighting/neural-network-for-generating-hdris/


JFL at Université Laval


It's amazing what a dedicated tinkerer can do with modern AI tech.
But what about serious research professionals? Like, for example Associate Professor Jean-François Lalonde from the Computer Vision And Systems Lab at the Université Laval in Quebec:





This is a problem JFL et al have been working on as far back as 2009. He started with algorithms that detect and analyze shadows in an image, take cues from the brightness gradient in blue skies as well as lit/unlit areas of vertical surfaces. His publication list contains several gems, all worth reading. But in his more recent work he takes the same brute-force approach as above: training a neural network do its magic.
With a few differences:

  • Training is done with a database of 2.100 real HDRIs and 14.000 real LDR panos.
  • Instead of Render+HDRI pairs, he trains using cropped Plate+HDRI pairs.
  • RGB color and light intensity maps are predicted separately.
  • Optionally an existing HDRI can be warped for to compensate for spacial variations in lighting (this point certainly warrants a deeper examination in a later blog post).


Because the AI now has a very clear understanding of what real HDRIs look like, the predicted results are rather spectacular:



They even confirmed the stability of this trained network by downloading a bunch of random stock photos and putting stuff into it —- lit fully automatically by the predicted HDRI:


Go check out the project page, it's filled with goodies! You get the paper, slides of the talk, many more examples, even the training set is made available for all the tinkerers:
In a more recent move, the group also published their code on GitHub. And for the artist-types without coding experience of their own, there is also an Online Demo of their AI. Just upload your own plate and get a predicted HDRI for download. It's technically only certified for indoor scenes (as this is what the AI was trained on).

Let's give it a test drive


I figured the Thunderdome is technically still considered indoor, right? I mean, it has neither doors nor walls, but that's kinda the point of the Thunderdome!!!
So here's what it made for me:



Hm. Looks a bit potato quality, but that should only really matter for reflective materials. Smart IBL has shown that tiny blurred images are perfectly fine for diffuse lighting, so let's bring it into Modo. Turns out the HDRI ended up in camera space, of course. That means it takes a bit of counter-rotation to compensate for the camera angle, as well as a little extra intensity boost. But that's pretty standard procedure, we have to do such little tweaks with any HDRI.



Not bad at all. No extra lights in the scene. The color tone and backlighting are pretty well represented. It's not a very scientific test, as my troll Mica is naturally a bit orange and his mossy coat does not take on diffuse light too well. Lighting the scene by hand I would probably go for more stark contrasts and harsher shadows. But it would probably take me longer. For an automatic result that's nothing to sneeze at.

Definitely good enough to run a quick comp:

Crushin' it at the Thunderdome

So yeah, I'm a believer now. Neural networks can indeed hallucinate useful HDRIs together.

Next Page