When we look at the cover page of a magazine,we often see lots of well-made, but also idealized photos of people. Idealized here means that the photographermade them in a studio, where they can add or remove light sources and move them aroundto bring out the best from their models. But most photos are not made in the studio,they are made out there in the wild where the lighting is what it is, and we can’tcontrol it too much.
So, with that, today, our question is what if we could change the lighting after the photo has been made? This work proposes a cool technique to performexactly that by enabling us to edit the shadows on a portrait photo that we would normallythink of deleting. Many of these have to do with the presenceof shadows, and you can see here that we can really edit these after the photo has beentaken. However, before we start taking a closer lookat the editing process, we have to note that there are different kinds of shadows.
One, there are shadows cast on us by externalobjects, let’s call those foreign shadows, and there is self-shadowing, which comes fromthe model’s own facial features. Let’s call those facial shadows. So why divide them into two classes? Simple, because we typically seek to removeforeign shadows, and edit facial shadows. The removal part can be done with a learningalgorithm, provided that we can teach it with a lot of training data. Let’s think about ways to synthesize sucha large dataset! Let’s start with the foreign shadows.
We need image pairs of test subjects withand without shadows to have the neural network learn about their relations. Since removing shadows is difficult withoutfurther interfering with the image, the authors opted to do it the other way around. In other words, they take a clean photo ofthe subject, that’s the one without the shadows, and then, and add shadows to it algorithmically. Very cool! And, the results are not bad at all, and getthis, they even accounted for subsurface scattering, which is the scattering of light under ourskin.
That makes a great deal of a difference. This is a reference from a paper we wrotewith scientists at the University of Zaragoza and the Activision Blizzard company to addthis beautiful effect to their games. Here is a shadow edge without subsurface scattering,quite dark, and with subsurface scattering, you see this beautiful glowing effect. Subsurface scattering indeed makes a greatdeal of difference around hard shadow edges, so huge thumbs up for the authors for includingan approximation of that. However, the synthesized photos are stilla little suspect.
We can still tell that they are synthesized. And that is kind of the point. The problem is not easy - previousmethods did not do too well on these examples when you compare them to the reference solution. And let’s see this new method. Wow, I can hardly believe my eyes. Nearly perfect. And it did learn all this on not real, butsynthetic images. And believe it or not, this was only the simplerpart. Now comes the hard part. Let’s look at how well it performs at editingthe facial shadows! We can pretend to edit both the size and theintensity of these light sources. The goal is to have a little more controlover the shadows in these photos, but, whatever we do with them, the outputs still have toremain realistic.
Here are the before and after results. The facial shadows have been weakened, anddepending on our artistic choices, we can also soften the image a great deal. Absolutely amazing. As a result, we now have a two-step algorithm,that first, removes foreign shadows, and is able to soften the remainder of the facialshadows, creating much more usable portrait photos of our friends, and all this after the photo has been made. The algorithm may fail to remove some highlydetailed shadows, you can see how the shadow of the hair remains in the output. In this other output, the hair shadows arehandled a little better, there is some dampening, but the symmetric nature of the facial shadowshere put the output results in an interesting. If you have any questions so please comment below.
ConversionConversion EmoticonEmoticon