The MoMo-PRT Model

In this chapter we are going to model the relationship between Morphable Model shape coefficients and transfer matrices of corresponding shape instances. We justify this approach under the assumption that similar shapes are subject to similar self-shadowing effects and light transfer in general. In this chapter, we map shape coefficients with transfer matrices to determine self-shadowing for the whole Morphable Model shape space.

First, we create the model that approximates the tradiance transfer matrices. Then we validate it by comparing renderings to a local-only illumination model and the exact global illumination model.

The Model in Detail

The model consists of two phases. One for model building and one for rendering:

Figure 1: Model building and rendering phase. In the model building phase we compute the transfer matrices with ray-tracing. Rendering is fast, because we approximate the transfer matrices by regressing them from the shape coefficients.

Figure 1: Model building and rendering phase. In the model building phase we compute the transfer matrices with ray-tracing. Rendering is fast, because we approximate the transfer matrices by regressing them from the shape coefficients.

Model Building

To approximate global illumination for the whole Morphable Model, we sample a few shapes in the shape space and simulate the surface-light interaction with ray tracing for them. We use the computed data to interpolate the surface-light interaction for the rest of the shape model. To do so, we find the mapping between shape coefficients and transfer matrices with multivariate linear regression on a per-vertex basis.

We sample different faces from the Morphable Model and perform a full transfer simulation for each mesh instance. We build a linear system of equations relating sampled model coefficients with their corresponding transfer matrices. The equation at a vertex i is:

\[T_{1..n}^i = W^i \theta_{1..n}.\]

Each columns of \(T_{1..n}^i\) corresponds to a sampled shape. They contain the vectorized transfer matrices of vertex \(i\) that we got with ray tracing.

Similarly, we concatenate the face coefficients into a matrix \(\theta_{1..n}\). Finally, we compute a matrix \(W^i\) for each vertex \(i\) that maps face coefficients to transfer matrices. Therefore, the resulting model is constituted of \(m\) matrices \(W^i, i=1..m\) where \(m\) is the number of vertices.

Rendering

During rendering, the model generates transfer data for each vertex based on the face coefficients. We multiply the matrices \(W^i, i=1..m\) with the coefficients of the mesh we want to render. The result is a vector of the transfer data for each vertex corresponding to the input coefficients. We reshape each vector back into a valid transfer matrix.

Figure 1 shows the process of building the transfer model, and how it is used during rendering.

Building the Model

Now that we have defined the inner mechanics of the model it is time to put it into action. Fortunately, the details are already implemented in the MoMo-PRT framework in its LinearTransferModel class and its companion object.

To begin, we need to specify which PRT technique we want to build our model for. In this example we use the standard LambertTechnique. It implements Lambertian reflectance. To be able to restart the tutorial without needing to recompute the transfer data every time, we cache occlusion and transfer data.

val technique = LambertTechnique
  .withDecorator(ProfiledTechnique.apply)
  .withDecorator(PersistentCaching.apply)
  .withCachingPath("tmp/prt-cache/model-builder")
val parameterModel = LambertParameterModel(technique)
  .withParameterModifier(_
    .withOcclusionRaycastSamples(900)
    .withLightBounces(0)
    .withShBands(3)
  )

Of course we need some training data to fit the model parameters to data. For that we load the morphable model. Additionally we specify the coefficients of the faces we want to use for fitting the model. In this instance, the coefficients of our training samples correspond to vertices of a hypercube spanning the first five dimensions of the face parameter space. You can use the built-in code-completion (Ctrl-Space) to list other configurations available in CoefficientGrid.

scalismo.initialize() // Don't forget to initialize scalismo somewhere
val momo = readMoMo("data/supplied/model2017-1_bfm_nomouth_l5.h5")

val coefficientSampling = CoefficientGrid.sampling_2p2_2p2_2p2_2p2_2p2
val coefficientSamples = CoefficientGrid(shapeRanges = coefficientSampling)
println("Number of face samples: " + coefficientSamples.size)

We limit the number of coefficients that will be used to predict the transfer matrices. Here, we choose that our model depends on the first five dimensions of the coefficient space. In this example, it would be pointless to include more coefficients because our training samples only vary in the first five coefficients as well. In general, the number of coefficients used should be chosen such that the effects we want to be able to simulate are contained. For example, if the model should be able to produce self-shadowing in the region of the naso-labial folds we should generate data that shows some variability in that region.

// Relevant coefficients for the model
val coefficientSelector = CoefficientSelector.RangeSelector(5, 0, 0, affineBias = true)

Finally, we build the model using the eponymous method in the LinearTransferModel object. Note that we need to pass the technique which is used to simulate the transfer for the training samples internally. Building the model might take some time - up to a few minutes, if you used the suggested parameters up to this point.

val linearTransferModel =
  LinearTransferModel.buildModel(
    technique,
    parameterModel,
    momo,
    coefficientSamples,
    coefficientSelector)

// Store to disk for later use...
LinearTransferModelIO.saveTransferModel(linearTransferModel, "data/model/model.h5").get

Rendering

Now let's bring our model into action by rendering some faces.

The MoMo-PRT framework is built around the RenderingModel trait which combines a ParameterModel with a TransferModel, both of which are needed to define a parametric PRT renderer. This modular approach neatly allows to combine different transfer modelling techniques with arbitrary PRT technique which might depend on more advanced rendering parameters. In order to use our linear transfer model we simply combine it with the parameterModel that we defined earlier.

val modelRenderer = RenderingModel(technique)
  .withParameterModel(parameterModel)
  .withTransferModel(linearTransferModel)
  .getParametricMoMoRenderer(momo).withClearColor(RGBA.WhiteTransparent)

Side note: Because our transfer model only captures self-shadowing (i.e. no reflections), we could choose to combine it with the GlossyTechnique and the corresponding GlossyParameterModel to create specular reflections.

Furthermore we are going to define three additional renderers to compare our model to: * Exact renderer (i.e. regular PRT, uses ray tracing). * One that always takes the transfer matrices of the mean shape. * One with a local illumination model.

val meanRenderer = RenderingModel(technique)
  .withParameterModel(parameterModel)
  .withTransferModelTemplate(MeanTransferModel(momo))
  .getParametricMoMoRenderer(momo).withClearColor(RGBA.WhiteTransparent)

val exactRenderer = RenderingModel(technique)
  .withParameterModel(parameterModel)
  .withTransferModelTemplate(ExactTransferModel.apply)
  .getParametricMoMoRenderer(momo).withClearColor(RGBA.WhiteTransparent)

val noShadowsRenderer = RenderingModel(technique)
  .withParameterModel(parameterModel.withParameterModifier(_.withoutOcclusion))
  .withTransferModelTemplate(ExactTransferModel.apply)
  .getParametricMoMoRenderer(momo).withClearColor(RGBA.WhiteTransparent)

With the renderers defined above, we are just one method call away from rendering an image. However, we still need to wrap it into InteractiveFigure for the tutorial before we can marvel at our real-time dynamic PRT rendering:

val renderPanel = new InteractiveRenderPanel
  with InteractiveMoMoInstance
  with DefaultLightingEnvironments  {
  override def render(param: RenderParameter): PixelImage[RGBA] = {
    // Please note:
    // you can control the currently active renderer with the keys 1-4.
    // Keyboard-control hints are printed to console output
    import KeyEvent._
    val renderer = input.getToggleGroup(VK_1, VK_2, VK_3, VK_4) match {
      case Some(VK_1) | None => modelRenderer
      case Some(VK_2) => noShadowsRenderer
      case Some(VK_3) => meanRenderer
      case Some(VK_4) => exactRenderer
    }

    renderer.renderImage(
      param
        .withMoMo(momoInstance)
        .withEnvironmentMap(light.rgb)
    )
  }

}
val figC2 = new InteractiveFigure("Demo", interactiveRenderPanel = renderPanel)
/** you can change the first few shape compoments of the model with the keys: q,a, w,s, e,d, r,f .*/

Validation

Comparing the different renderers in the previous InteractiveFigure we can already see that our linear transfer model approximates the exact transfer quite well and outperforms the mean transfer model in most cases - fair enough. But still we would like some quantitative analysis regarding our model's accuracy. Consequently, we are are going to evaluate the pixel error of our model for multiple face samples at different standard deviations to get a better idea of the approximation's quality.

First we are going to sample a few face parameters at different standard deviations from the mean face.

implicit val rnd: Random = scalismo.utils.Random(24 - 1 - 1994L) // (Fixed seed enables persistent caching)

val validationRadii = 0.0 to 8.0 by 1.0
val validationSamplesPerRadius = 5

val validationSamples = validationRadii map { r =>
  Seq.fill(validationSamplesPerRadius) {
    MoMoCoefficients(
      sampleOnSphere(r, components = 10),
      sampleOnSphere(r, components = 10),
      sampleOnSphere(0, components = 1)
    )
  }
}

Also we specify under which parameters we want to validate the model.

val validationParameters = RenderParameter.defaultSquare
  .withEnvironmentMap(validationLightA)

For each validation sample we compare the rendered images against the exact rendering in a nested loop and sum over repetitions.

val approximateRenderers = Seq(noShadowsRenderer, meanRenderer, modelRenderer)

val results = validationSamples map { samplesAtRadius =>
  val relativeErrors = samplesAtRadius map {
    sample =>
      val params = validationParameters.withMoMo(sample)
      val groundTruth = exactRenderer.renderImage(params)

      approximateRenderers map { renderer =>
        relativeImageError(renderer.renderImage(params), groundTruth)
      }
  }

  val aggregatedRelativeErrors = relativeErrors
    .transpose.map(_.sum(Numeric.DoubleIsFractional) / validationSamplesPerRadius * 100)

  aggregatedRelativeErrors
} transpose

Finally we can plot the collected data.

import org.sameersingh.scalaplot._
import org.sameersingh.scalaplot.Implicits._

val chart = xyChart(
  new XYData(
    validationRadii -> Y(results(0), label = "Local Illumination Model"),
    validationRadii -> Y(results(1), label = "Mean Transfer"),
    validationRadii -> Y(results(2), label = "MoMo Transfer")
  ),
  x = Axis(label = "mahalanobis distance to mean"),
  y = Axis(label = "average RGB magnitude to PRT in %"),
  showLegend = true
)

output(GUI, chart)

Obviously the PRT-based renderings outperform the local rendering without shadows by a rather significant margin. Moreover, the plot suggests that the linear transfer model generally produces more accurate results than the constant mean-transfer model for higher standard deviations under the selected rendering parameters.

Please note that there is still room for improvement. More accurate results can be achieved by adding more face coefficients to the model and fitting it with more samples (e.g. by randomly sampling training faces).


Tutorial Home | Next Chapter: Removing Self-Shadowing from the Texture Model | Previous Chapter: PRT Introduction