Lemony Snicket & ILMS shape capture system

ILM image based modeling system is a set of tools designed to recover detailed geometric information from photographs or film sequences.

It exploits texture and contour features to densely reconstruct surfaces with complicated shape. At one extreme, it can create useful approximate models from only a few silhouettes eg Minority Report, while at another extreme, it can integrate many image cues to produce dense high precision models comparable to laser scans eg Lemony Snicket, Pirates of the Caribbean Dead Mans Chest.

Image capture for Lemony Snicket The reconstruction process typically starts with two or more photos or frames of a subject, which are then undistorted and matchmoved to establish the projection geometry. Depending on the desired output, an artist will identify regions of texture, silhouette contours, point or image correspondences, and or 3d curves or surfaces to drive the reconstruction.

Models are then either produced directly, or templates are fit to the data to preserve existing topology and parameters.

The artist can optionally use the interactive editing tools to correct reconstruction errors, change resolution, filter noise, or even extrapolate solutions as needed.

Typical reconstruction time for a quick model from a couple snapshots might be 15 minutes, while a high precision facial model takes around 2 hours.

Image projection for Lemony Snicket

The system has been applied to a wide range of production problems over the past five years, including constructing CG versions of maquettes models, building an organic virtual environment from an outdoor photo survey, creating set geometry for match moves and character interaction, creating scene geometry in our 3d from 2d process for stereo films, and capturing detailed models of actor faces and bodies for digital double work.

It has evolved considerably based on feedback from ILMS own production teams, and is now preferred over Cyberscans for modeling human subjects by ILMS artists.

Texture mapped geometry for Lemony Snicket Laser based scanners address many of the same problems, and can actually produce better or faster results in some situations such as static objects or sets where there is time to scan.

However, they are often impractical or impossible to use under common production circumstances, such as when the set cannot be cleared and held long enough to do a scan shot setups rolling too quickly, plate or reference footage is available but no scan was taken, the scale of the subject is too large small, the subject cannot remain still for an extended period, or a scanner is simply unavailable or unable to be brought on location. In fact, the need to scan babies and children was one of the motivations for building our own solution.

Fully Textured Model

For Lemony Snicket Because the ILM system is image based, it is far more flexible than active scanning technologies, since it works for subjects of any size, allows instantaneous capture, can be used after the fact on unplanned footage, and the source photos provide backup validation for reconstruction results.

With common consumer cameras under controlled conditions, the ILM system can achieve point wise accuracy comparable to a Cyberscan, and quality can improve in the future as photographic equipment improves.

These cameras can also be synchronized to a beauty lighting package and or extra cameras to provide nearly simultaneous 1 30th of a second capture of production quality texture reference matched precisely to the geometry, something no other system currently provides.

Just as important, the interactive tools allow artists to generate useful reconstructions from incomplete or unplanned data, and decide for themselves how much post processing or cleanup eg smoothing they care to use.

Full Textured Model for Lemony Snicket Regarding image based approaches, photogrammetric tools are now in common use across the industry, generally reconstructing shape by triangulating isolated points.

ILM uses photogrammetry extensively at ILM, but the pointwise approach can be extremely laborious or too sparse for common cases like people, terrain, architectural detail, or other organic forms.

It can only serve as dimensional reference for these kinds of subjects, and modelers commonly complain of needing far more dense contours and detail to help construct production-quality models.