US Patent Awarded for Cognitech 3D Face Fusion that pioneers Video Superresolution of arbitrary in-3D motion human faces

Cognitech scientists and engineers continuously draw new computational ideas (algorithms) from the inexhaustible sources of scientific fields of Image Processing, 3D Computer Vision and Photogrammetry, and apply these algorithms to the imagery that comes from investigative and forensic field applications, such as CCTV videos, photos, etc. This is the true meaning of the so-called Forensic Video field


Cognitech 3D Face FusionTM and FBI on Boston Bombers Video

One such CCTV video (Boston Bombers video) was captured during the annual Boston Marathon on April 15, 2013. The video depicts suspects in a terrorist act that resulted in a terrible loss of human lives and serious injuries to the public at the scene. Several days later, the Federal Bureau of Investigation (FBI) released this CCTV Video to the public to assist with the identification of the perpetrators. FBI also released several images of the suspects, derived from the CCTV Video. 

Unfortunately, the CCTV Video and hence the derived and the released images were of a poor resolution quality, despite the clear (to professional eye) effort to enhance this imagery. The reason for the above image enhancement effort not yielding significant facial features improvements over the ‘raw’ video frames can be analyzed through 3D Image Velocity analysis. 

It turns out, the classical video Deblurring and video Resolution Enhancement methods (Superresolution) can not account for a video depicting a full 3D facial movement, such as observed in a 3D facial rotation out of the image plane ( i.e., turning towards and away from the camera).

On the other hand, the Computer Vision theory tells us that knowing the full 3D motion of the object implies knowing the 3D shape scan of this object (up to scale factor). But this information is usually not available with a face in the video(otherwise, we already identified the suspect through his 3D face scan). 

In a special case of License Plates in the video, this 3D shape is just a ‘flat plate’. This observation led Cognitech scientists to the Frame Fusion Superresolution algorithm in the 1990s, the first of its kind to enhance the resolution of License Plates (to read LP’s) in videos.

Fortunately, Cognitech scientists have found a key to this 3D ‘puzzle’ through the use of the ‘Generic Morphable 3D Face Model’. We call this method Cognitech 3D Face FusionTM.

The review of the Boston Bombers video reveals that the face of one of the suspects undergoes such fully 3D motion while he is walking by the capturing camera. 

Boston Bomber

See results and learn how Cognitech 3D Face FusionTM is applied to the original FBI released Boston Bombers video in the following interactive Tutorial;

FBI released Boston Bombers video using Cognitech 3D Face FusionTM 

The ‘Result’ at the end of this Tutorial compare suspect’s facial shots originally released by the FBI to the face shots derived via Cognitech 3D Face FusionTM. Indeed, One Look Is Worth A Thousand Words” Piqua, Ohio Newspaper, 1913 

Cognitech research and related background

Cognitech, Inc scientific team has developed an innovative computer vision-based technology and a fast algorithm that generally relates to 3D model-based object recognition (e. g. human faces and vehicles), identification, and related video resolution enhancement (Superresolution). Specifically, it accomplishes human face recognition, face comparison, and image/video Superresolution of facial images observed in an arbitrary 3D motion, and more particularly to determining and exploiting the fact that a sequence of two-dimensional images includes a representation of a three-dimensional object whose generic 3D model is available. 

In a particular case of facial images, the facial 3D model is retrieved from an indexed database. This realization produced the human face specific superresolution algorithm which receives a plurality of images ( e.g. observed moving face in one or several videos); matching features in each image to generic features of the 3D model( eyes, mouth, nose, etc.); determining a transform between each image and the 3D model using the matched features; for each image, mapping each image content to the 3D model, combining all the images mapped onto the surface of the 3D model, thus reconstructing a Superresolution ‘3D Face Fusion’. 

US Patent Awarded for Cognitech 3D Face FusionTM

This 3D Face Fusion now can be rotated, and significant resolution enhancement of the original facial images is obtained. The Cognitech’s 3D Face Fusion technology was awarded US Patent for each major component of this invention, which extends to any 2D to 3D objects recognition and Superresolution.  This US Patent: L. Rudin et al. co-authors, is – OBJECT RECOGNITION BASED ON 2D IMAGES AND 3D MODELS(# US 7587082B1). 

The US Patent claim relevant to the method of Cognitech 3D Face FusionTM is  “The method wherein the images include an image of a face of at least one individual.”

Below is the abstract of the Cognitech’s (# US 7587082B1) patent:

Methods and systems for performing object directed recognition based on two-dimensional images and three-dimensional models. Transforms are used to map features of an object to an image seen from a vantage point and to map features of images best seen from the vantage points to three-dimensional models. Mapping of features and images to the three-dimensional model and then to image planes of other images in comparison of corresponding features from the mapped images to original images allows for the determination of coherence between the mapped images and the original images.


Read more-


Talk with experts for Forensic Video Processing Software and Forensic Image Processing Software solutions. Contact Cognitech!

We hope you enjoyed this CogniKnowledge Blog! Stay tuned and don’t miss the next blogs. You can also follow us on Twitter, Facebook, Instagram, or YouTube: we post CogniKnowledge Blogs and ForensicFriday Blogs regularly so you won’t miss any!