Other Advocacy Entities

This section provides a short survey of industry advocacy and activities in support of 3DTV.

3D@Home Consortium

Recently (in 2008) the 3D@Home Consortium was formed with the mission to speed the commercialization of 3D into homes worldwide and provide the best possible viewing experience by facilitating the development of standards, roadmaps, and education for the entire 3D industry—from content, hardware, and software providers to consumers.

3D Consortium (3DC)

The 3D Consortium (3DC) aims at developing 3D stereoscopic display devices and increasing their take-up, promoting expansion of 3D contents, improving distribution, and contributing to the expansion and development of the 3D market. It was established in Japan in 2003 by five founding companies and 65 other companies including hardware manufacturers, software vendors, contents vendors, contents providers, systems integrators, image producers, broadcasting agencies, and academic organizations.

European Information Society Technologies (IST) Project ‘‘Advanced Three-Dimensional Television System  Technologies’’ (ATTEST)

This is a project where industries, research centers, and universities have joined forces to design a backwards-compatible, flexible, and modular broadcast 3DTV system. The ambitious aim of the European Information Society Technologies (IST) project ATTEST is to design a novel, backwards-compatible, and flexible broadcast 3DTV system. In contrast to former proposals that often relied on the basic concept of “stereoscopic” video, that is the capturing, transmission, and display of two separate video streams (one for the left eye and one for the right eye), this activity focuses on a data-in-conjunction-with-metadata approach. At the very heart of the described new concept is the generation and distribution of a novel data representation format that consists of monoscopic color video and associated per-pixel depth information. From these data, one or more “virtual” views of a real-world scene can be synthesized in real-time at the receiver side (i.e., a 3DTV STB) by means of the DIBR techniques. The modular architecture of the proposed system provides important features, such as backwards-compatibility to today’s 2D DTV, scalability in terms of receiver complexity, and adaptability to a wide range of different 2D and 3D displays.

3D Content Creation. For the generation of future 3D content, novel three-dimensional material is created by simultaneously capturing video and associated per-pixel depth information with an active range camera such as the so-called ZCamTM developed by 3DV Systems. Such devices usually integrate a high-speed pulsed infrared light source into a conventional broadcast TV camera and they relate the time of flight of the emitted and reflected light walls to direct measurements of the depth of the scene. However, it seems clear that the need for sufficient high-quality, three-dimensional content can only partially be satisfied with new recordings. It will therefore be necessary (especially in the introductory phase of the new broadcast technology) to also convert already existing 2D video material into 3D using so-called “structure from motion” algorithms. In principle, such (offline or online) methods process one or more monoscopic color video sequences to (i) establish a dense set of image point correspondences from which information about the recording camera, as well as the 3D structure of the scene can be derived or (ii) infer approximate depth information from the relative movements of automatically tracked image segments. Whatever 3D content generation approach is used in the end, the outcome in all cases consists of regular 2D color video in European DTV format (720 × 576 luminance pels, 25 Hz, interlaced) and an accompanying depth-image sequence with the same spatiotemporal resolution. Each of these depth-images stores depth information as 8-bit gray values with the gray level 0 specifying the furthest value and the gray level 255 defining the closest value. To translate this data representation format to real, metric depth values (that are required for the “virtual” view generation (and to be flexible with respect to 3D scenes with different depth characteristics, the gray values are normalized to two main depth clipping planes.

3DV Coding. To provide the future 3DTV viewers with threedimensional content, the monoscopic color video and the associated per-pixel depth information have to be compressed and transmitted over the conventional 2D DTV broadcast infrastructure. To ensure the required backwards-compatibility with existing 2D-TV STBs, the basic 2D color video has to be encoded using the standard MPEG-2 as MPEG-4 Visual or AVC tools currently required by the DVB Project in Europe.

Transmission. The DVB Project, a consortium of industries and academia responsible for the definition of today’s 2D DTV broadcast infrastructure in Europe, requires the use of the MPEG-2 systems layer specifications for the distribution of audiovisual data via cable (DVB-C), satellite (DVB-S), or terrestrial (DVB-T) transmitters.

‘‘Virtual’’ View Generation and 3D Display. At the receiver side of the proposed ATTEST system, the transmitted data is decoded in a 3DTV STB to retrieve the decompressed color video- and depth-image sequences (as well as the additional metadata). From this data representation format, a DIBR algorithm generates “virtual” left- and right-eye views for the three-dimensional reproduction of a real-world scene on a stereoscopic or autostereoscopic, singleor multiple-user 3DTV display. The backwards-compatible design of the system ensures that viewers who do not want to invest in a full 3DTV set are still able to watch the two-dimensional color video without any degradations in quality using their existing digital 2DTV STBs and displays.

3D4YOU

3D4YOU7 is funded under the ICT Work Programme 2007–2008, a thematic priority for research and development under the specific program “Cooperation” of the Seventh Framework Programme (2007–2013). The objectives of the project are

1. to deliver an end-to-end system for 3D high-quality media;
2. to develop practical multi-view and depth capture techniques;
3. to convert captured 3D content into a 3D broadcasting format;
4. to demonstrate the viability of the format in production and over broadcast chains;
5. to show reception of 3D content on 3D displays via the delivery chains;
6. to assess the project results in terms of human factors via perception tests;
7. to produce guidelines for 3D capturing to aid in the generation of 3D media production rules;
8. to propose exploitation plans for different 3D applications.

The 3D4YOU project aims at developing the key elements of a practical 3D television system, particularly, the definition of a 3D delivery format and guidelines for a 3D content creation process.

The 3D4YOU project will develop 3D capture techniques, convert captured content for broadcasting, and develop 3D coding for delivery via broadcast that is suitable to transmit and make public. 3D broadcasting is seen as the next major step in home entertainment. The cinema and computer games industries have already shown that there is considerable public demand for 3D content but the special glasses that are needed limits their appeal. 3D4YOU will address the consumer market that coexists with digital cinema and computer games. The 3D4YOU project aims to pave the way for the introduction of a 3D TV system. The project will build on previous European research on 3D, such as the FP5 project ATTEST that has enabled European organizations to become leaders in this field.

3D4YOU endeavors to establish practical 3DTV. The key success factor is 3D content. The project seeks to define a 3D delivery format and a content creation process. Establishing practical 3DTV will then be demonstrated by embedding this content creation process into a 3DTV production and delivery chain, including capture, image processing, delivery, and then display in the home. The project will adapt and improve on these elements of the chain so that every part integrates into a coherent interoperable delivery system. A key project’s objective is to provide a 3D content format that is independent of display technology, and backward compatible with 2D broadcasting. 3D images will be commonplace
in mass communication in the near future. Also, several major consumer electronics companies have made demonstrations of 3DTV displays that could be in the market within two years. The public’s potential interest in 3DTV can be seen by the success of 3D movies in recent years. 3D imaging is already present in many graphics applications (architecture, mechanical design, games, cartoons, and special effects for TV and movie production).

In recent years, multi-view display technologies have appeared that improve the immersive experience of 3D imaging that leads to the vision that 3DTV or similar services might become a reality in the near future. In the United States, the number of 3D-enabled digital cinemas is rapidly growing. By 2010, about 4300 theaters are expected to be equipped with 3D digital projectors with the number increasing every month. Also in Europe, the number of 3D theaters is growing. Several digital 3D films will surface in the months and years to come and several prominent filmmakers have committed to making their next productions in stereo 3D. The movie industry creates a platform for 3D movies, but there is no established solution to bring these movies to the domestic market. Therefore, the next challenge is to bring these 3D productions to the living room. 2D to 3D conversion and a flexible 3D format are an important strategic area. It has been recognized that multi-view video is a key technology that serves a wide variety of applications, including free viewpoint and 3DV applications for the home entertainment and surveillance business fields. Multi-view video coding and transmission systems are most likely to form the basis for next-generation TV broadcasting applications and facilities. Multi-view video will greatly improve the efficiency of current video coding solutions performing simulcasts of independent views. This project builds on the wealth of experience of the major players in European 3DTV and intends to bring the date of the start of 3D broadcasting a step closer by combining their expertise to define a 3D delivery format and a content creation process.

The key technical problems that currently hamper the introduction of 3DTV to the mass market are as follows:

1. It is difficult to capture 3DV directly using the current camera technology. At least two cameras need to operate simultaneously with an adjustable but known geometry. The offset of stereo cameras needs to be adjustable to
   capture depth, both close by and far away.
2. Stereo video (acquired with 2-cameras) is currently not sufficient input for glasses-free, multi-view autostereoscopic displays. The required processing, such as disparity estimation, is noise-sensitive resulting in low 3D picture quality.
3. 3D postproduction methods and 3DV standards are largely absenterimmature.

The 3D4YOU project will tackle these three problems. For instance, a creative combination of two or three high-resolution video cameras with one or two lowresolution depth range sensors may make it possible to create 3DV of good quality without the need for an excessive investment in equipment. This is in contrast to installing, say, 100 cameras for acquisition where the expense may hamper the introduction of such a system.

Developing tools for conversion of 3D formats will stimulate content creation companies to produce 3DV content at acceptable cost. The cost at which 3DV should be produced for commercial operation is not yet known. However, currently, 3DV production requires almost per frame user interaction in the video, which is certainly unacceptable. This immediately indicates the issue that needs to be solved: currently, fully automated generation of high 3DV is difficult; in the future it needs to be fully, or at least semi-automatic with an acceptable minimum of manual supervision during postproduction. 3D4YOU will research how to convert 3D content into a 3D broadcasting format and prove the viability of the format in production and over broadcast chains.

Once 3DV production becomes commercially attractive because acquisition techniques and standards mature, then this will impact the activities of content producers, broadcasters, and telecom companies. As a result, one may see that these companies may adopt new techniques for video production just because the output needs to be in 3D. Also, new companies could be founded that focus on acquiring 3DV and preparing it for postproduction. Here, there is room for differentiation since, for instance, the acquisition of a sport event will require large baselines between cameras and real-time transmission, whereas the shooting of narrative stories will require both small and large baselines and allows some manual postproduction for achieving optimal quality. These activities will require new equipment (or a creative combination of existing equipment) and new expertise.

3D4YOU will develop practical multi-view and depth capture techniques. Currently, the stereo video format is the de facto 3D standard that is used by the cinemas. Stereo acquisition may, for this reason, become widespread as an acquisition technique. Cinemas operate with glasses-based systems and can therefore use a theater-specific stereo format. This is not the case for the glasses-free autostereoscopic 3DTV that 3D4YOU foresees for the home. To allow glassesfree viewing with multiple people at home, a wide baseline is needed to cover the total range of viewing angles. The current stereo video that is intended for the cinema will need considerable postproduction to be suitable for viewing on a multi-view autostereoscopic display. Producing visual content will therefore, become more complex and may provide new opportunities for companies currently active in (3D) movie postproduction. According to the Networked and Electronic Media (NEM) Strategic Research Agenda, multi-view coding will form the basis for next-generation TV broadcast applications. Multi-view video has the advantage that it can serve different purposes. On the one hand, the multi-view input can be used for 3DTV. On the other hand, it can be shown on a normal TV where the viewer can select his or her preferred viewpoint of the action. Of course, a combination is possible where the viewer selects his or her preferred viewpoint on a 3DTV. However, multi-view acquisition with 30 views for example, will require 30 cameras to operate simultaneously. This initially requires a large investment. 3D4YOU therefore sees a gradual transition from stereo capture to systems with many views. 3D4YOU will investigate a mixture of 3DV acquisition techniques to produce an extended center view plus depth format (possibly with one or two extra views) that is, in principle, easier to produce, edit, and distribute. The success of such a simpler format relies on the ease (read cost!) at which it can be produced. One can conclude that the introduction of 3DTV to the mass market is hampered by (i) the lack of highquality 3DV content; (ii) by the lack of suitable 3D formats; and (iii) lack of appropriate format conversion techniques. The variety of new distribution media further complicates this.

Hence, one can identify the following major challenges that are expected to be overcome by the project:

1. Video Acquisition for 3D Content: Here, the practicalities of multi-view and depth capture techniques are of primary importance, the challenge is to find the trade off such as number of views to be recorded, and how to
   optimally integrate depth capture with multi-view. A further challenge is to define which shooting styles are most appropriate.
2. Conversion of Captured Multi-View Video to a 3D Broadcasting Format: The captured format needs new postproduction tools (like enhancement and regularization of depth maps or editing, mixing, fading, and compositing of V+D representations from different sources) and a conversion step generating a suitable transmission format that is compatible with used postproduction formats before the 3D content can be broadcast and displayed.
3. Coding Schemes for Compression and Transmission: A last challenge is to provide suitable coding schemes for compression and transmission that are based on the 3D broadcasting format under study and to demonstrate their feasibility in field trials under real distribution conditions.

By addressing these three challenges from an end-to-end systems point of view, the 3D4YOU project aims to pave the way to the definition of a 3D TV system suitable for a series of applications. Different requirements could be set depending on the application, but the basic underlying technologies (capture, format, and encoding) should maintain as much commonality as possible so as to favor the emergence of an industry based on those technologies.

3DPHONE

The 3DPHONE project aims to develop technologies and core applications enabling a new level of user experience by developing end-to-end all-3D imaging mobile phone. Its aim is to have all fundamental functions of the phone—media display, User Interface (UI), and personal information management (PIM) applications—realized in 3D. We will develop techniques for all-3D phone experience: mobile stereoscopic video, 3D UIs, 3D capture/content creation, compression, rendering, and 3D display. The research and development of algorithms for 3D audiovisual applications including personal communication, 3D visualization, and content management will be done.

The 3DPhone Project started on February 11, 2008. The duration of the project is 3 years and there are six participants from Turkey, Germany, Hungary, Spain, and Finland. The partners are Bilkent University, Fraunhofer, Holografika, TAT, Telefonica, and University of Helsinki. 3DPhone is funded by the European Community’s ICT programme in Framework Programme Seven.

The goal is to enable users to

• capture memories in 3D and communicate with others in 3D virtual spaces;
• interact with their device and applications in 3D;
• manage their personal media content in 3D.

The expected outcome will be simpler use and a more personalized look and feel. The project will bring state-of-the-art advances in mobile 3D technologies with the following activities:

• A mobile hardware and software platform will be implemented with both 3D image capture and 3D display capability, featuring both 3D displays and multiple cameras. The project will evaluate different 3D display
  and capture solutions and will implement the most suitable solution for hardware–software integration.
• UIs and applications that will capitalize on the 3D autostereoscopic illusion in the mobile handheld environment will be developed. The project will design and implement 3D and zoomable UI metaphors suitable for autostereoscopic displays.
• End-to-end 3DV algorithms and 3D data representation formats, targeted for 3D recording, 3D playback, and real-time 3DV communication will beinvestigated and implemented.
• Ergonomics and experience testing to measure any possible negative symptoms, such as eye strain created by stereoscopic content, will be performed. The project will research ergonomic conditions specific to the mobile handheld usage: in particular, the small screen, one hand holding the device, absence of complete keyboard, and limited input modalities.

In summary, the general requirements on 3DV algorithms on mobile phones are as follows:

• low power consumption,
• low complexity of algorithms,
• limited memory/storage for both RAM and mass storage,
• low memory bandwidth,
• low video resolution,
• limited data transmission rates and limited bitrates for 3DV signal.

These strong restrictions derived from terminal capabilities and from transmission bandwidth limitations usually result in relatively simple video processing algorithms to run on mobile phone devices. Typically, video coding standards take care of this by specific profiles and levels that only use a restricted and simple set of video coding algorithms and  low-resolution video. The H.264/AVC Baseline Profile for instance, uses only a simple subset of the rich video coding algorithms that the standard provides in general. For 3DV, the equivalent of such a low-complexity baseline profile for mobile phone devices still needs to be defined and developed. Obvious requirements of video processing and coding apply for 3DV on mobile phones as well, such as

• high coding efficiency (taking bitrate and quality into account);
• requirements specific for 3DV that apply for 3DV algorithms on mobile phones including
  • flexibility with regard to different 3D display types,
  • flexibility for individual adjustment of 3D impression.