Apple’s Mysterious Fisheye Projection

🌈 Abstract

The article discusses Apple's mysterious "fisheye" projection format, which is used to stream their immersive Apple TV+ videos, but has not been documented or made available for third-party use. The author, who has experience in the immersive video field, has investigated this format and shares their findings.

🙋 Q&A

[01] Apple's Fisheye Projection

1. What is Apple's fisheye projection format?

• Apple has a proprietary "fisheye" projection format for encoding their immersive videos, but it is undocumented and not available for third-party use.
• The author has analyzed a raw fisheye frame from an Apple TV+ intro clip and found that it is not a standard fisheye projection, but rather a custom format that appears to use a "radial stretching" technique.
• The author has experimented with various projection geometries and transformations to try to replicate the distortions seen in the Apple fisheye frame, but has not been able to achieve a 100% match.

2. What are the potential benefits of Apple's fisheye projection format?

• The author speculates that Apple's fisheye format may allow for more efficient encoding and preservation of image data compared to a standard equirectangular projection.
• Possible advantages could include handling distortion better, optimizing the placement of important pixels, and improving motion estimation efficiency during encoding.
• However, the author notes that the custom format also adds complexity to the toolchain, so the benefits must outweigh the drawbacks for Apple to have chosen this approach.

3. When might we learn more about Apple's fisheye projection?

• The author hopes that more details about Apple's fisheye projection will be revealed at a future WWDC (likely WWDC24), as Apple has not provided any official information about it so far.

[02] Immersive Video Concepts

1. What is the difference between 2D and 3D (stereoscopic) immersive video?

• 2D (monoscopic) immersive video has a 360-degree horizontal field of view and 180-degree vertical field of view.
• 3D (stereoscopic) immersive video packs two images into each frame, one for each eye, to create a more immersive 3D experience.

2. How are immersive video formats typically encoded and projected?

• Immersive video content is typically encoded using an equirectangular projection, which maps the spherical content onto a rectangular frame.
• During playback, the content is displayed as if the viewer's eyes are at the center of a sphere, with the video displayed on the inner surface.

</output_format>