Overview

It has been about two years since the Apple TV 4K was released, the average time between Apple TV refreshes. So, it is timely to consider what comes next. First off: I have no inside information about what Apple will release next. Rather, this is my wish list for the successor to the Apple TV 4K, as well as improvements to tvOS. It is what I would build were I the product owner of Apple TV. As a videophile, I admittedly overindex on features like video and audio fidelity. However, my suggestions also are based on our family’s initial experiences with Apple’s new gaming service, Apple Arcade.1

Our family’s use cases go beyond simple video streaming. In our home, we have seven Apple TV 4K settop boxes. With the exception of a few “old” Blu-ray players, we use our Apple TVs for everything: watching live TV & DVR with Plex and Channels, viewing streaming services (including Netflix, YouTube, Amazon Prime Video, HBO, Showtime, TED, Philo, and PBS), watching stored videos with Plex, listening to stored and streamed music services (including TIDAL, Google Play Music, SiriusXM, and Apple Music), viewing our Unifi Protect security cameras, and more.

The current Apple TV 4K supports videos up to 2160p resolution at 60 Hz including DolbyVision and HDR10 videos. 4K resolution videos with HDR look spectacular. But, the Apple TV 4K’s 60 Hz video output suffers from serious judder, particularly for movies.2 You can select “match frame rate”, which will output your video at its native frame rate. Your TV will then convert the video to 120 Hz but this will also introduce screen blanking when your TV switches frame rate.

It isn’t realistic to compare the Apple TV to the current Xbox One X or Playstation PS4 Pro gaming consoles. But, even lower resolution, lower complexity video games may hesitate on the current Apple TV’s A10X processor. Since the Apple TV 4K was released, Apple has made significant strides in both CPU and GPU development for mobile devices. With a faster processor (borrowed from its mobile cousins), the next generation Apple TV could support higher quality video games, rendered locally. Think Nintendo Switch style games on steroids. Or maybe earlier models of the Xbox One or PS4.

Video

When the Apple TV 4K was released, it forced all videos to be rendered at the same dynamic range and frame rate. Later, Apple added the capability to match the original dynamic range and/or match the original frame rate. Arguably, this was a quick fix to address deficiencies in Apple’s color space conversion and the limitations of videos played at 60 frames per second.

First, for a new model, a faster refresh rate is required: 4K at 120 Hz. This eliminates the judder that you see when watching movies that were originally filmed at 24 fps. Judder is particularly evident on scenes where the camera pans. Instead of a 3:2 pull-down that is then doubled by the Apple TV to 60 fps and then doubled by your TV to 120fps, 24fps movie content would be rendered directly at 120fps either at native frame rate multiplied by five (5:5 pulldown) for a cinematic experience or motion interpolation could be used for smoother, more life-like playback.3 Similarly, television content could be played at either four times its original frame rate or use motion interpolation.4

Support for 4K at 120 Hz (120 fps) will require HDMI 2.1 ports on both your TV and next-generation Apple TV, as well as new 48 Gbps cables. However, end-to-end support for HDMI 2.1 brings a number of advantages: capabilities like enhanced audio return channel, dynamic HDR, Auto Low-Latency Mode, variable refresh rates and quick frame transport technology. Additionally, it will eliminate frequent idiosyncrasies associated with HDMI-CEC negotiation. Effectively, most of the advanced video processing done by a 120hz TV panel would be shifted to the Apple TV. All of these would enable a better home theater experience at a lower total cost with a future Apple TV.

Second, Apple should support hardware accelerated, AI-enhanced, video upscaling.5 Perhaps, this might leverage the A13 chip’s Neural Engine functionality. This would free CPU/GPU cycles otherwise used for software-based upscaling of lower resolution content to 4K resolution and higher frame rates. Any application that renders 1080 or lower resolution content would benefit. Update: after I initially wrote this article, Nvidia’s 2019 Shield TV was announced, including AI-enhanced, video upscaling albeit limited to 30fps.

Third, for over-the-air TV, a hardware deinterlacer would be a important addition, freeing CPU/GPU resources. This is particularly valuable for the 1080i (interlaced) content broadcast by CBS, NBC, CW, and PBS.

Fourth, the next Apple TV should include support for AOMedia’s AV1 codec.6 This permits higher quality video at lower bitrates (avg 20% lower bandwidth) than either VP9 or HEVC/H.265. Notably, this would enable playback of 4K videos from YouTube.

Audio

Currently, Apple TV supports either Dolby MAT 2.0 (eight or 7.1 channels of lossless audio including Dolby Atmos objects)7 or Dolby Digital (six or 5.1 channels of lossy audio) formats. If an audio stream originates in another format, it must be transcoded via software to one of these two formats. Additionally, each application developer must separately license desired audio codecs, passing that cost on to consumers.

Ideally, Apple would include either pass-through support or hardware decoding for additional audio codecs including DTS, DTS-HD, DTS-X, and Dolby TrueHD. Of these, lossless audio codecs would significantly improve the audio experience. Native decoding of these codecs by the Apple TV would compensate for soundbars and home theater systems (like Sonos) that lack support for these codecs. With HDMI 2.1, the Apple TV’s hardware decoding to Dolby MAT 2.0 would enable home theater systems and soundbars to play lossless audio regardless of whether that systems natively supported the original codec.

A Faster Processor

The current Apple TV includes an A10X Fusion chipset. The iPhone 11 includes the A13 Bionic. Third party benchmarks of the two chipsets rank the A13’s graphics performance almost double that of the A10X8 and CPU performance up to 5x faster.910 I would expect an A13 (or hypothetical A13X) equipped Apple TV to exceed even the iPhone 11’s A13 performance because it would not be constrained by battery power. This would be similar to how the Apple TV 4K’s A10X had faster performance than contemporaneous iPads.111213

WiFi 6

Most homes do not have Ethernet adjacent to each television. So, most people necessarily connect their settop boxes via WiFi. The latest version of WiFi is WiFi 6, based on the IEEE 802.11ax standard. If there is anything that stresses a wireless network, it is high resolution video–particularly when multiple TVs are in use. The next Apple TV should include support for the fastest wireless connectivity just like the current Apple TV 4K includes support for Gigabit Ethernet.

Always-on Microphone

“Hey Siri!” The only way that you can talk to Siri today is to either press the microphone on the Apple TV remote or buy an Apple HomePod. There ought to be a built-in microphone in the settop box to enable “Hey Siri” searches. Alternatively, Apple should expose an API for smart speaker and soundbar manufacturers (like Sonos) to enable “Hey Siri” searches with results displayed on your TV by Apple TV.

Micro SD Card & USB4 Storage

Apple is notorious for using memory and storage to ladder customers to higher priced iPhones, Macs, and even Apple TV models. So, I have little hope that a future Apple TV will include support for expandable storage. Nevertheless, I believe that Apple’s settop boxes must respond, feature-wise, to smart home competitors that are priced much lower. Roku, Amazon Fire TV, Nvidia Shield TV and Xiaomi Mi Box all demonstrate how a micro SD card or USB thumbdrive can be used either as primary or supplemental storage. This is particularly useful for gaming subscription services–like Apple Arcade–where users sample a number of different games and need storage to cache game assets.

USB-C Connector for the Apple TV Remote

The Apple TV 4K’s remote control, like the iPhone 11, can be recharged with a Lightning cable. The MacBook and iPad Pro have already made the leap to the USB-C connector for power. The next Apple TV should also use USB-C connector to recharge its remote, if only to make it easier on consumers to have a standard charging cable for all types of devices.

Apple TV Remote Locator

We constantly lose our Apple TV remote. A simple solution would be a “Find My”-style remote locator, which causes the remote to sound a chime similar to locating your keys using Tile.

tvOS Software Improvements

The following software enhancements likely could be implemented on the older models like the Apple TV 4K and Apple TV 4.

Fixing Saved Passwords

My biggest headache using tvOS is entering and retaining saved passwords. Each application/service provider necessarily has its own password. For some reason, tvOS applications frequently lose their stored credentials and require them to be reentered. This happens across Netflix and TV Everywhere apps, in particular.

Yes, you can re-enter your credentials from a paired iOS device. But only the primary user can do that. This creates headaches both for the primary user to continually reenter credentials and for other family members when the primary user isn’t home to do that. Bottom line: saved credentials shouldn’t be forgotten. And, saved credentials should be centrally managed in the Apple Home app (leveraging Keychain) across all apps (e.g. Netflix) regardless of which Apple device first used that credential.

Fixing Multi-Profile App Installation Across Apple TVs

iOS 13 introduced multiple user profiles. Now, it is easy to switch between family members’ profiles by pressing and holding the TV icon on the Apple remote. But, that feature, combined with auto-installation of apps across all Apple TVs, can result in chaos as each Apple TV in your home requests the login credentials of the family member who installed an app on the first Apple TV. If your spouse first installed an app in Apple Arcade, now you need their credentials to install the app on other Apple TVs. And, you are prompted for each app each time, over and over again.

Better Third Party Game Controller Support

With tvOS 13, the Apple TV now includes third party game controller support for the Playstation DualShock 4 and the Xbox controllers. We purchased several PS4 DualShock controllers. Button function mapping is very idiosyncratic, not just within individual games, but across all Apple Arcade games. Buttons are mislabeled. The PS4 button invokes the control panel, not the home screen. The PS4 controller trackpad doesn’t work even when a game supports the Apple TV remote trackpad. We must frequently use the Apple TV remote to exit to the home screen to use other apps or games. There are games in Apple Arcade on tvOS that only work with Apple TV remote or an iPhone or iPad, not controllers. Ideally, each game would automatically know which controller (i.e PS4, Xbox, or legacy Apple TV controllers such as Nimbus SteelSeries) is connected and then map functions and in-game labels to that controller. There should be an option on tvOS to limit Apple Arcade’s results to games that require game controllers. Finally, tvOS should support firmware updates for the PS4 and Xbox controllers.

Context-Aware Siri Search

Why does Siri only search Apple services by default? Why must users first locate a search icon within an app to search within that application’s content library? If I am using an app, Siri should default to (or, at least, supplement) its native search results with app-specific results.

How broken is search? You can’t even use Siri to install an Apple Arcade app because it doesn’t appear in global search.

Support for Multiple Smart Assistants

It would be straightforward technically to permit Google Assistant and Amazon’s Alexa to show results on the Apple TV. Obviously, Apple would default to Siri by default. But, with each service having it’s own keyword, it isn’t difficult to invoke the proper smart assistant.

Apple Home Status on the Big Screen

All of your Apple TVs appear within Apple’s Home app on your iOS, iPadOS, and MacOS devices. But, Apple Home doesn’t display status or provide alerts on your TV via the Apple TV. This is an unfortunate reflection of Apple’s larger lack of focus on smart home integration, compared to Amazon and Google.

Dolby Atmos (beyond Streaming Services)

The Apple TV 4K supports Dolby Atmos, but only for Dolby Digital Plus enabled streaming services. If you are playing stored video with Dolby TrueHD that includes Dolby Atmos objects, application developers (such as Infuse) cannot passthrough Dolby Atmos via Dolby MAT 2.0.

Adjusting Colorspace (HDR) Conversion

If you leave Apple TV at its default setting for a DolbyVision-enabled TV, videos originally filmed or broadcast in standard dynamic range may look unnatural in terms of the brightness of their colors. My issue isn’t with HDR output per se but, rather, “brighter than reality” colorspace conversions and lighting transformation of SDR content. This continues to be an issue even with tvOS 13. In particular, I notice this on over-the-air television content like the news.

Vulkan 3D Graphics API

Ideally, tvOS (and iOS, iPadOS and MacOS) would include support for the Vulkan 3D graphics API. File this under “never gonna happen” due to Apple’s push for developers to use their Metal 2 graphics API. However, there was a time when Apple supported multiple open-source and industry standard initiatives. In fact, tvOS (along with iOS, iPadOS, and MacOS) still includes OpenGL though support has been deprecated and the bundled OpenGL version dates back to 2010, missing many current features.

For consumers using apps that already support Vulkan, tvOS support for the Vulkan 3D API would significantly enhance video quality and performance over the legacy OpenGL API. Developers could quickly deploy existing their Vulkan apps on tvOS. For example, both Channels and Plex use the MPV video engine. MPV currently supports Vulkan but not Metal. Similarly, other game developers have initially targeted Vulkan over Metal.

Alternatively, Apple should sponsor the development of Metal rendering abstractions for the most common video and game engines to accelerate native Metal support. This isn’t a major financial commitment relative to Apple’s resources.

Updated on July 5th, 2021