Apple’s new patent hints at powerful video collaboration tools

FaceTime has never quite lived up to its potential. Not only did it not become a cross-platform application that played nice with others, but by aiming full square at consumer markets, Apple never attracted a business audience to the platform. Even Apple itself uses Cisco’s Webex for business communications. But perhaps things will change, one day.

FaceTime, now with added video intelligence

A newly published Apple patent (US Patent Number 20230109787) describes numerous ways Apple may plan to help FaceTime become a more useful communications tool for enterprise users — at least those who use Apple’s systems.

Filed September 2022, published in April ’23, and first spotted by Patently Apple, the patent describes a FaceTime capable of gesture recognition (seemingly referred to as “Air Gestures”) with a particular focus on collaboration.

“The present disclosure generally relates to embodiments for video communication interface for managing content that is shared during a video communication session,” Apple explains in the patent abstract.

Many of the descriptions seem to reflect existing features within Desk View, so it is possible the patent refers to that. But it is also possible it hints at future improvements for that popular feature.

What improvements are described?

The improvements coalesce around imaging and gesture recognition. Skimming swiftly through the document, you find descriptions that give some sense of what these improvements might mean in practice:

• During a conference, a user might point at an object within the frame for the camera to focus on that object.
• Placing your finger on your lips might change your FaceTime background.
• Zoom in or out on part of the screen by pinching fingers together or away from one another.
• Zoom in on a chosen object by pointing at it.
• FaceTime becomes capable of translating text by converting it into the language used on the destination system.
• Apple has also described color management to ensure all participants are seeing the same shades.
• FaceTime may become able to take a drawing made on a physical notepad and turn it into a digital sketch, which all collaborators could then edit and review.
• Another interesting feature described in the patent talks about sharing different iterations of virtual demonstrations on a loop, so that collaborators can really get to grips with the different examples while making their decisions.
• You also get the capacity to share multiple views of an object within one conversation.

Some of the ideas described here are simply next steps to some of the popular features Apple more recently introduced within FaceTime, particularly Desk View, Continuity Camera, and Freeform. (When you use an iPhone and Mac, Desk View lets you show your face and an overhead view of your desk at the same time; Continuity Camera lets you use your iPhone as a webcam).

How might these enhancements work in practice?

Apple evidently wants to build on these features. For example, in Desk View you might be sharing a drawn sketch using a pad of paper placed within range of the downward facing camera. The camera will pick up on what you are doing and present it to others in a format that lets them zoom in and out, and also edit that image.

You’ll also be able to choose to share items within the meeting, launch presentations, and share your physical desk as a top down view using the camera on your Mac, iPad, or iPhone (which is what Desk View already does).

The introduction of gesture controls is more interesting. After all, if FaceTime can learn to recognize gestures during a collaboration session, why shouldn’t similar gesture recognition functions become usable within other applications? The Mac or iPad you are working with has a camera, after all.

Apple also glosses over personalization within a few lengthy paragraphs mostly concerning user privacy. However, use of these systems to share health-related information or symptoms seems to be part of Apple’s vision for FaceTime. I can easily imagine a doctor asking where a patient hurts and the patient pointing to the painful spot while the camera zooms in.

Keeping an AI on you

What’s also interesting about these new FaceTime features is the extent to which they relate to Apple’s ongoing work on machine vision intelligence. The company is doing a great deal of work concerning this in accessibility, and also within photos, video, and elsewhere on its systems.

This makes particular sense, given that the two biggest products everyone thinks Apple is working on (autonomous cars and mixed reality headsets) will really need reliable systems to understand the world around them. A car needs to know the difference between a tree and a road just as much as a goggles wearer needs to make sure they don’t trip over the sofa while they explore virtual worlds and conquer infinite space.

I do think one feature Apple could spend time thinking about is to exploit its existing Live Captions capacity within FaceTime to automate transcription, possibly extending to the provision of intelligent synopsis of conversations and auto-population of things like meeting requests, emails, or the collation of documents.

What next?

While it is worth noting that patented ideas don’t always become shipping products, the concepts described within the patent all seem to be within Apple’s existing capacity to provide, so I think enhancements based on them within FaceTime are likely.

In addition to which, I can’t help but reflect that Apple actually does cooperate with third-party developers of video collaboration software, so I imagine it’s possible we’ll see Webex and Zoom potentially offering similar features to those described in the patent, assuming Apple permits them access to the relevant APIs.

Cisco’s Webex, for example, recently introduced support for picture-in-picture and Stage Manager when run on an iPad.

Please follow me on Mastodon, or join me in the AppleHolic’s bar & grill and Apple Discussions groups on MeWe.