What is Meta’s awesome Surface Touch Typing Technology that Mark Zuckerberg highlighted?

Have you ever imagined typing on any flat surface by just touching it? Do you know, now you can type even when you don’t have a keyboard? Research is being conducted to develop the surface touch type technology and Meta seems to have crossed a milestone recently.

In a report by The Verge, Mark Zuckerberg, CEO of Meta, revealed his impressive typing speed of 100 words per minute (WPM) while donning a virtual reality (VR) headset. What's even more remarkable is Meta's claim that it can transform "any flat surface" into a virtual keyboard capable of achieving speeds of up to 120 WPM. This breakthrough represents a significant leap from Meta's earlier technology, as demonstrated by their 2020 "PinchType" method that averaged a mere 12 WPM. However, in the same year, their "surface touch typing" achieved an average speed of 73 WPM.

Surface Touch Typing Technology

Meta's latest development showcases its dedication to advancing text entry methods for VR and augmented reality (AR) environments. As per a blog by Meta. a groundbreaking text decoding technique that enables touch typing on a flat, uninstrumented surface. This method eliminates the need for physical keyboards or capacitive touch interfaces. Touch typing relies on hand motion captured through hand-tracking technology. This motion data is then directly decoded into text characters, resulting in a seamless and efficient typing experience.

Meta uses a temporal convolutional network, serving as a motion model that translates hand motion – represented as a sequence of hand pose features – into textual input. One key challenge addressed by Meta's researchers was accounting for erratic typing motion caused by finger drift, given the absence of haptic feedback from physical keys. To overcome this, the company integrated a language model as a text prior and employed beam search algorithms to intelligently combine the motion and language models. This fusion enables the accurate decoding of text from both ambiguous and erratic hand movements.

To validate their approach, Meta collected a dataset from 20 touch typists and subjected their model to various benchmarks, including contact-based text decoding and traditional physical keyboard typing. The results speak volumes: their proposed method leverages continuous hand pose data to outperform contact-based techniques in terms of text decoding accuracy. An offline study demonstrated parity with typing on a physical keyboard, achieving a speed of 73 WPM with an impressive 2.38% uncorrected error rate.