How do you describe a feeling? Not the mood you are in, but the physical sensation of what you touch? This is the challenging question asked when designing tactile effects for video. For creatives not used to working in a tactile medium, it can be somewhat intimidating.
An audio designer understands the intricacies of a sound. The film editor understands the complexities of the visual frame. However, the process for creating touch sensations in a digital format is actually not what you would expect. Most people would assume that it is the direct translation of your sense of feeling, but that is incorrect. Tactile design is closer to accompanying the motion of an actor or the sound of thunder.
Try this…try to think about the act of physically sensing something with your fingertips. For example, when you press the shutter button on your camera to snap a picture. Your mind has given that feeling a verb – like click or snap. You think that you hear a sound, but in reality, there is no sound. Your mind has interpreted that “feeling” to a verbal understanding of a sound, or even, something you might recognize visually. If you try to work that backward to create that “feeling” sensation, it does not work. The wave files don’t equal out.
The technology we use to create the haptics in mobile devices today – small actuators that jolt or spin – are not capable of absolute realism. They can’t deform to the shape of the cloth a character runs their hands through, nor can they simulate the sudden change in heat caused by an explosion. These technological constraints require that we rely on the use of illusion and metaphor, rather than realism, to determine the specifics of the tactile pattern. For example, when a ball bounces on the screen, you’re mind has already associated that visual with the real-life experience. Haptics’ role in this instance is to fill in the gaps to complete the picture further.
This may be daunting concept at first, but understanding few key guidelines can help you learn how to “create a feeling” for the specific scene in a movie, or an action point in a game.
1. As a traditional audio or video editor, you will need to suspend some of your respective best practices when working on haptic designs. While it is easy for those with an audio or video background to try to apply effects based on their respective disciplines, this new medium has new design parameters.
a. Keep open mind for the placement of effects, it may not make sense for the audio or video world, but might be a whole world of difference in haptics.
b. You have a basic knowledge of human perception, use it. It is key to imagine all of your senses working together to create a fuller picture.
2. Understand the key constraints of the actuators for which you are designing – and keep consumer experience in mind.
a. For example, short effects are great for building complex patterns while, long and strong buzzy effects will annoy the audience.
b. Haptic density can be pleasing when effects are frequent enough to fall into a regular cadence, like a musical beat, but there is no haptic equivalent to “room tone.”
3. Accept that while you cannot completely simulate reality, you may use the audio and video to fill in the gaps to generate a perceptual haptic realism—effects that the brain accepts as logical when coinciding with the other senses. Think of it as video + tactile + sound = whole picture.
Tricking the brain is one of the keys to a successful haptic track. Generally, a haptic designer will have the most success by applying tactile effects to the element most obvious in the scene, whether a video or audio element. This might be the largest object on screen, the actions of a speaking actor, or a loud Foley sound to shock the audience. Alternatively, a subtle hint towards the advancement of the story opens up the possibility of haptics for foreshadowing, much like smart editing or subtle audio cues.
Once you have moved past the obvious applications of haptics, you will discover “mood” haptics. These are tactile sensations that are not explicitly associated with any auditory or visual cues, but instead stand alone as their own expressive layer. Imagine a tense situation where the director wants to build suspense: haptics could employ the pattern of a heartbeat – that goes faster and faster as it gets closer to the scary moment climax.
While tactile design is a subtle and complex art, we have been able to successfully initiate both audio and film editors into the world of content haptics. Much like the traditional editing mediums, the more you experiment, the more you’ll internalize the design space and be able to achieve unique and pleasant effects. With regular feedback from a trained haptic designer, audio/video editors will be able to find success within the first few pieces of content they create – and claim touch as another medium that they can use to tell their story, creatively.
When Raj from CBS’s The Big Bang Theory was tasked by NASA to come up with a delivery system for a universal message in case one of NASA’s Discovery missions encounter aliens (The Communication Deterioration episode, aired April 16, 2015), he enlists his fellow scientists for help. After trading theories and arguing in a side debate on who is Alpha and who is Omega in the group, they come to the conclusion that they need to develop a device that can deliver a message through not only sight, but through other senses, or in scientific terms other “perceptual modalities.”
Their “aha” moment comes when they realize that aliens might communicate in a totally different way than humans — they might not have eyes or ears. Taking cue from the animal kingdom, the guys contemplate on the fact that animals communicate through scent, bees dance to talk to each other and whales have their songs. [Spoiler alert] While it can be questioned if their pin board haptic communication system does the job of welcoming the aliens, after all the aliens of the future want to eat Sheldon, haptics were used as part of the communication system to convey the message from Earth. In case, as Sheldon put it, “Who knows if they even have mouths.”
All comedy and science stuff aside, touch is the most powerful sense in the universe. It may be the universal language for all species of life, extraterrestrial, animal and humanoid. Just think about the iconic image of Steven Spielberg’s E.T. reaching out with his finger to touch his human friend Elliot. It is ingrained in our memory because it tugs at our heart strings and makes E.T. less alien.
At Immersion, we talk often about touch as a human sense and how its absence from our digital environments has forced us to be coy with the way that we connect with others through technology, such as 🙂 or 😉 or : or even a :o?. It is interesting to think and talk about how touch extends beyond the human species, (what dog doesn’t like a belly rub from its human?) yet we haven’t full embraced its capability throughout our world.
For us humans, there is an opportunity for haptics to be used as a communication language in situations where sight and sound aren’t enough. Touch stands on its own as a language. In fact, there are multiple ways in which touch can communicate. Touch can tell you when something is hot or cold. It can tell you when something is painful. It translates emotions, texture, proximity and value. These are all different kind of senses that can be interpreted through touch.
From a heartfelt hug to greet or console, to a tap on the shoulder to signal your presence, to holding hands while strolling through the park, humans build true connections through the sense of touch. As our world becomes more and more digital and a little less personal in the process, haptics bring back those feelings of being together in a powerful emotional way.
An example of how haptics are being used to communicate today is in the wearable device, which presents a unique opportunity for touch or haptic feedback because the device is always touching the wearer’s skin. Notifications that are not seen or heard can be felt. Immersion’s TouchSense Core, along with our Instinctive Alerts Framework, provides the technology that wearables need to communicate information to the user in the most intuitive and meaningful way.
As I spend my days and nights building out TouchSense Engage, Immersion’s platform for rich communication through haptics in content and media, I’m not thinking of how the technology will work for aliens. I do feel that there is such a rich future in the way technology and entertainment companies will embrace haptics. Soon enough, we won’t accept playing Angry Birds without feeling each successful demolition or watching Fast 7 without sensing the roar of the engine.
While NASA may not really be willing to send one of our haptic devices into space to communicate with aliens just yet, it’s important for us to remember how communication is changing and how the language of touch will be a part of the revolution.
The shift towards mobile is here. There is no denying that in the past 10 years the viewing habits of the consumer have trended towards consumption via smaller, more mobile devices. This poses a conundrum for directors who aim to craft their stories for a theatrical experience. How does a director who stages scenes for a 20 by 50 foot projection simultaneously optimize [it] for a four by two inch screen held in the hand? And, why should they?
Is there a way to creatively tackle this problem to maintain the immersive integrity of the big screen and simultaneously create a complementary unique immersive experience on the small screen?
The realities of digital distribution and modern viewing habits is acting as a force function on directors, producers and distributors. They must keep the small-screen experience in mind to ensure they reach the broadest possible audience, multiple times over. According to Nielsen ratings, mobile viewership is up in the double digits. In fact, at least 72 percent of mobile users use their device weekly to view video content. Couple those findings along with the lowest theatrical audience attendance numbers in years and the message from audiences of all demographics is, content needs to be accessible whenever, however and wherever.
Of course, with every new change to a cherished medium there are detractors. Many in the entertainment industry are trying to fight the focus on mobile for long form content. Some directors generally do not like viewers to watch a movie on a mobile device because it is next to impossible to capture the theatrical experience on such a small screen. The drop in theater attendance is especially daunting for them, since the theater provides an uninterrupted immersive and communal experience that is difficult, if not impossible, to replicate on a mobile device.
If mobile viewing is not going away anytime soon, and the theatrical experience is just that – a theatrical experience – then it’s time to create a mobile experience alongside or as a companion piece to every film created today. To accomplish this, the creative minds within the entertainment industry need to embrace a new creative process to capture the mobile user’s attention and allow filmmakers to adapt to the audience’s needs; so they can showcase aspects of the theatrical experience on a smaller screen. For this case, a whole new creative medium is needed. This is where we can make the case for haptics, the ability to bring the sense of touch as a part of the artistic story line.
Unlike other forms of entertainment, the mobile user is actually holding the viewing device in their hand, giving directors the opportunity to allow the viewer to feel the movie experience like never before. With the implementation of haptics, directors can convey feelings or moments with the use of touch feedback, enhancing the viewer’s experience and creating a more engaging, fun and exciting experience for the smaller screen. This is a unique opportunity, since mobile is the only medium where the audience has the movie at the tips of their fingers and this interaction cannot be rendered in the same way in a theatrical experience. The sense of touch is a whole new element to the media landscape and it will provide directors with a new creative outlet, in turn, providing merit to the small screen viewing experience so many viewers already utilize.
Immersion anticipates that haptics will be a core part of mobile content in the future, much like when sound was first introduced to moving pictures and adopted over time. Haptics can ensure broader audience reach as a new and compelling enhancement to the story that can only be felt on a mobile device. When haptics are added to content, it is proven that the viewer pays attention for a longer period of time. The experience becomes more immersive, just like adding surround sound or high-end visual effects.
Filmmakers have always embraced new technology. Because of this, they have improved technology and made the art of storytelling more and more compelling for their audience. Touch is a perfect creative solution for mobile. It differs from the other forms of digital technology that are being explore right now as it doesn’t try to alter audio or video. You are not required to buy fancy glasses, a headset, or better quality audio equipment in order to view it. Consumers are not expected to grow accustomed to touch. It’s already ingrained in the natural way that we experience the physical world. These are the reasons why haptics has the ability to revolutionize the entertainment space. It is the next natural addition in this ever-evolving medium; whether it be for mobile devices now or the viewing devices of the future.
Mobile has changed the dynamic of gaming in a dramatic way.
Over the last few years, game developers have grown accustomed to mobile as one of the platforms in which they can engage the consumers. The approach for designing a top-tier mobile game still includes the majority of the same components – game design, action, gamer engagement, sound design, user retention strategies. However the way that these components are executed in the mobile gaming platform can have a huge impact on success.
In a recent contributed article on Exploring Mobile Games as An Engaging Platform, posted in gamesauce, last month, I recapped a conversation I had during panel with a few industry leaders, namely Jeff Drobick of Tapjoy, Jeffrey Cooper of Samsung, and David Zemke of DeNA. On the panel we debated the balance between the left brain and the right brain when it came to designing mobile games.
The one consensus across the industry experts was that these creative game design remains the highest priority in gaining and retaining users. It makes sense. After all, you can’t even start to think about how you can make money until you have gamers playing your game. From that perspective every piece of the design is a balancing act. The mobile device is a unique platform because of the way gamers use their handsets. (See my blog post on Embracing the Mute Button). Maintaining a lasting relationship with the user is exponentially harder on the mobile device, however the opportunity to target a broader audience makes it a worthwhile effort.
This is an ever present component of how we look at tactile design at Immersion. There is huge benefit to adding tactile effects to games. We know it increases retention, improves people’s intent to share their experience with others, and provides users with an overall better impression of the game design quality. We also know that at some point the intent in a tactile design can go bad. This happens when design principles are not followed and the strategy for tactile design is not completely thought through.
Touch is a great engagement tool and to do it right can mean a lot to your users, and can result a more elegant game design, higher retention and new monetization opportunities. Getting the right balance in means that the focus should be on improving game design and increasing retention before monetization consideration. It is possible to use haptics for both. We should continue to explore how users play their mobile games to find other creative avenues for monetization as we consciously design haptics to improve game play and engagement.
For those of us in the content business, the big draw at this year’s International Consumer Electronics Show was the intense focus on immersive entertainment and technology. (CES highlights the focus on Immersive Entertainment from gaming and other entertainment companies on the show floor.)
When you think immersive anything, haptics has to be a part of the conversation. The sense of touch is able to transforms consumers’ digital experience unlike any other sense. It can trigger memories, personal experiences and emotions.
Whether virtual or augmented reality, mobile gaming, second screen experience or TV everywhere, the industry is delving deeper into how to create ultra-immersive, engaging entertainment experiences for the consumer. Dolby and DTS are making waves with immersive sound technology, driving up consumer expectation for a more realistic audio experience available in the home. Quantum Dot, SUHD, 4K and OLED technologies are creating quality in displays beyond what the eye can see. The sense of touch is an untapped potential in extending the experience to further engage the viewer.
This is an inflection point for the content and media industry. Mobile gaming and viewing is becoming arguably the first screen that reaches the consumer. Smaller screens and more public environments are impacting the viewing experience, changing the dynamic of the media and entertainment market. Immersion’s TouchSense Engage, on the other hand, adds another piece to the experience — the use of touch to directly connect with viewers.
There was so much more at this year’s CES – from androids and drones, to wearables and smart cars, ranging from to publicity stunts meant to grab headlines, to the very viable, and the already in the market. There’s much to explore. I’m excited to see what we can do together to create this new immersive world.
I’d be interested in hearing your thoughts on this year’s CES and what it means for the content industry. Reach me and our team at content@immersion.com.
Jason Patton,
Vice President & General Manager of Content & Media Business for Immersion