How do you describe a feeling? Not the mood you are in, but the physical sensation of what you touch? This is the challenging question asked when designing tactile effects for video. For creatives not used to working in a tactile medium, it can be somewhat intimidating.
An audio designer understands the intricacies of a sound. The film editor understands the complexities of the visual frame. However, the process for creating touch sensations in a digital format is actually not what you would expect. Most people would assume that it is the direct translation of your sense of feeling, but that is incorrect. Tactile design is closer to accompanying the motion of an actor or the sound of thunder.
Try this…try to think about the act of physically sensing something with your fingertips. For example, when you press the shutter button on your camera to snap a picture. Your mind has given that feeling a verb – like click or snap. You think that you hear a sound, but in reality, there is no sound. Your mind has interpreted that “feeling” to a verbal understanding of a sound, or even, something you might recognize visually. If you try to work that backward to create that “feeling” sensation, it does not work. The wave files don’t equal out.
The technology we use to create the haptics in mobile devices today – small actuators that jolt or spin – are not capable of absolute realism. They can’t deform to the shape of the cloth a character runs their hands through, nor can they simulate the sudden change in heat caused by an explosion. These technological constraints require that we rely on the use of illusion and metaphor, rather than realism, to determine the specifics of the tactile pattern. For example, when a ball bounces on the screen, you’re mind has already associated that visual with the real-life experience. Haptics’ role in this instance is to fill in the gaps to complete the picture further.
This may be daunting concept at first, but understanding few key guidelines can help you learn how to “create a feeling” for the specific scene in a movie, or an action point in a game.
1. As a traditional audio or video editor, you will need to suspend some of your respective best practices when working on haptic designs. While it is easy for those with an audio or video background to try to apply effects based on their respective disciplines, this new medium has new design parameters.
a. Keep open mind for the placement of effects, it may not make sense for the audio or video world, but might be a whole world of difference in haptics.
b. You have a basic knowledge of human perception, use it. It is key to imagine all of your senses working together to create a fuller picture.
2. Understand the key constraints of the actuators for which you are designing – and keep consumer experience in mind.
a. For example, short effects are great for building complex patterns while, long and strong buzzy effects will annoy the audience.
b. Haptic density can be pleasing when effects are frequent enough to fall into a regular cadence, like a musical beat, but there is no haptic equivalent to “room tone.”
3. Accept that while you cannot completely simulate reality, you may use the audio and video to fill in the gaps to generate a perceptual haptic realism—effects that the brain accepts as logical when coinciding with the other senses. Think of it as video + tactile + sound = whole picture.
Tricking the brain is one of the keys to a successful haptic track. Generally, a haptic designer will have the most success by applying tactile effects to the element most obvious in the scene, whether a video or audio element. This might be the largest object on screen, the actions of a speaking actor, or a loud Foley sound to shock the audience. Alternatively, a subtle hint towards the advancement of the story opens up the possibility of haptics for foreshadowing, much like smart editing or subtle audio cues.
Once you have moved past the obvious applications of haptics, you will discover “mood” haptics. These are tactile sensations that are not explicitly associated with any auditory or visual cues, but instead stand alone as their own expressive layer. Imagine a tense situation where the director wants to build suspense: haptics could employ the pattern of a heartbeat – that goes faster and faster as it gets closer to the scary moment climax.
While tactile design is a subtle and complex art, we have been able to successfully initiate both audio and film editors into the world of content haptics. Much like the traditional editing mediums, the more you experiment, the more you’ll internalize the design space and be able to achieve unique and pleasant effects. With regular feedback from a trained haptic designer, audio/video editors will be able to find success within the first few pieces of content they create – and claim touch as another medium that they can use to tell their story, creatively.
Earlier this year, LeTV, one of the top video apps in China, became the first to introduce haptics in movie trailer advertising to Chinese viewers. With the help of Immersion, LeTV launched a new “4D” channel in their Android LeTV app in June, featuring three haptic movie trailers: Expendables 3,” “Spectre” starring Daniel Craig as James Bond, and “Hotel Transylvania 2.”
To get a better understanding of how viewers are responding to this new creative capability, we’ve closely monitored responses on social media, the primary medium for all communications for businesses and consumer outreach in China. From what we’re able to see to date, LeTV “4D” channel was received well by consumers. It created considerable fanfare in the country, with many positive comments from users.
A snapshot of some of the user comments on Weibo.
While this is only one dimension of feedback from the campaign, positive commentary from users in China can be very telling of the overall sentiment. In China, social media is seen as an official communication medium, comparative to a corporate website.
This positive response reaffirms our belief that video enhanced with tactile effects provides a compelling entertainment experience.
As a designer, tactile video opens a new chapter of content user experiences on mobile devices. The sense of touch can be leveraged in a number of creative ways in future designs and content today. Being from China, my favorite phones for this experience are the Meizu MX4, Samsung Galaxy and Note series handsets. While the experience is good on many other phones, the quality is somewhat limited by capabilities of components and touch software in the phone. Great tactile content is only achievable when great chemistry is created between the hardware, software, and design.
It can be said that video enhanced with tactile effects can be a hard concept to imagine at first, but when users are able to feel the realistic tactile sensations of guns shooting, cartoon characters bouncing, and motors running, it comes as second nature to have tactile effects as a part of the entertainment experience, and users walk away delighted.
Since the launch of the channel in June, LeTV has added a new trailer featuring the “Minions” movie. We expect that as this feature evolves and LeTV continues to market its 4D channel, we’ll be able to bring more meaningful and interesting tactile experiences to consumers.
We’re just getting started.
Rennie Zhang works at Immersion as a UX design generalist focused on the China market.
We live in an age where screens are commonplace; our cars, our watches and even our grocery store checkouts have screens. But as humans, our basic nature still makes us crave physical experiences that digital interactions often don’t provide. In the past couple of months, our team has been working on our submission for SXSW 2016. (Look for my submission on “Feel it to Believe it – Everything needs Haptics”SXSW 2016). Earlier this year, I was fortunate enough to be able to speak at SXSW on ways designers can implement touch in the content they create. In the short months since then, there has been much more development and awareness on what haptic technology can add to the experience.
The goal of most digital content—brand experiences, entertainment, ads andapps—is to create an immersive, emotional and meaningful experience; but the content is trapped behind glass. Many consider the manipulation of content through the glass as “direct interaction,” but that is not true, since touch is not engaged.
In the case of touchscreens, we feel sensations on flat, smooth surfaces, and, in the case of gesture interfaces, we feel nothing but empty space. Tactile design is an emerging design practice, enabled (finally!) since now most content can be enhanced by tactile feedback. When tactile design is applied to the experiences delivered to phones and tablets, we can create a more immersive, physical and emotional experience than before.
Mobile content creators must consider the implementation of touch in their design. If they choose not to include, it should be considered a design choice and needs to be part of the conversation.
Mobile content is often produced with high production value. It looks and sounds great on a high fidelity playback environment. But, the consumer is likely to be viewing promotional content while waiting in line, sitting on a train or distracted by their environment; on a four-inch screen, with poor audio. So how can your content still be immersive? The use of tactile design engages your body and thus your attention. In this way, tactile design fits well with the content creator’s intent: it reaches people wherever they are.
Tactile Designs is like any other user experience design discipline. There are basic design rules that you need to understand as you’re working with the medium. In the spirit of encouraging everyone to think about touch in the experience they are creating, whether it is video content, gaming content or a device user interface, here are my design rules for the sense of touch (the same ones I shared at SXSW 2015).
The five rules to good tactile design
1. Understand the design space. Learn what tactile design is capable of, and how to apply it. This is a new field for most designers, so there’s an exploratory step that needs to be taken before it can be effectively integrated into the production process.
2. Follow the action. Your content is full of motion and action. What are the “tactile moments” in your brand? For example, Xiaomi partnered with Coca Cola to make a Coke theme for its mobile phone. The lock screen looks like a soda can top, and consumers unlock their phones by opening the can. That iconic feeling of opening a Coke can was re-created with tactile and sound design. Other brands have other iconic tactile moments. What’s yours?
3. Create an illusion. When you combine tactile with audio and visual, you can make your experience utterly convincing. One example is a demo where you see a ball rolling around on the screen, and you feel it “bounce” against the sides of the screen. The tactile bounce effects make this illusion very strong, and instead of seeing a picture of a ball rolling around, you begin to believe there might actually be a ball in the phone. When you take the visual away by turning off the screen, you still believe the ball is in there just because you feel it. Once you build an illusion with tactile design, the experience is more sticky and memorable than it would be without.
4. Build real connections between people and between your brand and your customers. Our goal is often to create real, emotional bonds. We talk all the time about how we want our brands to be emotional, social and meaningful. The tactile channel is the most direct way to achieve this because it literally touches people, moves people.
It’s important to note that good design is paramount to creating a touch experience. People have a lot invested in touch – it represents their emotions and expressions and can have a heavy impact on their experience.
Digital devices can now give us the physical experiences we crave. Thinking about what this means for the future, and you’re brought to an interesting conclusion: because these physical experiences are made with software, they’re not limited to just simulations of familiar physical experiences. Instead, they’re only limited by the imaginations of creatives. What will virtual physical experiences be like when they have no basis in material reality?