This month we\u2019re going\nto delve into the middle layer \u2013 the software layer. We\u2019ll primarily focus on\nsoftware that is used to build products and applications and leave application-level\nsoftware for a later post. <\/p>\n\n\n\n The software layer interfaces\nwith the driver electronics (below) as well as with applications (above) that\nutilize haptics. <\/p>\n\n\n\n The software layer is responsible for orchestrating the\nhaptic hardware by generating signals that can be rendered on that hardware in\nresponse to application state or user interactions. It is important to think\nabout the representation of these signals to understand the choices and\ncapabilities available as related to system-level software.<\/p>\n\n\n\n At the lowest level, it is common for haptic driver ICs to\nprovide some type of buffer or other commands in on-chip storage that can be triggered,\nand which will result in a voltage being generated. The data stored in these\nbuffers is normally very close to the signal that is sent to the amplifier and\nis always driver IC-specific. This capability is typically used for haptic\nproducts that don\u2019t have an operating system since the buffer can be triggered\ndirectly with an I2C signal or other low-level commands. Normally these effects\nare used for button-like effects because this configuration has very low\nlatency and limited dependence on software state. The effects themselves are\nalso specific to the actuator and driver present in the system. While in this\ncontext, the capabilities of haptic effects are highly dependent on the\nhardware choice, upgrading to better haptic hardware doesn\u2019t necessarily equate\nto more capabilities. With a higher\ngraded actuator and higher performing driver IC,<\/a> the software layer plays a\ngreater role in leveraging the capabilities of the haptic hardware. <\/p>\n\n\n\n A more general representation of haptic effects is a timed\nseries of amplitude values with a regular sampling rate. Operating systems such\nas Android and iOS can digest buffers of such signal data, typically with a 1ms\nsampling rate, and render them to the available haptic hardware. This format is\nlow level, but it is easy to generate using audio tools like Audacity. A key\nchallenge with this effect encoding is that the haptic experience that will be\ngenerated will vary widely in practice due to mechanical variance at the motor\nlevel. As I noted in last month\u2019s blog, every haptic actuator has distinct and\ndifferent performance characteristics. With a basic signal level\nrepresentation, these variances must be taken into account when the effects are\ncreated. For products that all use the same motor, this can be acceptable.\nStill, for application software that is intended to run on different devices\n(e.g., different Android models), the resulting experience will vary from \u2018exactly\nas intended<\/em>\u2019 to \u2018noisy and confusing.<\/em>\u2019<\/p>\n\n\n\n Another approach to haptic effect representation is to\ncapture the design intent of the effect using an abstract signal\nrepresentation. Abstract signal representations are conceptual by design and\nmust be interpreted by the OS software and then used to synthesize a signal\nlevel effect (usually at runtime). The synthesis step can be accomplished by\ntransforming the conceptual effect description using a motor model that\nrepresents the actual hardware on the device. This approach effectively\ndecouples and abstracts the hardware dependence from the effect design activity\nand is an important component of a general haptic software solution, particularly\nfor markets that have applications that will run on different hardware.<\/p>\n\n\n\n There are a few important in-market haptic software APIs to consider, but, notably, there is a wide variance in approach and sophistication. This variance likely represents a key barrier for application developers who want to make use of haptics since they will normally need to reimplement\/test their application on each platform and hardware model to ensure consistency of experience.<\/p>\n\n\n\n Core Haptics was released in mid-2019 and is one of the most sophisticated in-market examples of a haptic API. Core Haptics uses an abstraction model that enables effects to be specified using ADSR (see Wikipedia<\/a>) envelopes. These envelopes are parsed at runtime within iOS and used to synthesize a motor-specific signal that is sent to the amplifier and ultimately to the actuator. Core Haptics is a rich and expressive API, and I highly recommend reading this blog post at Lofelt<\/a>, which provides a great technical overview.<\/p>\n\n\n\n Android has supported vibration control since at least version 2.0. The API is documented on this website<\/a>. Android doesn\u2019t support effect abstraction and instead provides only a low-level signal interface, which is represented as a buffer of 8-bit amplitudes that are sampled at 1ms. Although this API is very capable, it doesn\u2019t provide any hardware abstraction to developers and thus suffers from the issues described above.<\/p>\n\n\n\n Last year, Khronos Group released OpenXR 1.0 (link<\/a>),\nwhich includes support for haptic feedback devices. This API is targeted at XR\nand gaming use cases and is intended to provide consistent, low-level hardware\nabstraction that enables high-performance applications. The effect encoding in\nOpenXR is a list of individual vibration effects, each of which has a fixed\nduration, frequency, and amplitude. Conceptually, this API is a bit more\nsophisticated than the signal level API in Android. Still, it doesn\u2019t\nmeaningfully abstract the tight hardware dependence that is a hallmark of\nhaptic feedback and will likely suffer from the same inconsistencies present in\nAndroid if the same effect lists are played on devices with different\nactuators.<\/p>\n\n\n\n Given that background, let\u2019s look at some use cases of these haptic APIs and discuss the capabilities and shortcomings of each.<\/p>\n\n\n\n Short, high amplitude\neffects are the hallmark of button replacement use cases. Recall the quality of\nthe Apple haptic home button when it was first released. Many users were\nconvinced that it was a real button. This type of use case is normally\nsupported as a driver IC-level effect buffer. The advantage of this is that the\nsystem has very low latency. Since buttons are normally static, it is\nacceptable to have limited software control over the effect itself.<\/p>\n\n\n\n Mobile games have traditionally not made extensive use of the built-in haptic capabilities of mobile devices. Now that Apple has enabled developers to create haptic effects in their apps, we expect this to change. The ADSR abstraction introduced in Core Haptics, along with tight integration with audio playback, is ideal for haptic game effect design. This API enables developers to create haptic and audio effects together and manage them as similar assets. It would be nice to see similar functionality come to Android devices in Android 11.<\/p>\n\n\n\n <\/p>\n\n\n\n The new PS5 Dual Sense controller will have both HD haptics and active trigger feedback. Both of these capabilities are sophisticated and have a lot of potential for game experiences. However, it is not clear that the OpenXR API is sufficient to enable this platform, and Sony has not (publicly) released any developer tools. It would be ideal if an abstraction-level effect encoding and API existed to facilitate widespread and cross-platform usage of advanced haptics in the next generation of console and PC gaming.<\/p>\n\n\n\n This month we reviewed the software layer of the haptic stack. As with the hardware layer, there are a lot of disparate choices and approaches, and each one impacts the ability of a product to fulfill its haptic features and requirements. This layer is also the layer that could benefit from enhanced industry standardization both in terms of APIs and in terms of effect encodings. Next month, we\u2019ll look at the top of the stack and examine how haptic experiences are designed and how this ties together the entire stack. We\u2019ll see you then!<\/p>\n\n\n\n Related articles: <\/p>\n\n\n\n The Haptic Stack<\/a><\/p>\n\n\n\n![\"\"](\"https:\/\/www.immersion.com\/wp-content\/uploads\/2020\/05\/Article_stack-software-1024x420.png\")
<\/figure>\n\n\n\nEffect\nRepresentations<\/h2>\n\n\n\n
![\"\"](\"https:\/\/www.immersion.com\/wp-content\/uploads\/2020\/05\/Article_software-diagram-1024x242.png\")
<\/figure>\n\n\n\nSome Important APIs<\/h4>\n\n\n\n
![\"\"](\"https:\/\/www.immersion.com\/wp-content\/uploads\/2020\/05\/Article_logo-apple_1.jpg\"){kind=logo}
<\/figure>\n\n\n\nCore Haptics (iOS)<\/h4>\n\n\n\n
![\"\"](\"https:\/\/www.immersion.com\/wp-content\/uploads\/2020\/05\/Article_logo-google_1.jpg\"){kind=logo}
<\/figure>\n\n\n\nAndroid Vibrate (Google)<\/h4>\n\n\n\n
![\"\"](\"https:\/\/www.immersion.com\/wp-content\/uploads\/2020\/05\/Article_logo-khronos_1.jpg\"){kind=logo}
<\/figure>\n\n\n\nOpen XR 1.0 (Khronos)<\/h4>\n\n\n\n
Software Use Cases<\/h2>\n\n\n\n
Mobile\nand Automotive Confirmation Effects<\/h4>\n\n\n\n
![\"\"](\"https:\/\/www.immersion.com\/wp-content\/uploads\/2020\/05\/Article_mechanical-switches-effect.jpg\")
<\/figure><\/div>\n\n\n\nMobile Gaming Effects<\/h4>\n\n\n\n
![\"\"](\"https:\/\/www.immersion.com\/wp-content\/uploads\/2020\/05\/Article_Game-Controller-Effects.jpg\")
A. A short double pulse with a finite space between the pulses. The second pulse is much lower in magnitude than the first.
B. A quick repeating pattern of pulses that ends with a long decay on the last pulse
C. Four separate pulses with alternating strong and moderate magnitudes
<\/figcaption><\/figure><\/div>\n\n\n\nGame\nController Effects<\/h4>\n\n\n\n
![\"\"](\"https:\/\/www.immersion.com\/wp-content\/uploads\/2020\/05\/Article_Gunshots.jpg\")
An explosion followed by gunshots <\/figcaption><\/figure><\/div>\n\n\n\nSummary<\/h3>\n\n\n\n