Enhancing Virtual Experiences: A Holistic Approach to Immersive Special Effects ^†

Abstract

To create a more immersive experience, electronic content developers utilize hardware solutions that not only display images and produce sounds but also manipulate the viewer’s real environment. These devices can control visual effects like lighting variations and fog, emit scents, simulate liquid effects, and provide vibration or locomotion sensations, such as moving the viewer’s chair. The goal is to emulate additional sensations for the viewers and engender the belief that they are truly present within the virtual environment. These devices are typically found in specially designed cinemas referred to as xD cinemas, such as 4D, 5D, 9D, etc., where each effect is treated as an additional dimension, enhancing the overall experience. Currently, all of these effects are triggered by timers. The system determines which effect to play based on timers. This approach is problematic, for it requires programming each device for each movie. In this research, we address this problem by introducing the idea of Special Effect Tags (SETs) that can be added in the subtitle files. The SETs aim to serve as a standard that will allow the various devices to know when each artificial phenomenon should be triggered. They are generic and can support infinite artificial phenomena, also known as dimensions. This paper introduces the idea of a common special effect framework and a generic architecture of a special effects player that is independent of any specific hardware solutions.

1. Introduction

Ever since Morton Heilig (1926–1997), an American cinematographer and a virtual reality (VR) pioneer, developed the ‘Sensorama Simulator’ [1], researchers have been trying to enhance digital content (such as movies) with new special effects sensations that are more environment-based. Examples of such sensations include smoke, moving chairs, scent, wind, temperature changes, haptic feedback, atmospheric effects (like fog or mist), dynamic lighting, vibration, and spatial audio. They are designed to immerse the audience in a multi-sensory experience, enhancing the realism and emotional impact of the content they are engaging with, whether in movies, VR experiences, theme park attractions, or interactive exhibits.

Although xD (e.g., 4D, 7D, etc.) cinemas are the typical place in which special effects sensations could be experienced [2], further advancement in hardware and software technologies made the generation of such effects more affordable and easier to implement in various entertainment settings. As a result, these captivating experiences have started to extend beyond specialized venues, reaching a wider audience through home entertainment systems, VR setups, interactive exhibits, and even mobile devices. This democratization of special effects technology has opened up new creative possibilities and opportunities for content developers to engage audiences in innovative and immersive ways. There are various types of special effect generators, including wearable devices and environment-placed gadgets. In this research, we have undertaken the task of classifying these devices into five distinct categories, based on the following features:

• Light manipulation (dimming of color light lamps),
• Smoke/fog/haze generation (stage fog machines),
• Temperature manipulation (making a surface hot or cold),
• Haptics (kinesthetic and tactile, vibration and movement of an object, e.g., a chair),
• Olfactory displays (devices that release scents).

New hardware and software solutions enable the integration of digital content, such as movies and video games, with a range of physical special effects, such as the release of scents, motion, and room light control, among other things. These effects aim to enhance the content’s immersion [3] and influence the user’s mood [4]. Experiments conducted with olfactory displays [5], thermal manipulators [6], and light manipulators [7] have demonstrated that each of these technologies can provide additional sensory clues that enrich the overall content experience.

There have been several attempts to associate individual effects with digital content. In one study [8], researchers deployed deep learning to associate the release of scents via an olfactory display with runtime game images (screenshots). Building on this work, another study [9] expanded the association of scents to include both images and audio. In [5], the researchers proposed an automatic association of scent release with subtitles using text processing and a dictionary. In the realm of VR, the researchers in [10] developed a system called ‘Season Traveler,’ which integrated a Peltier-based thermal sensor and an olfactory display into a VR headset. This system released scents and adjusted the temperature according to the season within the virtual world. In a different approach, the researchers in [11] designed an application-specific olfactory display that associated virtual cooking-related digital content with the release of scents. Beyond olfactory displays, the entertainment field has also utilized special atmospheric effects through smoke/fog/haze generators, commonly known as stage fog machines [12]. However, unlike other special effect generators, there may be potential health issues associated with long exposure to stage fog machines [12]. The experience of watching movies can also be improved by haptics [13]. Haptics are usually divided into kinesthetic and tactile. Kinesthetics deal with the position of the body parts, whereas tactiles handle the sensation of touch. Haptic generators do not have to be large, expensive electro-mechanical devices. For example, the researchers in [14] developed a small size device called DualVib that simulates a dynamic mass by simultaneously delivering pseudo-force and texture feedback.

The researchers in [7] manually associated a Philips Hue lights system with story-telling. In their work, they associated imaginary story scenes, such as outdoor, forest, river/lake, water, indoor and more to various light scenes (two lamps with various color combinations). The storyteller could manually select the light scenes in accordance with the corresponding part of the story. During their evaluation, they found that this approach was effective, but it was difficult for the storyteller to select the appropriate light scene. In summary, there is still potential for further research in the field of integrating special effects in digital content. Specifically, although researchers managed to associate special effect sensations with digital content, this remains application-specific, as it appears from [8,9]. However, none of these solutions is generic, resulting in xD cinemas relying on timer-based solutions. To the best of our knowledge, this is the first system that proposes the association of any type of special effects with digital content. The next section explains, in a generic way, the special effect ecosystem. In Section 3, we introduce the various subtitle formats. Section 4 presents an architecture of a special effect player. Section 5 discusses this work. Finally, Section 6 concludes this work and lays down some proposals for future work.

2. The Special Effect Ecosystem

While the proposed solution is generic, it assumes that the software implementing it has access to Internet of Things (IoT) devices that generate various special effects. It is also assumed that these devices are interconnected in smart homes or other similar platforms. Smart homes are platforms that utilize IoT technologies to provide home automation and quality of life solutions to consumers [15]. Our solution introduces the concepts of a special effect controller and effect generators. A special effects controller can be a video player/plug-in or any other similar software that is responsible for displaying the audio-visual content (such as a movie) on a monitor or any other device used to generate audio-visual content. The effect generators are IoT devices that generate special effects. They are connected to an IoT hub (also known as bridge) such as a Hue Bridge or to an online service hub such as WeLink, usually via Wi-Fi.

As it can be seen in Figure 1, the effect generators are connected with the ‘effects controller’ via an online service hub or an IoT hub. The controller will read the subtitles to know which effect should be played. This process is explained in detail in Section 4.

3. Special Effect Tags in Various Subtitle Formats

There is no widely accepted format for subtitles. Different applications tend to favor different formats.

Table 1. Popular subtitle format.

Name	File Extensions	Application
SubRip Subtitle	SRT
SubStation Alpha	SSA
	SBV	YouTube format
Distribution Format Exchange Profile	DFXP
Web Video Text Track	VTT
Timed Text Markup Language	TTML
Untimed Text Transcript	TXT

presents the most popular subtitle format.

The original idea of the authors was to include the special effect tags (SETs) as comments in the subtitles. However, popular subtitle formats, such as SRT, do not support comments. Hence, we decided to include it at the end of the actual text that appears in the film, using the square bracket tags “[” and “]” which are supported by SRT. The following presents an example:

2

00:05:16,400 --> 00:05:25,300

Who goes there?

[Sound of heavy rain]

This is already used in movies and is not expected to cause disturbance to the audience. If the video player is loading the subtitles from the .srt file, then the special effect generator will be able to play the special effects even if the user has disabled the subtitles. The special effects controller will read all of the text within square brackets and decide if and which effect should be played. The phenomenon to be played and its density (how impactful) needs to be extracted from the text. In this proposal, the density of the effect is fuzzy (it can be low, medium, or high). In general, the words in

Table 2. Density extraction.

Low	Medium	High
Low, Inferior, Small, Minor, Lessened, Below, Under, Low-lying, Low-set, Low-slung, Lowered, Shallow, Base, Diminished, Low-pitched, Subdued, Soft, Muted, Humble, Modest, Deficient, Poor, Weak, Tiny, Miniature, Petite, Little, Mini, Compact, Minuscule, Micro, Diminutive, Minute, Pint-sized, Teensy, Wee, Short, Bitty, Dinky, Dwarf, Trivial, Negligible, Insignificant, Limited, Modest, Few, Minor, Slight, Minimal, Pale, Soft, Gentle, Easy, Lightweight, Featherweight, Unencumbered, Trivial, Flimsy, Lighthearted, Frivolous, Unserious, Light, Thin, Slight.	Intermediate, Middle, Midsize, Median, Mean, Central, Average, Standard, Norm, Common, Typical, Midrange, Moderate, Regular, Medial, Medium-sized, Halfway, Fair, Balanced, Normal.	Abundance, Plenitude, Profusion, Surplus, Glut, Plethora, Excess, Mass, Plenty, Copiousness, Ample, Large quantity, Overabundance, Bounty, Slew, Swarm, Teeming, Flock, Heap, Loads, Tons, Scores, Flood, Avalanche, Outpouring, Torrent, Cluster, Pack, Throng, Horde, Majority, Hefty, Weighty, Massive, Large, Big, Bulky, Solid, Substantial, Robust, Dense, Voluminous, Severe, Grave, Serious, Profound, Extreme, Intense, Strong, Deep, Significant, Onerous, Harsh, Ponderous, Bright, Titanic, Hulking, Sturdy, Plump, Lead-footed, Broad, Thick, Gross, Hard, Mighty, Stiff, Sizable, Mammoth, Gargantuan, Giant, Herculean, High, Alarming, Enormous.

determine the density extraction.

For example, with the word “heavy” in the “High” column of Table 2, the effect generator should dispense a large amount of water to the audience. It is worth noting that if no density word is identified, then the effect should be considered as medium.

The second element to be identified is the actual effect that should be played. This element cannot be hardcoded because different effect generators support different effects and in different ways. For example, “[Cold Weather]” can be implemented by pointing a fan towards the user or by reducing the room temperature through the air conditioner. To address this issue, producers of the effect generator need to also provide a mapping table that specifies which word corresponds to which effect. As a general guideline, we propose special effects based on the following categories: temperature, locomotion/haptics, visibility, and scents. The effect generator may use multiple effects at the same time to simulate a virtual phenomenon. The mapping should specify the potential words that will appear in the subtitle tags and their corresponding effect. For example, “rain” could be mapped to spray water.

The starting and ending times of the effect will be determined by the start and finish time of the subtitle. However, if the user wishes to separate it, then they need to specify the effect as a separate subtitle.

The proposed approach can also be applied to movies with any subtitles format or subtitles that are embedded into the video as long as the virtual phenomenon is described within the square brackets. This can be achieved by taking frequent screenshots of the movie and then using an Optical Character Recognition (OCR) library to extract text from the image. Then, the code will extract the text within square brackets and follow the process specified earlier.

The following python code uses OpenCV for image recognition and Tesseract OCR (pytesseract) library for its text recognition capabilities in order to extract the potential special effect.

import cv2

import pytesseract

import re

def extract_special_effect(image_path):

# Load the image

movie_image = cv2.imread(image_path)

# Use OCR

pytesseract.pytesseract.tesseract_cmd = r’/usr/bin/tesseract’

subtitle = pytesseract.image_to_string(movie_image)

# Use a regular expression to read the text within the square brackets

bracketed_text = re.findall(‘\［(.*?)\］’, subtitle)

# Return the list of bracketed text

return bracketed_text

The above sample code can be used for extracting the special effects from the movie subtitles, in case the “srt” file is not available. The above implementation simply serves as a proof of concept that extracting the subtitled text directly from films is possible.

4. The Special Effect Player

The special effect player can be in the form of plug-ins or separate applications. In this paper, we provide guidelines for the development of generic special effect players. The key point is that the player should be able to support different hardware capable of playing different effects. To this end, we propose the use of a generic API method with the following signature: “play_effect(effect, density, duration):void”. Here, the token “effect” is the name of the effect to be played. The token “density” can take any of the values low, medium, or high, as explained earlier in Section 3. And the token “duration” refers to the length of time that the playing of the actual effect will last. Figure 2 visualizes a generic procedure for the development of special effect players.

All function and attribute names mentioned in the following text are for explanation reasons only. The application will have three main components. The “srt_reader” reads the subtitles from the “srt” file, and for each subtitle, it calculates the duration by considering the start and the finish times. It then creates an object that will store the start time in an attribute called “start_time: string”, the previously calculated duration in milliseconds in an attribute called “duration: long,” and the text inside square brackets (if any) is an attribute called “effect:string”. If the subtitle does not include code inside the square brackets, then no object should be created. The objects should be stored in a list called “effects”.

The second component, called “extract_effect”, is a function that, by going through all of the elements of the “effects” list, checks for each object whether the value of the attribute “effect” is an actual effect that should be played or stopped. This can be achieved through a mapping function that checks a mapping dictionary provided by the developer of the effect generation (the actual hardware device that will generate the effect). If the text of the attribute “effect” includes a word from the dictionary, e.g., “rain”, and a corresponding density determiner, e.g., “heavy” (as explained in the previous section), the “extract_effect” function overwrites the value of the effect with the identified effect, resulting in a new value for the effect rain, e.g., “rain: high”. If no effect is identified, then the object will be removed from the list.

The last component will include the function that will play the effects. The function with signature “play_effect (effect, density, duration): void” will take the name of the effect, the density, and the duration as a parameter and will ask the effect generator to play the effect via the API of the device.

The main program will mainly include a timer, which during each interval checks whether the current time has reached the starting time of the first object in the list. If so, a method termed “play_effect” is called with the current object properties.

In order to automate the whole process, each of the above parts can be passed as prompts to generative AI tools such as chatGPT4 [16,17,18,19] to generate a template code; however, the hardware developer should provide the mapping dictionary and an API that the “play_effect” method should call.

5. Discussion

Currently, it does not make much sense for film producers to include special effect clues or tags within the subtitles of their films, except possibly for providing more clues to their audiences about the virtual conditions of the digital content. This is because special effect generators are not widely available. On the other hand, one can claim that such devices are not widely available because films do not support them. While this paper presents a futuristic idea, it is possible for next-generation movie players to support special effects, based on an agreed standard. This work showed that this is possible by proposing such an approach which can even work with existing video content, as long as it provides clues within the subtitle tags. Adopting such an approach could motivate firm producers to include tags with more clues in their subtitles. This could then lead to hardware manufacturers producing more ‘effect generators’ at lower costs, which eventually will allow this technology to be adopted by average consumers and enhance the experience of home entertainment systems.

6. Conclusions

This paper presented a comprehensive approach that encompasses various aspects of integrating immersive special effects into digital content. Our approach relies on the idea of enhancing subtitle files through embedding clues about the virtual firm special conditions inside square brackets. We showed how these clues can be converted to special effects such as spraying, scent release, modifying light, temperature manipulation, vibration, and motion, like the ones available in multidimensional cinemas (4D, 7D, etc.). Specifically, we proposed an algorithmic procedure for how to automate the extraction of these special effects from corresponding “srt” subtitle files or even firm images. Our approach also facilitates the conversion of the extracted special effects into a generic format that can be played by hardware defined as special effect generators. This paper also provided guidelines for how special effect players can be developed. The main limitation of this work is that it is still at the conceptual level and a lot of implementation details were not included. For example, it was not specified how to synchronize the movie, especially if the viewer pauses, goes backward or forward, or restarts it. In the future, we aim to implement a holistic system that will include a special modified film, a special effects player, and the special effect generator device and to further test and develop the approaches proposed in this paper.