Immersive Sense Room

A client in a highly dynamic industry decided to explore how people react to intensive, immersive AV experiences.
We responded by constructing a powerful AV facility with several dimensions, harnessed by a custom API.

Our Challenge

Research & Development projects are a dream for vow voltage integrators for a few reasons. These sorts of projects generally involve exploring new technologies, or assembling new experiences, and the raw creativity that these efforts draw upon–the most exciting aspects
of working in our field. When our staff get an opportunity to unpack new technologies, determine their strengths and exploit them to their fullest by combining them with other devices, there’s a sense of achievement that’s enjoyed by both us, and our client.

In this case, the challenge was dripping with potential: The client identified an opportunity to explore new dimensions to their products and services by experimenting with external effects on the activities that their business is centred around. We understand branding, UX, and the incredible importance of the customer experience, so when we were approached about this R&D project, we were impressed to see it happening at all, and jumped at the opportunity to help by building an incredible facility.

The project would require:

• An immersive, 3-dimensional video-driven experience in an existing room of 19×19-foot dimensions

• A way to extend a regular HD-aspect video, so that stock footage could be the base content of an immersive experience

• A powerful audio system that would be capable of producing sounds at ‘highly impressionable’ levels, including very deep bass

• A need to pickup sounds in the space as a form of input for interactive response

• Camera coverage of the entire room for video analytic input for interactive response

• A way to incorporate climatic changes as part of the immersive experience

• Control means for the systems both for direct setup, and as a dashboard controls for a running experience

• The integration of heart-rate sensors worn by the occupants

• A way to assemble and create tailored, immersive experiences programmatically

The Solution

We began by examining exactly how to accomplish the primary driver of the experience - the projected video. The 19’ square space had a good ceiling height, and we were faced with hanging all of our in-room systems above the ceiling canopy, in a black painted, hidden assembly. To project video onto an entire 19 × 8½-foot wall surface, we would require a projector with special optics, since a regular projector would sit about 25-50 feet from a screen of that size. A powerful short-throw projector with a unique lens was needed to both fill the wall, and allow for the geometric image adjustment required. With a projector of this size, it became obvious that this installation would require careful coordination of its many ceiling-hung components to be properly interleaved and remain serviceable after installation.

The existing space had HVAC ducting and plumbing running throughout its ceiling, so we carefully dimensioned it all, and began planning the equipment placement amidst it all.

The Various Components 

The worth of an integrated system is truly in the interoperability of it’s various components, and to that end, the loftiest systems involve a plethora of components. In this case we deployed the following:

• 1× high powered short-thow primary laser phosphor projector

• 6× flanking short throw projectors for image extension

• 14× high power coloured LED wall washers to illuminate the flanking walls in colour that’s sampled from the outside regions of the primary video, and passed to a DMX processor to replicate those colours on the walls

• 8× high-power ceiling speakers, and 2 concealed, ported subwoofers

• 1× ceiling hung choir microphone

• 1× 360 degree high resolution camera, with video analytics

• 1× iPad with custom Kutano control application for live experiences

• 1× software API for the client’s IT team to extend functionality programmatically (the true meat of their R&D platform)

What’s an API?

An Application Program interface is a heavy lifter in the world of software development and integrated systems. It’s essentially the interpreter between two populations with differently encoded languages. When a system is developed, and it’s got a software or electronic communications component, that software (or firmware) may need to interact with other software. The folks who will interact with it, will need to know how to do so.

An API can take various forms, but it’s generally a kit including specific preparation within the software platform, and detailed documentation. Preparation done within the software provides the elements allowing inter-communication, available data outputs and inputs, and the code-level interoperability necessary to build around the platform. The documentation component provides information about the software platform and any hardware elements that it works with, an outline about the I/O available to the user inputs and outputs, and instructions about the use of the API software environment. Critically, it spells out the code syntax and format necessary for the systems to talk.

Without API’s there would be effectively no efficient interoperability between disparate systems, involving software.

The Results

Completing this project involved several steps of commissioning individual systems, including this moment when video and LED were first used in combination to fill the room with intense colour and a an organic, warbling shape that undulates in front of the occupant.

Each test of this highly-energetic AV platform proved extremely impressionable, making for one of the best completion sequences of any project we’ve taken on. With such a unique and powerful system, every dimension can be taken to an almost off-putting level. Incredibly loud or deep, rumbling sound, combined with intense colour, and the sheer saturation of the tone itself makes the occupant feel almost helpless to prevent its absorption into their very being, which was exactly what was called for in this experimental facility.

A custom program that our developers created samples the average colour from the left and right regions of the video being projected, then reproduces those colours the sides of the room in real time. The effect is to envelop the occupant within an environment that’s driven by the head-to-toe video they stand before. The effectiveness of this unique approach is undeniable, making for an enveloping video experience, where one is entirely surrounded and bombarded by qualia that work in concert to immerse the occupant in an alternate space.

Control and Development

Once the components are commissioned and ready for use, the client must be provided a way to utilize them all, in a few different manners.

First, the client must be provided direct control to set the room up manually to do anything that they want to, within the capabilities of the individual components. The LED wall washers, for example, may be used on their own, or perhaps with music played through the audio system to create an emotive environment. 

In order to provide direct controls, we adapted our proprietary control application, used generally to control our installed systems of any sort (such as an automated video conferencing ad meeting room). Due to the extensive unique control needs in this space, we decided to create an offshoot, dedicated purpose application, called Kipaji (which means “magic” in the Swahili tongue). Developing a control application is a matter of breaking down all the relevant functionality, determining the most logical and intuitive approach to it, then storyboarding the pathways from screen to screen in the graphic user interface that the user will interact with.

Controlling an Experience

Unlike directly enacting individual components, the true magic of this facility is it’s operation in concert. To this end, we worked with the client’s team to create and setup several scripted sequences and functions to allow them to begin using the facility. Each of these sequences involves a combination of components used in varying ways. An example of a preset function is a ‘Themed Environment’, whereby the lights and screen create a visual environment, such as an Arctic storm, and the audio system plays its sound track. The air temperature is dropped to it’s coldest setting, and outside observers can control volume, light level, and use a PA system to communicate with the occupants of the space. Kipaji, running on an iPad outside the room, is used as the control platform to both select this function and control it once under way.

Extending the Experience

The long term purpose of the space surrounds researching various effects on occupants while they perform physical activities in the room. In order for the client to have access to develop further experiment, and control their experiments in endless ways, the system must be extensible. That is, it’s raw functionality must be exposed for programming additional experiences and sequences, and even to introduce additional inputs that the occupants might wear while performing their activities. An example is a heart rate monitor that the client decided would help gauge the effect of various immersive experiences. In order to introduce a new device, the API allows the client’s programmers to add entry points fir the data that the monitor exports. The client then adapts the data to confirm to the programmatic language described in our API, and the device can be invoked. How that added device is controlled is a matter of how they choose to program their own custom sequences, using their own software.

Another example of ho the API documentation is so essential to the client’s development is it’s listing of the identities, and addresses of each networked device. Because the components of the system are all networked devices, the addresses of each allows the client to find and connect to them. The concert of control that ensues from the creation of an effective API is the backbone of this project - the initiating intent, and the vehicle that drives it to eventual successes for our client.

Many hours were spent working out and perfecting the approach, logic, programming and composition of the API. The process begins with gathering a detailed understanding of the clients desires and intentions for their own use of the facility. This includes exploring the software of their choosing to program upon the platform to sequence their own integrated experiences, as well as extending added functionality beyond that provided at hand over.

The overlap between our team, and the client’s is a key aspect of collaboration that makes an entirely unique system like this one highly valuable and effective. 

Thanks to Our Partners

This highly dynamic project would not have been possible without the detailed coordination and contributions from the following partners:

• Our client (with respect for their privacy)

• NEC Global

• Tannoy

• James Loudspeakers

• Blizzard Lighting