The term 'multimodal interactions' (MMI) on the HCI field refers to the situation where a system offers many ways of interaction.
This talk combines an academic subject with the daily JS we already know. It is an effort to bring together two, at first, different worlds looking for a win-win situation. On the one hand, academia could enjoy getting a novel approach to a particular problem thanks to our awesome JS. On the other, the industry can get new ways of interaction which can be applied on a variety of context and/or products. In short, the talk contains: - A quick introduction to MMI (Multimodal Interactions) - A particular novel approach to add support for MMI on Web Apps. - Short list of related open problems or possible research lines to pursuit.
17. Getting started
npm install plusultra
Docker-compose up
mkdir myMultimodalDeviceDriver
cd myMultimodalDeviceDriver
npm install gyes
18. Creating a MM Device Driver
var Inherits = require( 'inherits' );
var ModalityChannel = require( 'gyes' ).ModalityDriver;
module.exports = MyModality;
function MyModality( opts ){
// constructor code...
}
Inherits(MyModality, ModalityChannel);
MyModality.prototype.fission = function( time ){ … };
MyModality.prototype.newEvent = function( data ){ … };
19. Inside the app
// lets create a new plusultra client
plusultraClient = new gyes( app_key, socketURI, options );
plusultraClient.authenticate( app_key );
// create a new modality (this is like metadata)
var myMod = new gyes.Modality( modName, 'input', {} );
// instance a driver for the modality
var modDriver = new myMultimodalDeviceDriver( options );
myMod.use( modDriver );
// finally add the modality into the system
plusultraClient.addModality( app_key, airMod );
21. ➔ Experimenting with WebRTC to enable P2P interactions.
➔ Define Interpretations as components. W3C MMI. React can
help?
➔ Improve API. Follow a reactive/functional approach. Cycle/rxjs?
Future Work
