Empathic Extended Reality

•

0 j'aime•206 vues

Joaquim Jorge

Empathic Extended Reality Exploring Physiological Data Sharing in Virtual Environments by Arindam Dey

Ingénierie

Empathic Extended Reality:
Exploring Physiological Data
Sharing in Virtual Environments
ARINDAM DEY
EMPATHIC EXTENDED REALITY AND PERVASIVE COMPUTING LAB
UNIVERSITY OF QUEENSLAND
A.DEY@UQ.EDU.AU

https://www.youtube.com/watch?v=t3g2IBSaYb4

Empathy
“Seeing with the Eyes of another,
Listening with the Ears of another,
and Feeling with the Heart of another …”
Alfred Adler

Empathic Computing
Understanding
Systems that can understand your feeling and emotions
Experiencing
Systems that can help you better experience the world
Sharing
Systems that help you better share your experiences with others
Sensors
Extended
Reality

“Virtual Reality is the ultimate empathy
machine …”
- Chris Milk

https://www.youtube.com/watch?v=digE62wpHOk
Project Syria – Nonny de la Peña

Empathic XR
Extended Reality Systems that can
Measure | Share | Adapt to | Manipulate
Emotion and Cognition in real time
-Physiological (ECG, EDA, EMG, Pupil)
-Neurological (EEG)
-Behavioural (Speech, Posture, Movement)

Sharing Physiological States in Collaborative VR
ACM CHI’17

Data Collected
- Raw heart-rate
- Positive and negative affect schedule (PANAS)
- Relative head orientation
Results
- Higher positive affect
- More communication between collaborators
- Observer interaction needed
Sharing Physiological States in Collaborative VR
ACM CHI’17

Multi-Sensory Heart Rate Feedback
Results
- Audio-Haptic ranked best
- More interaction needed for higher presence
Data Collected
- PANAS
- Ranking
- Raw heart-rate
OzCHI’18

Manipulating Heart Rate Feedback in VR
ACM CHI Play’18
PANAS

Multi-Sensory Heart Rate Feedback in Collaborative VR
IEEE ISMAR’18

Multi-Sensory Heart Rate Feedback in Collaborative VR
IEEE ISMAR’18
PANAS
Social
Presence

Shared and Manipulated Heartrate Feedback in VR
IEEE ISMAR’19
Active
Passive
Manipulation -20%, 0%, +20%

Shared and Manipulated Heartrate Feedback in VR
Social
Presence
IEEE ISMAR’19

Lessons Learned
Social presences increases with
- Shared awareness cues
- Higher interaction
Shared physiological feedback can
- Increase communication
- Alter emotions but not real signals
- Increase awareness but may not change action

Emotion Sensing HMDs
Looxid Emteq Neurable
HMDs equipped with neurological and physiological sensors

Cognitively Adaptive Training in VR
Looxid
Training difficulty level adjusted based on alpha activity in real time
Low level High level
IEEE VR’19

Cognitively Adaptive Training in VR
Looxid
IEEE VR’19
Alpha activity changed without significant change in task completion time

Quantifying Presence
Looxid
Self-reported Questionnaires vs. real-time Physiological and Neurological Data

Quantifying Presence with Brain Signals
Looxid
High Presence
Low Presence

Next steps: Using EEG and other physiological sensors
Looxid
• Emotionally and cognitively adapt interfaces for learning and training
• Understand how brain synchronization works in collaborative XR
• Effectively share emotional and cognitive states to create empathy in
collaborative XR

Thank You!
Looxid
Arindam Dey
Empathic Extended Reality and Pervasive Computing Lab
Co-Innovation Group | University of Queensland
http://empathicxr.org | a.dey@uq.edu.au

Contenu connexe

Similaire à Empathic Extended Reality

Research Summary - Empathic Extended Reality

Dr. Arindam Dey

Emotion recognition from biometrics is relevant to a wide range of application domains, including healthcare and software development. Existing approaches usually adopt multi-electrodes sensors that could be expensive or uncomfortable to be used in real-life situations. We investigate whether we can reliably recognize high vs. low emotional valence and arousal by relying on noninvasive low-cost EEG, EMG, and GSR sensors. We report the results of an empirical study involving 19 subjects in a laboratory setting for emotion elicitation. We achieve state-of-the-art classification performance for both valence and arousal even in a cross-subject classification setting, which eliminates the need for individual training and tuning of classification models. Furthermore, we will discuss our ongoing work on the recognition of affective and cognitive states of software engineer during their daily programming tasks.

Emotion Detection Using Noninvasive Low-cost Sensors

Nicole Novielli

Linking brain systems and mental functions requires accurate descriptions of behavioral tasks and fine demarcations of brain regions. Functional Magnetic Resonance Imaging (fMRI) has contributed to the investigation of brain regions involved in a variety of cognitive processes. However, to date, no data collection has systematically addressed the functional mapping of cognitive mechanisms at a fine spatial scale. The Individual Brain Charting (IBC) project stands for a high-resolution multi-task fMRI dataset that intends to provide the objective basis toward a comprehensive functional atlas of the human brain. The data refer to a permanent cohort performing many different tasks. The large amount of task-fMRI data on the same subjects yields a precise mapping of the underlying functions, free from both inter-subject and inter-site variability. The first release of the IBC dataset consists of data acquired from thirteen participants during performance of a dozen of tasks. Raw data from this release are publicly available in the OpenNeuro repository and derived statistical maps can be found in NeuroVault. These maps reveal a successful cognitive encoding of many psychological domains in large areas of the human brain. Indeed, main findings of the original studies were replicated at higher resolution. Our results thus provide a comprehensive revision of the neural correlates underlying behavior, highlighting nonetheless the spatial variability of functional signatures between participants. In addition, this dataset supports investigations using alternative approaches to group-level analysis of task-specific studies. For instance, such rich task-wise dataset can be applied to mega-analytic encoding models towards the development of a brain-atlasing framework, by systematically mapping functional signatures associated with the cognitive components of the tasks.

Individual Brain Charting, a high-resolution fMRI dataset for cognitive mappi...

Ana Luísa Pinho

Serious Games and digital interactive technologies for Healthcare

imaginary srl

Pannese healthcare

Lucia Pannese

NeuroVR 1.5 - Open Source VR system for neuroscience and behavioral healthcare

Riva Giuseppe

Software development is an intellectual activity requiring creativity and problem-solving skills, which are known to be influenced by emotions. Developers experience a wide range of affective states during programming tasks, which may have an impact on their job performance and wellbeing. Early recognition of negative emotions, such as stress or frustration can enable just-in-time intervention for developers and team managers, in order to prevent burnout and undesired turnover. In this talk, I will provide an overview of recent research findings of sentiment analysis in software engineering (SE), address the open challenges, and provide empirically-based guidelines for safe (re)use of SE-specific tools in order to obtain meaningful results.

Keynote@QUATIC - Recognizing Developer's Emotions: Advances and Open Challenges

Nicole Novielli

Abstract—The development of Microsoft Kinect opened up the research field of computational emotions to a wide range of applications, such as learning environments, which are excellent candidates to trial computational emotions based algorithms but were never feasible for given consumer technologies. Whilst Kinect is accessible and affordable technology it comes with its’ own additional challenges such as the limited number of extracted Action Units (AUs). This paper presents a new approach that attempts at finding patterns of interaction between AUs and each other on one hand and patterns that link the related AUs to a given emotion. In doing so, this paper presents the ground work necessary to reach a model for dynamically generating personified set of rules relating AUs and emotions implicitly encoding a person individuality in expressing emotions. Index Terms – computational psychoanalysis, emotion modelling, user-centred emotion detection, sentiment analysis, Kinect, personified adaptive interfaces

Cognitive Reasoning and Inferences through Psychologically based Personalised...

Aladdin Ayesh

Hpai class 13 - perception ii - 033020

melendez321

GSR_PhobiAR_04

Daniel Podgornik Abramovici

Experimental psychology, cognitive science or, more recently, cognitive neuroscience, is the main framework to place human information processing under extensive empirical scrutiny. The last decade has seen a surge of interest in the application of psychological measurements for evaluating increasingly complex human-technology interactions. While most welcome from the psychological perspective, we propose that the use of these methodologies should not rely only on the application of sophisticated measurement tools, but also on the application of contemporary knowledge on psychological phenomena and dynamics of human information processing. In addition, we argue that the latest developments in multimodal signals and data mining techniques offer a unique opportunity to extend psychological methodologies to large scale testing grounds. Thus, the application of psychological knowledge to information retrieval research will not only be beneficial for the latter, but for the former as well, inasmuch as information retrieval provides a real field of application for its hypotheses about human information processing.

SIGIR15-NeuroIR

Telefonica Research

The importance of sensor networks and ai in supporting smart grid in the indu...

yohan2012

Empathy amplifier presentation ms4

Luis Santiago

Reality As A Knowledge Medium

fridolin.wild

Is the rising technology a threat?

Fabrizio Gramuglio

Walter Greenleaf's presentation to the Virtual Medicine 2019 Conference at Cedars-Sinai Medical Center Immersive Environments, Machine Learning, Neuroimaging, And Wearable Sensing Technology - Treating Depression, Addictions, and Facilitating Behavior Change Using A Precision Medicine Model Based on the methods used in the ENGAGE Study Walter Greenleaf, PhD Virtual Human Interaction Lab | Stanford University Precision medicine models for treating depression, managing addictions and achieving sustained behavior change are largely outside of current clinical practice. Yet, changing self-regulatory behavior is fundamental to the self-management of complex lifestyle-related chronic conditions such as depression and substance use disorder - two top contributors to the global burden of disease and disability. To optimize treatments and address these burdens, methods to facilitate behavior change and self-regulation must be better understood in relation to their neurobiological underpinnings. Treatment strategies can then be developed that leverage the recent advances in immersive environments, machine learning, and wearable sensing technology and apply them to treat depression, manage addictions, and facilitate behavior change using a precision medicine model that is personalized to the individual. This presentation will review the conceptual framework and protocol for a large multi-subject longitudinal study named Project ENGAGE. The ENGAGE study integrates neuroscience with behavioral science to better understand the self-regulation related mechanisms of behavior change for improving mood and weight outcomes among adults with comorbid depression and obesity. We collect assays of three self-regulation targets (emotion, cognition, and self-reflection) in multiple settings: neuroimaging and behavioral lab-based measures, virtual reality, and passive smartphone sampling. By connecting human neuroscience and behavioral science in this manner within the ENGAGE study, we can develop a prototype for elucidating the underlying self-regulation mechanisms of behavior change outcomes and their application in optimizing intervention strategies for multiple chronic diseases. https://www.ncbi.nlm.nih.gov/pubmed/29074231 Behav Res Ther. 2018 Feb;101:58-70. doi: 10.1016/j.brat.2017.09.012. Epub 2017 Oct 7. The ENGAGE study: Integrating neuroimaging, virtual reality and smartphone sensing to understand self-regulation for managing depression and obesity in a precision medicine model. Leanne M. Williams, Adam Pines, Andrea N. Goldstein-Piekarski, Lisa G. Rosas, Monica Kullar, Matthew D. Sacchet, Olivier Gevaert, Jeremy Bailenson, Philip W. Lavori, Paul Dagum, Brian Wandell, Carlos Correa, Walter Greenleaf, Trisha Suppes, L. Michael Perry, Joshua M. Smyth, Megan A. Lewis, Elizabeth M. Venditti, Mark Snowden, Janine M. Simmons, Jun Ma

Immersive Environments, Machine Learning, Neuroimaging, & Wearable Sensing Te...

Stanford University

Virtual Reality Technology, Machine Learning, Biosensing Converging to Transform Healthcare By Walter Greenleaf The oncoming wave of Virtual Reality and Augmented Reality technology will impact medicine, clinical care, and personal health and wellness. Although entertainment, social connection, and gaming will drive the initial adoption of VR and AR technology, the deepest and most significant impact of the next generation of VR/AR technology will be to enhance clinical care and to improve personal health and wellness. VR and AR technology will also help facilitate the shift of medicine to direct personal care. We know from decades of clinical research that VR/AR technology can provide breakthrough solutions that address the most difficult problems in healthcare - ranging from mood disorders such as Anxiety and Depression to PTSD, Addictions, Autism, Cognitive Aging, Stroke Recovery, and Physical Rehabilitation, to name just a few. VR technology can also improve clinical measurements and assessments by making them more objective and functional, and improve medical training such as surgical skill training and procedure planning by applying simulation-based learning principles. Personal health and wellness will be improved by using VR to promote healthy lifestyles and to reduce stress and anxiety. As the cost of healthcare rises, VR technology can serve as an effective telemedicine platform to reduce costs of care delivery, and improve clinical efficiency.

Virtual Reality Technology, Machine Learning, Biosensing Converging to Transf...

Stanford University

Functional specialization in human cognition: a large-scale neuroimaging init...

Ana Luísa Pinho

Citizen Science on the Move conference 25, 26 & 27 june 2012

Ronald Lenz

Human perceptual systems—for sight, sound, taste, touch, and smell (and maybe even embodied proprioception)—may offer some guidelines for how to build multimedia e-learning: immersive 3D simulations, imagery for analysis, sight-and-sound distributions of information channels, and other applications. This will offer a brief overview of human perception (with a little human cognition thrown in) and some light applications to the design of e-learning.

Building the Knowledge of Human Perception into E-Learning

Shalin Hai-Jew

Similaire à Empathic Extended Reality (20)

Research Summary - Empathic Extended Reality

Emotion Detection Using Noninvasive Low-cost Sensors

Individual Brain Charting, a high-resolution fMRI dataset for cognitive mappi...

Serious Games and digital interactive technologies for Healthcare

Pannese healthcare

NeuroVR 1.5 - Open Source VR system for neuroscience and behavioral healthcare

Keynote@QUATIC - Recognizing Developer's Emotions: Advances and Open Challenges

Cognitive Reasoning and Inferences through Psychologically based Personalised...

Hpai class 13 - perception ii - 033020

GSR_PhobiAR_04

SIGIR15-NeuroIR

The importance of sensor networks and ai in supporting smart grid in the indu...

Empathy amplifier presentation ms4

Reality As A Knowledge Medium

Is the rising technology a threat?

Immersive Environments, Machine Learning, Neuroimaging, & Wearable Sensing Te...

Virtual Reality Technology, Machine Learning, Biosensing Converging to Transf...

Functional specialization in human cognition: a large-scale neuroimaging init...

Citizen Science on the Move conference 25, 26 & 27 june 2012

Building the Knowledge of Human Perception into E-Learning

Plus de Joaquim Jorge

Presentation Title: AI & Labor Markets Presenter: Joaquim Jorge Description: Explore the transformative impact of Artificial Intelligence (AI) on labor markets in this comprehensive presentation by Joaquim Jorge. This insightful slideset delves into the opportunities and challenges that AI integration brings to various industries, highlighting key AI techniques and their real-world applications. Bias in Hiring and Firing: The presentation critically examines biases in AI systems used for hiring and firing decisions: Hiring Bias: Instances where AI systems, like LinkedIn’s recommendation system and OpenAI's GPT, have shown biases in résumé ranking and job advertisements, including gender bias and cost-efficiency algorithms inadvertently favoring male candidates. Firing Bias: AI's role in monitoring productivity and making termination decisions, with examples from Amazon’s “Time off Task” system and Uber’s driver performance metrics, highlighting unfair terminations affecting minority groups. Mitigation Strategies: Bias Audits: Regularly auditing AI systems to identify and mitigate biases. Diverse Training Data: Ensuring training data are diverse and representative of all demographic groups. Human Oversight: Implementing human oversight to review and validate AI decisions. Explainable AI (XAI): Making AI decisions transparent and accountable to detect and correct biases. Future of Labor Markets: The presentation explores potential futures of labor markets with AI, presenting both utopian and dystopian scenarios: Utopian Scenario: AI could lead to increased worker satisfaction by automating repetitive tasks, creating new career opportunities, and reducing physical labor demands, resulting in better work-life balance and economic opportunities. Dystopian Scenario: AI could widen the economic divide, increase job precarity, and erode worker rights. Risks include increased surveillance, loss of autonomy, and the social and psychological impacts of job displacement. Key Takeaways: Understand the role and impact of different AI technologies in various sectors. Recognize and address biases in AI systems, especially in hiring and firing decisions. Explore potential futures of labor markets with AI integration. Learn strategies for ensuring ethical and fair AI applications. This presentation is essential for professionals, researchers, and policymakers interested in the intersection of AI and labor markets, providing a detailed analysis of current trends, challenges, and future possibilities.

AI+Labor Markets Presentation to CSM-16-may-2024

Joaquim Jorge

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

Recent advances in VR and AR technology have enabled interactive graphics applications to support healthcare professionals in training, diagnosis, planning, and treatment. This field has progressed enough to warrant a course that can inspire new ideas within the graphics community. Medical images create virtual human anatomy models, allowing for natural interaction and visualization in healthcare scenarios. VR and AR are conceptually different and suited for different types of problems. VR is immersive and suitable for learning anatomy, surgical skills, and analyzing 3D medical data. AR, on the other hand, overlays helpful information onto the physical environment, making it useful for tasks such as communication with patients and training assistants. However, several challenges, such as nonstandard equipment and disorientation, still limit the widespread use of these technologies. This course covers current advances and challenges in this area, including integrating AI techniques. About the Speaker: Joaquim Jorge holds the UNESCO Chair of Artificial Intelligence & Extended Reality at the University of Lisboa, Portugal. He joined Eurographics in 1986 and ACM/SIGGRAPH in 1989. He is Editor-in-Chief of the Computers and Graphics Journal, Eurographics Fellow, ACM Distinguished Member, and member of IEEE Computer Society Board of Governors. He organized 50+ conferences, including Eurographics 2016 (IPC CO-Chair), IEEE VR 2020/21/22 as co-(papers)chair, and ACM IUI 2012 (IPC co-chair). He served on 210+ program committees and (co)authored over 360 peer-reviewed publications and five books. His research interests include graphics, virtual reality, and advanced HCI techniques applied to health technologies. Websites: https://en.wikipedia.org/wiki/Joaquim_Jorge_(computer_scientist) Google Scholar: https://scholar.google.com/citations?user=RgiMdpAAAAAJ&hl=en D.S. Lopes, D. Medeiros, S.F. Paulo, P.B. Borges, V. Nunes, V. Mascarenhas, M. Veiga, J.A. Jorge, Interaction Techniques for Immersive CT Colonography: A Professional Assessment, In: Frangi A., Schnabel J., Davatzikos C., Alberola-López C., Fichtinger G. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. MICCAI 2018. Lecture Notes in Computer Science, vol 11071, Pages 629–637, Springer, Cham, 2018. DOI: 10.1007/978-3-030-00934-2_70 M. Sousa, D. Mendes, S. Paulo, N. Matela, J. Jorge, D.S. Lopes, VRRRRoom: Virtual Reality for radiologists in the reading room, Proceedings of the 35th Annual ACM Conference on Human Factors in Computing Systems (CHI 2017), New York: ACM Press, 2017. DOI: 10.1145/3025453.3025566 D.S. Lopes, P.F. Parreira, S.F. Paulo, V. Nunes, P.A. Rego, M.C. Neves, P.S. Rodrigues, J.A. Jorge, On the utility of 3D hand cursors to explore medical volume datasets with a touchless interface, Journal of Biomedical Informatics, 72, Pages 140–149, 2017. DOI: 10.1016/j.jbi.2017.07.009

Matera Summer School AI&XR 2023.pdf

Joaquim Jorge

Invited paper from the Computers & Graphics Journal Presented at the SMI 2023 Conference in Genoa, Italy on July 4th 2023. Abstract: Highly complex and dense models of 3D objects have recently become indispensable in digital industries. Mesh decimation then plays a crucial role in the production pipeline to efficiently get visually convincing yet compact expressions of complex meshes. However, the current pipeline typically does not allow artists control the decimation process, just a simplification rate. Thus a preferred approach in production settings splits the process into a first pass of saliency detection highlighting areas of greater detail, and allowing artists to iterate until satisfied before simplifying the model. We propose a novel, efficient multi-scale method to compute mesh saliency at coarse and finer scales, based on fast mesh entropy of local surface measurements. Unlike previous approaches, we ensure a robust and straightforward calculation of mesh saliency even for densely tessellated models with millions of polygons. Moreover, we introduce a new adaptive subsampling and interpolation algorithm for saliency estimation. Our implementation achieves speedups of up to three orders of magnitude over prior approaches. Experimental results showcase its resilience to problem scenarios that efficiently scales up to process multi-million vertex meshes. Our evaluation with artists in the entertainment industry also demonstrates its applicability to real use-case scenarios. Authors: Rafael Kuffner dos Anjos, Leeds, UK Richard Andrew Roberts and Benjamin Allen, VUW, NZ, Joaquim Jorge, INESC-ID, U Lisboa, PT Ken Anjyo, OLM, JP

Presentation at SMI 2023

Joaquim Jorge

Public speaking is often referred to as a soft skill. However, your career, your job opportunities, promotion, and tenure (for professors) are intrinsically tied to being able to speak in public and persuade others of your ideas. Being able to "sell" your work, opinions, ideas, and points of view can be as important as being good at Math, Coding, and doing good science. Many tend to ignore this (at their peril). This talk will present ways to make your ideas, thoughts, and theses look appealing, engaging and engaging to others. While I am using PowerPoint, these practices and techniques will work equally well if you have PDF slides, Keynote presentations, or even (gasp!) Prezi... I will also cover slide preparation techniques, practical multimedia advice, and delivery how-tos (and not-tos). Bio: Joaquim Jorge heads the Graphics and Interaction Research Line at INESC-ID and is a Full Professor of Computer Science at Instituto Superior Técnico (IST), the School of Engineering of the University of Lisboa, Portugal, and is Editor-in-Chief of the Computers and Graphics Journal. He received Ph.D. and MSc degrees in Computer Science from Rensselaer Polytechnic Institute, Troy, NY, in 1995 and a BSEE from IST in 1984. An organizer of 50+ scientific conferences, Jorge has preferred over 42 invited talks and public presentations. He is a Distinguished Speaker and Member of the Association for Computing Machinery (ACM) and a Distinguished Visitor of the Institute of Electrical and Electronics Engineers (IEEE). He received the IFIP Silver Core in 2014 and is a Fellow of the Eurographics Association.

How to Craft and Deliver Winning Presentations

Joaquim Jorge

Virtual and Augmented Reality in Health Care & Rehabilitation

Joaquim Jorge

The growing interest of Augmented Reality (AR) together with the renaissance of Virtual Reality (VR) has opened new approaches and techniques on how professionals interact with medical imagery, plan, train and perform surgeries and also help people with special needs in Rehabilitation tasks. Indeed, many medical specialties already rely on 2D and 3D image data for diagnosis, surgical planning, surgical navigation, medical education or patient-clinician communication. However, the vast majority of current medical interfaces and interaction techniques continue unchanged, while the most innovative solutions have not unleashed the full potential of VR and AR. This is probably because extending conventional workstations to accommodate VR and AR interaction paradigms is not free of challenges. Notably, VR and AR-based workstations, besides having to render complex anatomical data in interactive frame rates, must promote proper anatomical insight, boost visual memory through seamless visual collaboration between professionals, free interaction from being seated at a desk (e.g., using mouse and keyboard) to adopt non-stationary postures and freely walk within a workspace, and must also support a fluid exchange of image data and 3D models as this fosters interesting discussions to solve clinical cases. Moreover, VR and AR-based techniques must also be designed according to good human-computer interaction principles since it is well known that medical professionals can be resistant to changes in their workflow. In this course, we will survey recent approaches to healthcare, including diagnosis, surgical training, planning, and followup as well as AR/MR/VR tools for patient rehabilitation. We discuss challenges, techniques, and principles in applying Extended Reality in these contexts and outline opportunities for future research. References: https://dl.acm.org/citation.cfm?id=3359418 This course was also presented in Toronto in two additional talks.

Siggraph asia 2019 Course slides

Joaquim Jorge

Siggraph asia 2019 panel pesentation ian hall

Joaquim Jorge

Morie frontiers nov2019

Joaquim Jorge

Embodiment cognition

Joaquim Jorge

Debevec vrai-s-asia2019

Joaquim Jorge

Glassner Puppies and Dragons vr+ai workshop

Joaquim Jorge

Siggraph Asia 2019 VR & AI workshop-wiles-handout

Joaquim Jorge

Augmented Reality and Artificial Intelligence: Friends with Benefits?

Joaquim Jorge

Empathic Computing

Joaquim Jorge

Augmented Reality and Artificial Intelligence: Friends with Benefits?

Joaquim Jorge

3D reconstruction from anatomical slices allows anatomists to reconstruct real structures by tracing organs from a lengthy series of cryosections. Notwithstanding, conventional interfaces rely on isolated single-user experiences using mouse-based input for tracing. In this work, we present Anatomy Studio, a collaborative mixed-reality approach, combined with tablets and styli, to assist anatomists by easing manual image segmentation and exploration tasks. We contribute novel interaction techniques intended to promote spatial understanding and expedite manual segmentation. By using mid-air interactions and interactive surfaces, anatomists can easily access any cryosection and edit contours, while following other user's contributions. A user study including experienced anatomists and medical professionals, conducted in real working sessions, demonstrates that Anatomy Studio is appropriate and useful for 3D reconstruction. Results indicate that our approach encourages closely-coupled collaborations and group discussion. We also discuss the implications of our work and provide domain insights.

Anatomy Studio: a Tool for Virtual Dissection Through Augmented 3D Reconstruc...

Joaquim Jorge

top 10 ways to get your paper rejected at Computers and Graphics Journal

Joaquim Jorge

VROOOM: Reading room conditions such as illumination, ambient light, human factors, and display luminance, play an important role in how radiologists analyze and interpret images. Indeed, serious diagnostic errors can appear when studying images through everyday monitors, namely whenever professionals are ill-positioned with respect to the display or visualize images under improper light and luminance conditions. In this work, we show that virtual reality head-mounted displays can assist radiodiagnostics by considerably diminishing the effects of unsuitable ambient conditions. Our approach combines virtual reality displays with interactive surfaces to support professional radiologists in analyzing medical images and formulating diagnostics. We evaluated our prototype with senior radiologists and radiology residents, and results indicate that our approach constitutes a viable, flexible, portable and cost-efficient option to traditional radiology reading rooms. ICOLONIC: CT Colonography (CTC) is considered the leading imaging technique for colorectal cancer (CRC) screening. However, conventional CTC systems rely on clumsy 2D input devices and stationary displays that make it hard to perceive the colon structure in 3D. To visualize such anatomically complex data, the immersion and freedom of movement afforded by Virtual Reality (VR) systems bear the promise to assist clinicians to improve 3D reading, hence, enabling more expedite diagnoses. To this end, we propose iCOLONIC, a set of interaction techniques using VR to perform CTC reading. iCOLONIC combines immersive Fly-Through navigation with positional tracking, multi-scale representations and mini-maps to guide radiologists and surgeons while navigating throughout the colon. Contrary to stationary VR solutions, iCOLONIC allows users to freely walk within a workspace to analyze both local and global 3D features.

Virtual Reality for Health Applications - siggraph asia 2018

Joaquim Jorge

ACM Europe October 2016

Joaquim Jorge

Plus de Joaquim Jorge (20)

AI+Labor Markets Presentation to CSM-16-may-2024

Artificial Intelligence: Facts and Myths

Matera Summer School AI&XR 2023.pdf

Presentation at SMI 2023

How to Craft and Deliver Winning Presentations

Virtual and Augmented Reality in Health Care & Rehabilitation

Siggraph asia 2019 Course slides

Siggraph asia 2019 panel pesentation ian hall

Morie frontiers nov2019

Embodiment cognition

Debevec vrai-s-asia2019

Glassner Puppies and Dragons vr+ai workshop

Siggraph Asia 2019 VR & AI workshop-wiles-handout

Augmented Reality and Artificial Intelligence: Friends with Benefits?

Empathic Computing

Augmented Reality and Artificial Intelligence: Friends with Benefits?

Anatomy Studio: a Tool for Virtual Dissection Through Augmented 3D Reconstruc...

top 10 ways to get your paper rejected at Computers and Graphics Journal

Virtual Reality for Health Applications - siggraph asia 2018

ACM Europe October 2016

Dernier

Introduction-to- Metrology and Quality.pptx

ProfASKolap

Circuit Breakers for Engineering Students

kannan348865

1_Introduction + EAM Vocabulary + how to navigate in EAM.pdf

AldoGarca30

8th International Conference on Soft Computing, Mathematics and Control (SMC 2024) will provide an excellent international forum for sharing knowledge and results in theory, methodology and applications impacts and challenges of Soft Computing, Mathematics and Control. The conference documents practical and theoretical results which make a fundamental contribution for the development of Soft Computing, Mathematics and Control. The aim of the conference is to provide a platform to the researchers and practitioners from both academia as well as industry to meet and share cutting-edge development in the field.

8th International Conference on Soft Computing, Mathematics and Control (SMC ...

josephjonse

Databricks Generative AI Fundamentals .pdf

VinayVadlagattu

Autodesk Construction Cloud (Autodesk Build).pptx

Mustafa Ahmed

Theory of Time 2024 (Universal Theory for Everything)

Ramkumar k

21P35A0312 Internship eccccccReport.docx

rahulmanepalli02

Danikor Product Catalog- Screw Feeder.pdf

thietkevietthinh

handbook on reinforce concrete and detailing

AshishSingh1301

Overview of Transformation in Computer Graphics

ChandrakantDivate1

Scouring of cotton and wool fabric with effective scouring method

vimal412355

History of Indian Railways - the story of Growth & Modernization

Emaan Sharma

Computer Graphics - Windowing and Clipping

ChandrakantDivate1

Computer Graphics Introduction To Curves

ChandrakantDivate1

Fundamentals of Internet of Things (IoT) Part-2

ChandrakantDivate1

Involute of a circle,Square, pentagon,HexagonInvolute_Engineering Drawing.pdf

JNTUA

Dr Mrs A A Miraje C Programming PPT.pptx

ProfAAMiraje

Dynamo Scripts for Task IDs and Space Naming.pptx

Mustafa Ahmed

Path loss model, OKUMURA Model, Hata Model

DrAjayKumarYadav4

Dernier (20)

Introduction-to- Metrology and Quality.pptx

Circuit Breakers for Engineering Students

1_Introduction + EAM Vocabulary + how to navigate in EAM.pdf

8th International Conference on Soft Computing, Mathematics and Control (SMC ...

Databricks Generative AI Fundamentals .pdf

Autodesk Construction Cloud (Autodesk Build).pptx

Theory of Time 2024 (Universal Theory for Everything)

21P35A0312 Internship eccccccReport.docx

Danikor Product Catalog- Screw Feeder.pdf

handbook on reinforce concrete and detailing

Overview of Transformation in Computer Graphics

Scouring of cotton and wool fabric with effective scouring method

History of Indian Railways - the story of Growth & Modernization

Computer Graphics - Windowing and Clipping

Computer Graphics Introduction To Curves

Fundamentals of Internet of Things (IoT) Part-2

Involute of a circle,Square, pentagon,HexagonInvolute_Engineering Drawing.pdf

Dr Mrs A A Miraje C Programming PPT.pptx

Dynamo Scripts for Task IDs and Space Naming.pptx

Path loss model, OKUMURA Model, Hata Model

Empathic Extended Reality

1. Empathic Extended Reality: Exploring Physiological Data Sharing in Virtual Environments ARINDAM DEY EMPATHIC EXTENDED REALITY AND PERVASIVE COMPUTING LAB UNIVERSITY OF QUEENSLAND A.DEY@UQ.EDU.AU

3. https://www.youtube.com/watch?v=t3g2IBSaYb4

6. Empathy “Seeing with the Eyes of another, Listening with the Ears of another, and Feeling with the Heart of another …” Alfred Adler

7. Empathic Computing Understanding Systems that can understand your feeling and emotions Experiencing Systems that can help you better experience the world Sharing Systems that help you better share your experiences with others Sensors Extended Reality

8. “Virtual Reality is the ultimate empathy machine …” - Chris Milk

9. https://www.youtube.com/watch?v=digE62wpHOk Project Syria – Nonny de la Peña

10. Empathic XR Extended Reality Systems that can Measure | Share | Adapt to | Manipulate Emotion and Cognition in real time -Physiological (ECG, EDA, EMG, Pupil) -Neurological (EEG) -Behavioural (Speech, Posture, Movement)

11. Sharing Physiological States in Collaborative VR ACM CHI’17

12.

13. Data Collected - Raw heart-rate - Positive and negative affect schedule (PANAS) - Relative head orientation Results - Higher positive affect - More communication between collaborators - Observer interaction needed Sharing Physiological States in Collaborative VR ACM CHI’17

14. Multi-Sensory Heart Rate Feedback

15. Multi-Sensory Heart Rate Feedback Results - Audio-Haptic ranked best - More interaction needed for higher presence Data Collected - PANAS - Ranking - Raw heart-rate OzCHI’18

16. Manipulating Heart Rate Feedback in VR ACM CHI Play’18 PANAS

17. Multi-Sensory Heart Rate Feedback in Collaborative VR IEEE ISMAR’18

18. Multi-Sensory Heart Rate Feedback in Collaborative VR IEEE ISMAR’18 PANAS Social Presence

19. Shared and Manipulated Heartrate Feedback in VR IEEE ISMAR’19 Active Passive Manipulation -20%, 0%, +20%

20. Shared and Manipulated Heartrate Feedback in VR Social Presence IEEE ISMAR’19

21. Lessons Learned Social presences increases with - Shared awareness cues - Higher interaction Shared physiological feedback can - Increase communication - Alter emotions but not real signals - Increase awareness but may not change action

22. Brain Signals and VR

23. Emotion Sensing HMDs Looxid Emteq Neurable HMDs equipped with neurological and physiological sensors

24. Cognitively Adaptive Training in VR Looxid Training difficulty level adjusted based on alpha activity in real time Low level High level IEEE VR’19

25. Cognitively Adaptive Training in VR Looxid IEEE VR’19 Alpha activity changed without significant change in task completion time

26. Quantifying Presence Looxid Self-reported Questionnaires vs. real-time Physiological and Neurological Data

27. Quantifying Presence with Brain Signals Looxid High Presence Low Presence

28. Quantifying Presence with Brain Signals Looxid High Presence Low Presence

29. Next steps: Using EEG and other physiological sensors Looxid • Emotionally and cognitively adapt interfaces for learning and training • Understand how brain synchronization works in collaborative XR • Effectively share emotional and cognitive states to create empathy in collaborative XR

30. Thank You! Looxid Arindam Dey Empathic Extended Reality and Pervasive Computing Lab Co-Innovation Group | University of Queensland http://empathicxr.org | a.dey@uq.edu.au

Empathic Extended Reality

Recommandé

Recommandé

Contenu connexe

Similaire à Empathic Extended Reality

Similaire à Empathic Extended Reality (20)

Plus de Joaquim Jorge

Plus de Joaquim Jorge (20)

Dernier

Dernier (20)

Empathic Extended Reality