Member Login

ARTICLE: Addressing gender pay disparities in engineering

Manufacturing is one of the top 3 engineering-heavy sectors in Australia, employing more than 46,000 qualified engineers. The manufacturing sector currently has a 70% male workforce, as discussed by Australian Cobotics Centre PhD candidate Akash Hettiarachchi in his recent webinar. The importance of gender equity to Australia’s global competitiveness in manufacturing was also highlighted in a recent parliamentary inquiry, which recommended a national strategy to attract and retain under-represented groups (including women) to advanced manufacturing careers. Manufacturing organisations, government departments and industry bodies are making concerted efforts to increase gender balance in the sector so they can achieve the benefits of a diverse workforce. 

At present, only 14% of engineers working in Australia are women. I was recently invited by the Australasian Tunnelling Society and Engineers Australia to present and be part of a panel at an International Women in Engineering Day (INWED) event, Bridging the Gap: Addressing Gender Pay Disparities in Engineering. INWED celebrates women’s contribution to the engineering profession and the 2024 theme is Enhanced by Engineering. However, in all industry sectors and occupations in Australia and most of the world, women’s contribution is still under-valued in terms of pay.  

The current gender pay gap in Australia (the difference between the average earnings of men and women), is 21.7% including full time, part time and casual workers and payments such as bonuses, overtime and commission. This means that on average, for every $1 a male worker makes, a female worker makes 78 cents. The gap is still 13.7% even when only including the base salaries of full-time workers. National statistics, the international Global Gender Gap Index, company reporting, and research show that a gap exists even when considerations such as experience and education are controlled for, and only part of the gap can be attributed to different career choices. A gender pay gap exists across nations, industries, occupations and at different levels of pay. It is however higher in male dominated industry sectors, industries with higher bonus, overtime or commission payments, higher paid roles, and organisations with fewer women in leadership. 

At the Bridging the Gap event, we discussed the gender pay gap, the policy and reporting framework in Australia, and actions that individuals, managers and organisations can take to address pay disparities.  

For the first time in 2024, the Workplace Gender Equality Agency (WGEA) published the gender pay gaps of all private sector employers with 100 or more staff members. The WGEA Data Explorer provides a rich source of data for anyone interested in the gender equity performance, policies and strategies of their own and other organisations. As well as gender pay gap data, policy and action, you can use the WGEA Data Explorer to see and compare industry and employer data on other indicators including the composition of the workforce and boards, access to and use of flexible work and parental leave by men, women and managers, employee consultation and harassment. Initiatives such as conducting and acting on the results of a gender pay audit, making pay more transparent, increasing the proportion of women in leadership, identifying and removing gender bias from recruitment and promotion decisions, and encouraging men to access flexible work and parental leave can all improve the gender pay gap.  

Australian Cobotics Centre Program 5 (The Human-Robot Workforce) has several researchers with experience in researching gender equity. We can assist companies of all sizes to consider how they can evaluate gender equity and realise the benefits for their organisation.  

Meet our E.P.I.C. Researcher, Jasper Vermeulen

Jasper Vermeulen is a PhD researcher based at Queensland University of Technology and his project is part of the Designing Socio-technical Robotic Systems at the Australian Cobotics Centre. We interviewed Jasper recently to find out more about why she does what he does.

  • Tell us a bit about yourself and your research with the Centre? Include the long-term impact of what you are doing.

I have always been fascinated by how novel technologies integrate into our daily lives. Human-Robot Collaboration (HRC) offers an exciting opportunity to enhance human qualities and working conditions rather than replace human effort. My research focuses on uncovering crucial human factors in HRC applications, particularly manufacturing and robot-assisted surgery. By examining the real-world experiences of individuals collaborating with robots, I aim to design better HRC systems for Industry 5.0. My work seeks to improve the efficiency and safety of HRC, making these technologies more user-friendly and effective in complex environments.

Why did you decide to be a part of the Australian Cobotics Centre?

HRC is a rapidly evolving field with many unexplored avenues. Being part of the Australian Cobotics Centre allows me to contribute to the foundation of future work by enhancing human efforts through Collaborative Robotics. The Centre offers a unique opportunity to foster industry connections and make a direct impact through my research. Collaborating closely with industry practitioners helps bridge the gap between academia and industry, ensuring that my work effectively addresses practical challenges.

  • What project are you most proud of throughout your career and why?

I am particularly proud of my current projects with the Australian Cobotics Centre, which focus on human factors in surgery and manufacturing. These studies are grounded in real-world scenarios, like assembly line processes and robot-assisted surgical procedures. By emphasising user experience and leveraging action research with industry partners, I aim to create systems where humans and robots work together seamlessly. This approach not only centres around human needs but also tackles practical challenges, enhancing efficiency and safety in both industries.

  • What do you hope the long-term impact of your work will be?

I hope my research contributes to a deeper understanding of human experiences with HRC, aiding both academic researchers and industry practitioners. As robots become more embedded in our daily lives, understanding the human factors involved in this collaboration is crucial. My work aims to ensure that HRC systems are designed to effectively enhance human capabilities and work conditions.

  • Aside from your research, what topic could you give an hour-long presentation on with little to no preparation?

I could give an hour-long presentation on smart home technology, which I find fascinating. While smart home devices offer convenience, connectedness, and entertainment, they also present privacy risks and surveillance concerns. My extensive research on this topic highlights the need for better education on the potential drawbacks of these technologies. With the rapid growth of smart home appliances, there’s plenty of material to discuss in an hour-long presentation.

Read more about Jasper’s project titled ‘Human Factors in Collaborative Robotics’ HERE.

ARTICLE: Enhancing Hydraulic Maintenance Operations with Multi-modal Feedback

Hydraulic systems are integral to industrial applications that require significant force, such as mining and manufacturing. Despite their power and efficiency, traditional hydraulic systems pose operational risks, especially when relying on binary controls and low-resolution feedback mechanisms. To address these challenges, a research team from the University of Technology, Sydney, led by Danial Rizvi, explored the potential of multi-modal feedback to enhance safety and performance in hydraulic maintenance operations.

The Challenges of Traditional Hydraulic Systems

In industrial settings, hydraulic systems are essential for tasks like installing and removing bushings and bearings. However, these systems typically use binary controls, limiting operators to simple open or close actions. This lack of precision can lead to operational errors and safety risks. Operators often rely on visual and auditory cues, which can be inconsistent and unreliable, increasing the potential for accidents and equipment failure.

Multi-modal Feedback: A New Approach

The research aimed to improve hydraulic maintenance operations by integrating haptic feedback through an adaptive trigger mechanism. This approach provides operators with tactile feedback, simulating the pressure build-up in hydraulic systems. The study compared the effectiveness of this haptic feedback against traditional visual and auditory cues.

Methodology

The team conducted a user study involving 10 participants operating a simulated hydraulic system using a re-programmed DualSense controller. This controller provided four types of feedback: force (through adaptive trigger resistance), visual (pressure readings), sound (auditory cues), and vibration (tactile cues). Participants performed tasks under different feedback conditions to evaluate the impact on performance and user experience.

Performance Analysis

The study measured three key performance metrics: elapsed time, final pressure (PSI), and extension percentage. The results showed no significant differences in task performance across the different feedback types. However, participants expressed a preference for the adaptive trigger in subjective evaluations, noting that it enhanced their control and reduced cognitive load.

Subjective Ratings

Participants rated their comfort and confidence with each feedback type. The adaptive trigger received the highest median comfort rating, while the vibration feedback was the least preferred. Overall, the study found that while all feedback types enabled participants to achieve the desired hydraulic pressures, the adaptive trigger offered slight advantages in user comfort and perceived control.

Implications for Industrial Maintenance

The integration of haptic feedback into hydraulic systems holds promise for improving safety and efficiency in industrial maintenance. By providing operators with more precise and intuitive control mechanisms, multi-modal feedback systems can reduce reliance on less reliable sensory cues and enhance overall operational safety.

Future Research

Further research is needed to explore the long-term benefits of multi-modal feedback in diverse industrial environments. Expanding the participant pool and incorporating real-world scenarios will help validate these findings and refine the technology for broader application.

Conclusion

The study conducted by the University of Technology, Sydney, demonstrates the potential of multi-modal feedback to enhance hydraulic maintenance operations. While traditional feedback mechanisms remain effective, the adaptive trigger offers additional benefits in user comfort and control. As industries continue to evolve, integrating advanced feedback systems into hydraulic operations can lead to safer and more efficient maintenance practices.

References:

  • Danial Rizvi, Dinh Tung Le, Munia Ahamed, Sheila Sutjipto, Gavin Paul. “Multi-modal Feedback for Enhanced Hydraulic Maintenance Operations.” University of Technology, Sydney.

Welcoming a new Industry Partner – Workr Labs

We’re delighted to welcome Workr Labs Inc. as our newest industry partner to the Australian Cobotics Centre! Led by a talented team including Ken Macken and Richard Pruss, Workr Labs brings a wealth of expertise in software-defined robotics and manufacturing.

About Workr Labs

Workr Labs is a forward-thinking startup with a mission to develop a software platform for industrial robotics that is both accessible and user-friendly for businesses of all sizes. Their innovative approach to Human-Robot Interaction streamlines task distribution and optimization, reducing the need for extensive expertise and making robotics more user-friendly. This aligns perfectly with our goal of integrating advanced technology into the workplace in a way that is intuitive and efficient.

The Importance of This Partnership

Workr Labs’ vision ties closely with the Centre’s research programs, particularly those focused on addressing human and design considerations that need to be factored in with new technology. We believe our collaboration will bring significant benefits, including:

  1. Enhanced Research Synergy: By combining our research expertise with Workr Labs’ innovative solutions, we aim to push the boundaries of what is possible in the field of collaborative robotics. This partnership will help us develop more intuitive, adaptable, and capable cobots.
  2. Industry Advancement: Our collaboration will provide our industry partners early access to new developments, facilitating the broader adoption of cobots in manufacturing. This means businesses can integrate advanced robotics into their processes more seamlessly and efficiently.
  3. Real-World Application: Along with our other industry partners (ARM Hub, B&R Enclosures, Cook Medical, InfraBuild, IR4 PTY LTD, Stryker, TAFE Queensland, and Weld Australia), Workr Labs will offer our PhD and Postdoctoral researchers fantastic opportunities to apply their research to real-world industry problems. This practical experience is invaluable for both our researchers and the industries they will serve.

Looking Ahead

We are thrilled about the potential of this partnership and look forward to the innovative projects and advancements that will arise from our collaboration with Workr Labs. Together, we aim to make significant strides in the field of collaborative robotics, benefiting both industry and the workforce.

Stay tuned for more updates as our projects progress! We are excited to share our journey and the milestones we achieve along the way.

ARTICLE: Industry 4.0 Awareness and Experience Workshop

These workshops were organised and run by Swinburne University of Technology’s Factory of the Future and were funded through the Victorian Government’s Digital Jobs for Manufacturing (DJFM) program. 

This article is written by PhD researcher from Swinburne University of Technology, Jagannatha Pyaraka.

In a series of enlightening workshops, Swinburne University of Technology has taken significant step in bridging the gap between industry professionals and the transformative potential of Industry 4.0 technologies. Over the past few weeks, four workshops were organized at strategic locations to maximize outreach and impact. The workshops were held at the VGBO office in Bundoora, Holiday Inn Dandenong, Rydges Geelong, and Mercure Ballarat. These sessions aimed to raise awareness and provide hands-on experience with collaborative robots (cobots), a foundation of modern industrial automation and other Industry 4.0 technologies such as AR, VR and wearable sensors.

The workshops attracted operations managers, CEOs, CFOs, and other key decision-makers eager to understand the practical applications and benefits of cobots in their respective fields. Accompanied by my ACC colleague, Dr. Anushani Bibile, we used the easily portable and cost-effective UFactory xArm6 cobot to demonstrate cobotics functionality.

The workshops commenced with an introduction to collaborative robots. Unlike traditional industrial robots, which often require extensive programming and are confined to specific tasks, cobots are designed to share a workspace with humans. Their ease of programming, adaptability to various tasks, and advanced safety features make them suitable for dynamic and evolving industrial environments.

To illustrate these points, we demonstrated a program involving the stacking of four objects. The objects were placed in predefined positions, and xArm6 was tasked with picking each object and stacking them. This exercise highlighted the cobot’s ability to perform repetitive tasks and its intuitive programming interface. Using Blockly, a visual programming language, participants observed how quickly and easily they could teach the cobot to execute tasks.

Following the demonstration, participants had the opportunity to interact with xArm6. They used Blockly to program the cobot for a simple pick-and-place task. This exercise allowed them to experience the user-friendly interface and the cobot’s responsiveness. The feedback was positive, with many participants noting how quickly they could learn to program and operate the cobot.

The hands-on session helped to remove common misconceptions about the complexity and inflexibility of industrial automation. By the end of the workshop, participants had a better understanding of how cobots can be integrated into their operations to enhance productivity, safety, and cost-effectiveness.

The workshops also emphasized the cost-effectiveness of cobots. Unlike traditional robots that require significant investment in programming and setup, cobots like the xArm6 offer an affordable solution without compromising performance. Their advanced safety systems, which allow them to operate safely alongside human workers, make them a viable option for businesses of all sizes.

Specific feedback from participants highlighted the positive impact and value of these sessions. One attendee noted, “The workshop provided a great insight into how Industry 4.0 can better impact our business and automate our processes.” Another participant appreciated the practical demonstrations, stating, “It was great to see the practical applications during the demonstrations.” Many attendees emphasized that the hands-on experience was invaluable, with one remarking, “Cobots demo was very stimulating. Thoroughly enjoyed the workshop.”

Before the workshop, common reactions included uncertainty about the complexity and applicability of cobots in their operations. After the sessions, many participants expressed confidence in integrating these technologies into their workflows, recognizing the potential for improved efficiency and innovation.

Overall, these workshops effectively bridged the knowledge gap for attendees, providing them with the tools and understanding necessary to embrace Industry 4.0 technologies. As more companies recognize the benefits of automation, the demand for cobots is set to rise, paving the way for a more efficient and innovative industrial landscape.

 

Meet our E.P.I.C. Researcher, Yuan Liu

Yuan Liu is a PhD researcher based at Queensland University of Technology and his project is part of the Designing Socio-technical Robotic Systems at the Australian Cobotics Centre. We interviewed Yuan recently to find out more about why she does what he does.

  • Tell us a bit about yourself and your research with the Centre? Include the long-term impact of what you are doing.

In the realm of Human-Robot Collaboration (HRC), particularly in complex and dynamic environments such as assembly lines and robot-assisted surgeries, the quality of human decisions plays a pivotal role in determining outcomes. Several factors influence these decisions, including the physical and cognitive workload experienced by human operators, the design of user interfaces, and the configuration of the workspace. One promising solution for enhancing decision making in these settings involves the adoption of Extended Reality (XR) technologies. XR can offer intuitive communication, effective data visualization, user-friendly interfaces, and facilitate the ergonomic design of workspaces.

My research focuses on identifying the specific factors that impact human decision making within HRC and investigates how XR technologies can be leveraged to improve these processes. This research will address the gap regarding human decision making in HRC, develop a framework of factors affect human decision making, investigate human decision-making process in detail during HRC task, guide future XR design in HRC.

Why did you decide to be a part of the Australian Cobotics Centre?

Prior to commencing my research with ACC, I completed a master’s degree in Interactive Media, where I engaged in a project that piqued my interest in the application of immersive technologies across various fields. This experience laid the foundation for my current research focus. ACC, where I am currently conducting my research, specialises in Human-Robot Collaboration (HRC). This centre is closely aligned with real industry demands and fosters an interdisciplinary approach, bringing together researchers from diverse fields to address all facets of HRC. This multidisciplinary environment is ideal for exploring the integration and impact of immersive technologies within human-robot interaction contexts.

  • What project are you most proud of throughout your career and why?

Throughout my research career thus far, I take particular pride in my ongoing projects within both the manufacturing and healthcare industries. These studies are firmly grounded in realistic scenarios-specifically, assembly line processes in manufacturing and robot-assisted surgical procedures in healthcare. My approach is deeply rooted in industry-relevant research, employing Human-Centred Design and Research through Design strategies to ensure that the studies are not only oriented around human needs but also address real-world challenges effectively. This focus aims to optimise user interaction and enhance the practicality of technological implementations in complex and dynamic environments.

  • What do you hope the long-term impact of your work will be?

I aspire for my research to make a substantial contribution to the field of human decision making within the context of Human-Robot Collaboration (HRC). My research introduces innovative applications for Extended Reality (XR) technologies in HRC, emphasising their role in enhancing human-in-the-loop systems. This is particularly relevant in the advanced manufacturing sectors of Industry 4.0 and Industry 5.0, where XR technologies are pivotal in supporting complex decision-making processes. By integrating these technologies, my work aims to facilitate more intuitive and effective collaborations between humans and robots, thereby driving efficiency and innovation in modern manufacturing environments.

  • Aside from your research, what topic could you give an hour-long presentation on with little to no preparation?

I could confidently give a presentation on emerging digital technologies, a topic I have pursued with passion for several years. My extensive knowledge, fueled by continuous exploration and reading, along with a solid background in design and technology, enables me to provide a detailed discussion. I can effectively engage an audience by sharing insightful perspectives on how these innovative technologies have profoundly influenced and reshaped our world.

Read more about Yuan’s project titled ‘Augmented and Virtual Reality in Collaborative Robotics’ HERE.

Cook Medical Placement

Our University of Technology Sydney PhD researcher, Louis Fernandez, has just completed a one-month placement with our industry partner, Cook Medical.

During his placement, Louis not only gathered data for his PhD research but also conducted a demonstration (alongside A/Prof Jared Donovan) for Cook Medical staff at their Continuous Improvement Expo. This demonstration allowed staff to interact with a cobot and experience how user-friendly it is to operate.

Louis also managed a couple of visits to QUT (Queensland University of Technology)), where he caught up with fellow members of the Human Robot Interaction research program, including James Dwyer, Dr Stine Johansen, and Prof Markus Rittenbruch.

We extend our thanks to Kettina Materna from Cook Medical for supervising Louis during his placement and to Louis’ Principal Supervisor, Dr Marc Carmichael, for organising the visit.

Such placements offer students valuable insights into the challenges faced by manufacturers when implementing technology. By engaging partners in the initial design process, solutions are more likely to be embraced by staff.

Learn more about Louis’ project – HERE

ARTICLE: Enhancing Collaboration Between Humans and Robots: The Critical Role of Human Factors Research

This article is written by Jasper Vermeulen, PhD researcher at the Australian Cobotics Centre.

 

Integrating collaborative robots (cobots) in factory environments offers substantial benefits for businesses, including increased operational efficiency and greater product customisation. Compared to traditional industrial robots, cobots are often smaller in size, offering both versatility in various tasks and cost-efficiency. From a technological perspective, the use of cobots can lead to significant improvements in processes.

Cobots: a double-edged sword?

While the advantages of cobots are clear, from a human-centric perspective, a more nuanced conclusion is required. In reality, cobots can present both benefits and challenges for operators. Cobots can help reduce physical strain and mitigate repetitive tasks. On the other hand, cobots may also increase mental effort and working closely together with cobots could cause stress. Furthermore, depending on the workspace and task, working with cobots could affect an operator’s posture for better or worse. This complexity highlights the need for studies into the operator’s experiences of working alongside cobots.

The Discipline of Human Factors

Human Factors is a field dedicated to the study of interactions between humans, technologies, and their environments. This scientific discipline is crucial for enhancing the safety and efficiency of socio-technical systems through interdisciplinary research. Specifically, in the realm of human-cobot collaboration, the discipline of Human Factors plays a pivotal role. By integrating diverse research perspectives—from Robotics and Usability Engineering to Design and Psychology—this discipline enables researchers to dissect and understand complex interactions and complex systems. More importantly, it provides a framework for translating these insights into practical applications, offering concrete design recommendations and effective technology implementation strategies.

Beyond safety

While safety in Human-Robot Interaction has been a central point in Human Factors research, studies specifically addressing human-cobot collaboration are relatively new. Traditionally, much research was aimed at safeguarding the human operator, ensuring their physical safety. Nevertheless, if we aim to improve the overall system performance and well-being of operators, we need to consider additional factors, besides safety. For instance, cobots typically operate at lower speeds as a safety measure, however, experienced operators might prefer a faster pace depending on the task and context. This suggests that speed adjustments could be made without compromising safety.

Looking Forward

As the adoption of cobots continues to grow in industrial settings, it is crucial to deepen our understanding of the factors influencing human-cobot collaboration. Researchers in Human Factors can offer valuable insights by examining the diverse experiences of human operators in cobot-assisted tasks, considering individual differences, different kinds of tasks, various workspaces and cobot capabilities.

Ultimately, while cobots offer the potential to streamline processes, enhance customisation, and reduce costs, their implementation should also focus on improving human operators’ physical safety and mental health. These considerations emphasise the importance of adopting new technologies in genuinely advantageous ways, ensuring a balanced approach to innovation and worker well-being.

Stay Informed on Human Factors in Human-Robot Collaboration

If you’re interested in the latest advancements in human factors research within the field of Human-Robot Collaboration, make sure to follow the activities of Program 3.1 at the Australian Cobotics Centre. We conduct human-centred research using real-world case studies in partnership with industry leaders, focusing on the impact of human factors on operators in practical cobot applications. Our current projects include exploring cobot integration in manufacturing tasks and investigating human factors in robot-assisted surgeries.

Follow our progress on the Australian Cobotics Centre’s LinkedIn page for the latest updates and insights.

ARTICLE: Robotic Blended Sonification: Consequential Robot Sound as Creative Material for Human-Robot Interaction

This article is written by Stine S. Johansen, Jared Donovan, Markus Rittenbruch (Human-Robot-Interaction Program) at Australian Cobotics Centre, and Yanto Browning, Anthony Brumpton (QUT)

Abstract
Current research in robotic sounds generally focuses on either masking the consequential sound produced by the robot or on sonifying data about the robot to create a synthetic robot sound. We propose to capture, modify, and utilise rather than mask the sounds that robots are already producing. In short, this approach relies on capturing a robot’s sounds, processing them according to contextual information (e.g., collaborators’ proximity or particular work sequences), and playing back the modified sound. Previous research indicates the usefulness of non-semantic, and even mechanical, sounds as a communication tool for conveying robotic affect and function. Adding to this, this paper presents a novel approach which makes two key contributions: (1) a technique for real-time capture and processing of consequential robot sounds, and (2) an approach to explore these sounds through direct human-robot interaction. Drawing on methodologies from design, human-robot interaction, and creative practice, the resulting ‘Robotic Blended Sonification’ is a concept which transforms the consequential robot sounds into a creative material that can be explored artistically and within application-based studies.

Keywords
Robotics, Sound, Sonification, Human-Robot Collaboration, Participatory Art, Transdisciplinary

Introduction and Background
The use of sound as a communication technique for robots is an emerging topic of interest in the field of Human-Robot Interaction (HRI). Termed the “Robot Soundscape”, Robinson et al. mapped various contexts in which sound can play a role in HRI. This includes “sound uttered by robots, sound and music performed by robots, sound as background to HRI scenarios, sound associated with robot movement, and sound responsive to human actions” [7, p. 37]. As such, robot sound encompasses both semantic and non-semantic communication as well as the sounds that robots inherently produce through their mechanical configurations. With reference to product design research, the latter is often referred to as “consequential sound” [11]. This short paper investigates the research question: How can consequential robot sound be used as a material for creative exploration of sound in HRI?

This research offers two key contributions: (1) an approach to using, rather than masking [9], sounds directly produced by the robot in real-time, and (2) offering a way to explore those sounds through direct interactions with a robot. As an initial implication, this enables explorations of the sound through creative and open-ended prototyping. In the longer-term, this has the potential of leveraging and extending collaborators’ existing tacit knowledge about the sounds that mechanical systems make during particular task sequences as well as during normal operation versus breakdowns. Examples of using other communication modalities exist, mostly relying on visual feedback. Visual feedback allows collaborators to see, e.g., intended robotic trajectory and whether it is safe to move closer to the robot at any time. This assumes, however, that the human-robot collaboration follows a schedule in which the collaborator is aware of approximately when they can approach the robot. Sometimes, this timing is not possible to schedule, and collaborators must maintain visual focus on their task. This means that it is crucial to investigate ways of providing information about the robot’s task flow and appropriate timings for collaborative tasks. In other words, there is a need for non-visual feedback modalities that enable collaborators to switch between coexistence and collaboration with the robot. In order to achieve this aim, it is necessary to make these non-visual modalities of robot interaction available for exploration as creative ‘materials’ for prototyping new forms of human-robot interaction.

Prototyping sound design for social robots has received particular attention in prior research, e.g., movement sonification for social HRI [4]. However, this knowledge cannot be directly transferred when designing affective communication, including sound, for robots that are not anthropomorphic, e.g., mobile field robots, industrial robots for manufacturing, and other typical utilitarian robots [1]. In prior research of consequential robot sound, Moore et al. studied the sounds of robot servos and outlined a roadmap for research into “consequential sonic interaction design” [6]. The authors state that robot sound experiences are subjective and call for approaches that address this rather than, e.g., upgrade the quality of a servo to reduce noise objectively. Frid et al. also explored mechanical sounds of the Nao robot for movement sonification in social HRI [4]. They evaluated this through Amazon Mechanical Turk, where participants rated the sounds according to different perceptual measures Extending this into ways of modifying robot sounds, robotic sonification that conveys intent without requiring visual focus has been created by mapping movements in each degree of freedom for a robot arm to pitch and timbre [12]. The sound in that study, however, was created from sample motor sounds as opposed to the actual and real time consequential sounds of the robot. Another way this has been investigated is with video of a moving robot, Fetch, overlaid with either mechanical, harmonic, and musical sound to communicate the robot’s inner workings and movement [8]. This previous research indicates that people can identify nuances of robotic sounds but has yet to address if that is also the case for real time consequential robot sounds.

Robotic Blended Sonification
Robot sound has received increasing interest throughout the past decade, particularly for designing sounds uttered or performed by robots, background sound, sonification, or masking consequential robot sound [9]. Extending this previous research, we contribute with a novel approach to utilising and designing with consequential robot sound. Our approach for ‘Robotic Blended Sonification’ bridges prior research on consequential sound, movement sonification, and sound that is responsive to human actions. Furthermore, it relies on the real-time sounds of the robot as opposed to pre-made recordings that are subsequently aligned to movements. A challenge for selecting the sounds a robot could make is that people have a strong set of pre-existing associations between robots and certain kinds of sounds. On one hand, this might provide a basis for helping people to interpret an intended meaning or signal from a sound (e.g., a danger signal), but it also risks that robot sounds remain cliched (beeps and boops), and may ultimately limit the creative potentials for robotic sound design. In this sense, Robotic Blended Sonification is an appealing approach because it offers the possibility of developing a sonic palette grounded in the physical reality of the robot, while also allowing for aspects of these sounds to be amplified, attenuated, or manipulated to create new meanings. Blended sonification has previously been described as “the process of manipulating physical interaction sounds or environmental sounds in such a way that the resulting sound signal carries additional information of interest while the formed auditory gestalt is still perceived as coherent auditory event” [10]. As such, it is an approach to augment existing sounds for purposes such as conveying information to people indirectly.

To achieve real-time robotic blended sonification, we use a series of electromagnetic field microphones placed at key articulation points on the robot. Our current setup uses a Universal Robots UR10 collaborative robotic arm. The recorded signals are amplified and sent to a Digital Audio Workstation (DAW), where they can be blended with sampled and synthesized elements and processed in distinct ways to create interactive soundscapes. Simultaneously to the real-time capture of the robot’s audio signals, we enable direct interactions with the robot through the Grasshopper programming environment within Rhinoceros 3D (Rhino) and the RobotExMachina bridge and Grasshopper plugin [3]. We capture the real-time pose of the robot’s Tool Center Point (TCP) in Grasshopper. Interaction is made possible via the Open Sound Control (OSC) protocol, with the Grasshopper programming environment sending a series of OSC values for the TCP. The real-time positional data also includes the pitch, roll, and yaw of each section of the robotic arm. Interaction with the robot arm is enabled through the Fologram plugin for Grasshopper and Rhino. The virtual robot is anchored to the position of the physical robot. The distance between the base of the robot and a smartphone is then calculated and used to direct the TCP towards the collaborator. This enables realtime interaction for exploring sounds for different motions and speeds. For our prototype, OSC messages from the robotic movements are received in the Ableton Live DAW, along with the Max/MSP programming environment, and then assigned to distinct parameters of digital signal processing tools to alter elements of the soundscape. The plan for the initial prototype setup is to use five discrete speakers: A quadraphonic
setup to allow for 360 degree coverage in a small installation space, along with a point source speaker located at the base of the robotic arm. The number of speakers is scalable to the size of the installation space and intent of the installation. The point source speaker alone is enough to gather data on the effects of robotic blended sonification on HRI, while multi-speaker configurations allow for better coverage in larger environments, enable investigations for non-dyadic human-robot interactions, and provide more creative options when it comes to designing soundscapes.

Directions for Future Research
Ways of using non-musical instruments for musical expressions have a long history within sound and music art. Early examples include the work of John Cage, e.g., Child of Tree (1975) where a solo percussionist performs with electrically amplified plant materials [2], or the more recent concert Inner Out (2015) by Nicola Giannini where melting ice blocks are turned into percussive elements [5]. In a similar manner, our approach enables performance with robotic sound, subsequently allowing for a creative exploration of how those sounds affect and could be utilised for better human-robot collaborations. With the proposed approach, we identify new immediate avenues for research in the form of the following research questions:

Robot Sound as Creative Material
In what ways can the consequential sound of a robot be used as a creative material in explorations of robot sound design? This can entail investigations through different configurations, including dyadic and non-dyadic interactions, levels of human-robot proximity, and different spatial arrangements. Furthermore, the interaction itself will play a crucial part in the way that the sound is both created and experienced, e.g., whether a collaborator is touching the robot physically or, as in our current setup, is interacting on a distance.

Processing Consequential Robot Sound
In what ways can or should we process the consequential sound material? Two key points are connected to this. First, the consequential sound forms a basis for the resulting sound output which can be modified in various ways. Future research can entail exploring these, including the fact that different robots produce different consequential sounds that subsequently, will lead to different meaningful modifications. Second, our approach can be complemented by capturing data from the surrounding environment to use as input for sound processing.

Engaging People in Reflection
How can we prompt people’s reflections about consequential robot sounds through direct interaction? While prior research has demonstrated ways to investigate consequential robot sound, e.g., through overlaying video with mechanical sounds, our approach enables people to explore sounds that result from their own interactions with a robot. This can be utilised for both structured and unstructured setups, depending on the purpose of the investigation. In our current setup, we invite for artistic exploration and expression. For more utilitarian purposes, the setup can be created in the context within which a robot is or could be present. This could support other existing methods for mapping and designing interventions into soundscapes.

Conclusion
In this short paper, we have described a novel approach for exploring and prototyping with consequential robot sound. This approach extends prior research by providing a technique for capturing, processing, and reproducing sounds in real-time during collaborators’ interactions with the robot.

Acknowledgments
This research is jointly funded through the Australian Research Council Industrial Transformation Training Centre (ITTC) for Collaborative Robotics in Advanced Manufacturing under grant IC200100001 and the QUT Centre for Robotics.

References
[1] Bethel, C. L., and Murphy, R. R. 2006. Auditory and other non-verbal expressions of affect for robots. In AAAI fall symposium: aurally informed performance, 1–5.
[2] Cage, J. 1975. Child of Tree. Peters Edition EP 66685. https://www.johncage.org/pp/ John-Cage-Work-Detail.cfm?work_ID=40.
[3] del Castello, G. 2023. RobotExMachina. GitHub repository. https://github.com/RobotExMachina.
[4] Frid, E.; Bresin, R.; and Alexanderson, S. 2018. Perception of mechanical sounds inherent to expressive gestures of a nao robot-implications for movement sonification of humanoids.
[5] Giannini, N. 2015. Inner Out. Nicola Giannini. https://www.nicolagiannini.com/ portfolio/inner-out-2/.
[6] Moore, D.; Tennent, H.; Martelaro, N.; and Ju, W. 2017. Making noise intentional: A study of servo sound perception. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’17, 12–21. New York, NY, USA: Association for Computing Machinery.
[7] Robinson, F. A.; Bown, O.; and Velonaki, M. 2023. The robot soundscape. In Cultural Robotics: Social Robots and Their Emergent Cultural Ecologies. Springer. 35–65.
[8] Robinson, F. A.; Velonaki, M.; and Bown, O. 2021. Smooth operator: Tuning robot perception through artificial movement sound. In Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’21, 53–62. New York, NY, USA: Association for Computing Machinery.
[9] Trovato, G.; Paredes, R.; Balvin, J.; Cuellar, F.; Thomsen, N. B.; Bech, S.; and Tan, Z.-H. 2018. The sound or silence: investigating the influence of robot noise on proxemics. In 2018 27th IEEE international symposium on robot and human interactive communication (RO-MAN), 713–718. IEEE.
[10] Tunnermann, R.; Hammerschmidt, J.; and Hermann, T. ¨ 2013. Blended sonification: Sonification for casual interaction. In ICAD 2013-Proceedings of the International Conference on Auditory Display.
[11] Van Egmond, R. 2008. The experience of product sounds. In Product experience. Elsevier. 69–89.
[12] Zahray, L.; Savery, R.; Syrkett, L.; and Weinberg, G. 2020. Robot gesture sonification to enhance awareness of robot status and enjoyment of interaction. In 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO MAN), 978–985. IEEE.

Author Biographies
* Stine S. Johansen is a Postdoctoral Research Fellow in the Australian Cobotics Centre. Her research focuses on designing interactions with and visualisations of complex cyberphysical systems.
* Yanto Browning is Lecturer at Queensland University of Technology in music and interactive technologies, with extensive experience as audio engineer.
* Anthony Brumpton is artist academic working in the field of Aural Scenography. He likes the sounds of birds more than planes, but thinks there is a place for both.
* Jared Donovan is Associate Professor at Queensland University of Technology. His research focuses on finding better ways for people to be able to interact with new interactive technologies in their work, currently focusing on the design of robotics to improve manufacturing.
* Markus Rittenbruch, Professor of Interaction Design at Queensland University of Technology, specialises in the participatory design of collaborative technologies. His research also explores designerly approaches to study how collaborative robots can better support people in work settings.

Meet our E.P.I.C. Researcher, James Dwyer

James Dwyer is a PhD researcher based at Queensland University of Technology and his project is part of the Human-Robot-Interaction Program at the Australian Cobotics Centre. We interviewed James recently to find out more about why he does what he does.

  • Tell us a bit about yourself and your research with the Centre?

Beginning my academic journey with a Bachelor of Psychology, I have always been deeply interested in human behaviour and the ways in which people communicate and connect with each other. After finishing this degree and entering the working world, I found a job in customer support, where I encountered firsthand the frustrations caused by poorly designed products and the impact technology has on people’s daily lives, sparking my interest in studying Industrial Design. Through this degree, I became interested in human-centred design as a way of approaching the design process to better meet user needs. My transition into design research, particularly focusing on HRI, was a natural progression, driven by a desire to blend my interests in people, technology, and prototyping as a research tool. At the Australian Cobotics Centre, my PhD research aims to develop an HRC prototyping toolkit that supports collaborative design approaches within manufacturing and surgical contexts. This work seeks to fill a crucial gap by integrating end-user needs and context of use into the design of cobots through collaborative design approaches. Working with industry partners Stryker and Cook Medical will be an invaluable part of this process, grounding my research in practical application and amplifying its impact. In the long term, I envision my research contributing significantly to the field by advancing HRC methodologies, promoting a more human-centred approach to HRC research, and broadening the scope of cobot applications across various industries.

Why did you decide to be a part of the Australian Cobotics Centre?

My decision to join the Australian Cobotics Centre was born from my interest in the Project 2.2 research topic, “Human-Robot Interaction (HRI) prototyping toolkit.” This topic aligned with my interest in prototyping as a research approach and engaging with end users through collaborative design processes, particularly in the context of HRI research. This project extends the work of my master’s research, where I developed a virtual reality-based HRI prototype. This experience was enlightening, presenting me with numerous ideas for future projects and highlighting the vast possibilities that Extended Reality (XR) technologies hold for prototyping and design research. This initial interest led me to look further into the Australian Cobotics Centre. What resonated with me about the Centre was its commitment to people-focused and innovative research. The Centre’s emphasis on collaboration between academia and industry also aligned with my belief in the importance of applied research that tackles real-world problems.

  • What project are you most proud of throughout your career and why?

One of the highlights of my career so far has been my involvement as the creative lead in the Soundline Project during my time with the ARS Electronica Futurelab Academy at QUT. The project explored how technology can facilitate group flow, transforming festival cues, lines and waiting areas, which might be considered an ‘error’ in the festival experiences, into playful and creative opportunities. Soundline was designed to allow people at any level of musical proficiency to contribute meaningfully to a collective soundscape through physical interaction with five different musical instruments. The instruments allowed participants to create music through physical movement, fostering a unique collaborative experience between the performers and audience members. This project is particularly dear to me for several reasons. It was a testament to the power of interdisciplinary collaboration, combining elements of design, technology, music, and dance performance. The hands-on experience of guiding the project from its initial concept to its public execution was also invaluable, teaching me the importance of an iterative design approach that integrated different perspectives and skill sets. Furthermore, the recognition of our work as a finalist for the IxDA Interaction Awards in 2019 was an affirming milestone, underscoring the project’s impact and the potential of interdisciplinary practice in design practice and technological development.

  • What do you hope the long-term impact of your work will be?

My research aims to address a critical gap in Human-Robot Interaction (HRI) by developing a prototyping toolkit that supports collaborative design approaches with end-users, a facet often overlooked in current practices. The need for a more human-centred approach in HRC research is evident, as current trends lean heavily towards technology-centric methods. The socio-cultural and socio-technical challenges presented by implementing cobots in manufacturing and surgical contexts necessitate a balance between technological advancements and human needs. By advocating for a design process that respects and integrates worker preferences and concerns, my work contributes to a more inclusive and considerate approach to cobot implementation, potentially leading to safer, more comfortable work environments. The development of a prototyping toolkit that encourages co-design could significantly enhance interdisciplinary collaboration, bridging gaps between fields such as engineering, social science, and design. By making the design process more inclusive and participatory, this toolkit has the potential to influence broader discussions on technology implementation, worker participation, and the ethical considerations of integrating robotics in the workplace.

  • Aside from your research, what topic could you give an hour-long presentation on with little to no preparation?

Beyond my academic pursuits, my passions are varied and deeply rooted in the explorative and speculative realms of science fiction and design and the creative pursuits of music and art. This fascination extends to the disciplined and intricate world of martial arts, where I have spent half my life trying to achieve some level of proficiency but have gained, at the very least, a great appreciation for the mental discipline the practice instils in my daily life. Music also holds a special place in my heart. While I still struggle with theory, the process of song construction and the communal experience of improvisation captivates me, providing a unique form of collective engagement and reflection. Art, too, is a refuge for me, albeit my sketches may not win awards. The immersive process of striving to capture the right expression or scene is a form of meditation, a way to lose myself in creativity. Similarly, the joy of understanding processes through the act of making or deconstructing complex ideas is a thread that runs through all my hobbies. While I might describe myself as a “jack of all trades and master of none”, this eclectic mix of interests is interconnected, each informing and enriching the others; these activities reflect and reinforce my approach to life and work, revealing my weaknesses and areas for growth. Rather than talking about these topics, however, I often find I am more interested in delving into the depths of someone else’s expertise. In this, I find myself in the role of the perpetual student, eager to absorb and understand more.