The increasing connection of components of industrial machine production processes to form so-called cyber-physical systems is vehemently changing the working world of humans. The monitoring and control of automated machines is increasingly becoming the focus of operators.
The bachelor thesis, therefore, examines how processes can be optimized in the age of the fourth industrial revolution through augmented reality (AR) and how machine operators can be meaningfully supported. Its potential and the requirements for such an application in the context of a large print shop are analyzed and evaluated. The knowledge gained is bundled in the design of a system for the scenario of a potentially occurring error and its rectification. Different operating contexts are addressed in a uniform concept for various devices. The user is supported in solving the problem both at the location-based desktop in the control center and on the move in the production hall. The focus here is on using a head-mounted display for process optimization. This is evaluated in this thesis with the help of a usability test.
The bachelor thesis contributed to the ‚Industry Showcase 2016‘ of Ergosign GmbH. The showcase provides decision-makers from the industry with forward-looking insights into the potential of current technologies.
The following sections present essential parts of the project and are excerpts from the bachelor’s thesis translated into English.
Getting to Know the Users: Personas
A persona is a fictitious archetypal user of a system with specific characteristics, behavioral traits, and goals. It is a powerful tool for better understanding potential users of the system to be designed and for defining goals and functions. Furthermore, personas make it possible to communicate more efficiently within a team or with decision-makers, create a consistent design, and measure a system’s effectiveness. Personas can also be used to describe scenarios more concretely and to form a uniform mental model of the project participants.
The effects of Industry 4.0 are merging traditional user roles and changing the range of tasks workers perform. The machine operator is much more in the role of the supervising authority. The following persona describes a master printer and was created against the background of these developments. It combines characteristics of various conventional user profiles, such as shift supervisor and machine operator, and is adapted to the specific context of a large print shop.
Describing the Context of Use: Scenarios
Scenarios contextualize the operation of a (planned) application within a story. They describe how a particular user or type of user could optimally interact with a system in a particular situation to achieve a specific goal (Cooper, Reinmann, and Cronin 2007, 109). Based on this knowledge, the concept’s functions and requirements can be derived. Generally, a scenario answers the W questions: Who does what? Where? How? When? And why? Necessary functionalities and other requirements for the concept can be derived from this knowledge base.
The scenarios created describe the occurrence of an error and its rectification in a large print shop. They are illustrated and described below using scribbles. In the description, reference is made to the persona of Florian Hummel to reinforce the reader’s mental model and make the application context more tangible.
Head-Mounted Display (HMD)
(1) Florian Hummel registers an error on the print shop’s control room monitor. A printing press is at a standstill because a motor protection switch has been triggered. He wants to rectify the fault as quickly as possible, as the costs increase with every minute of downtime.
(2) Instead of wasting valuable time leafing through room plans and endless operating instructions, Florian sets up a head-mounted display …
(3) … and uses it to navigate to the defective component via AR overlays…
(4) … while keeping an eye on essential machine values and his surroundings at all times if required.
(5) Step-by-step instructions appear at the relevant location depending on the context … (here: opening the cabinet with a specific key)
(6) … and helps Florian to rectify the fault confidently (even without prior knowledge if necessary).(Here: Reset circuit breaker)
(7) Florian is informed of the successful rectification and instructed to reset the system to its original state (here: close the cabinet).
(8) Once the fault has been rectified, the machine is back in operation. Florian can again display current data, such as key performance indicators, via the HMD.
Alternatively, the print foreman Florian Hummel could already be in the production hall with the head-mounted display when the error occurs. He receives a notification in the same way as on the control station interface and can start navigation and troubleshooting directly via the HMD.
Smart Watch
Florian could also use his smartwatch instead of the HMD to localize and rectify the source of the error. The scenario for using the smartwatch is described briefly below.
(1) Florian receives a notification on the Smartwatch, …
(2) … can also navigate to the defective component…
(3) … and rectify the fault with the help of step-by-step instructions.
(4) As soon as the fault has been rectified, Florian is shown a confirmation.
Concept
The concept developed is presented below. In addition to the design of the AR application for a head-mounted display, the thesis also includes the interface design for a desktop system in the control station, a human-machine interface (HMI) controlled by touch gestures on the printing press, and a smartwatch worn on the arm of the machine operator. The concept thus covers all the use cases envisaged in the scenario. As it would only be functional to develop the design for the HMD separately by coordinating the application with the other devices in the context of Industry 4.0, these are also created for completeness.
However, this thesis focuses on the application of augmented reality. Due to the time frame, only the essential screens of the other user interfaces were worked out and developed for the Ergosign showcase and for this bachelor thesis, so the focus is on the design of the interface of the head-mounted display.
Multimodal Interaction
An application for the industrial sector must ensure that the user can operate it even in critical situations and keeps sight of his environment when operating it. Therefore, it is necessary to consider the user’s context and situation. Information should always be available when needed and not hinder, distract, or overwhelm the user.
The application, therefore, has a straightforward navigation structure but primarily enables context-based interactions. The application designed below is a multimodal system. Communicating and interacting with such a system is possible via several input and output channels. These are called modalities and, according to Tzovaras (2008), can address the classic five human senses (sight, hearing, touch, smell, taste) as well as proprioception, i.e., the perception of one’s own body in space (Tzovaras 2008).
In the scenario, system users have various devices and modalities for interaction. The desktop computer in the control center has a keyboard, mouse, and voice input; a touch interface on the machine; the smartwatch on the wrist with touch and voice input; and the head-mounted display with voice and, above all, gesture input as well as hard buttons as a fallback. The smart devices also use and provide further information on the situation and context via the built-in sensors, and the strengths of the individual devices can be used depending on the context and task. In the noisy environment of a machine hall, for example, gestures can be used to interact with the head-mounted display, and the devices work together in the overall concept created below and are synchronized via a web interface.
Initialization of the Troubleshooting Process
If an error occurs, the user can start the troubleshooting process via the respective interface in the control station, on the machine, on the smartwatch, or on the head-mounted display. The system, therefore, offers the user complete flexibility and adapts to their situation. When the process is started, it is assigned to troubleshooting in the system, and the display of the devices registered to it is triggered accordingly. Another employee who may have registered the error on another device knows that a colleague has rectified it.
Wireframes: Navigation and Troubleshooting with the HMD
Wireframes: Navigation and Troubleshooting with the (android-based) Smart Watch
Prototyping & Evaluation
The empirical usability evaluation (N = 8) was carried out with a high-fidelity prototype based on wireframes and took place in June 2016 in the usability laboratory of Ergosign GmbH in Saarbrücken. Each of the eight sessions lasted between 20 and 25 minutes and included an introduction, a short preliminary interview, a usability test, and a final survey.
The test focused on evaluating the operating concept for the head-mounted display and its interaction with the control station interface. The evaluation looked at the specific question of whether the designed application can provide users with meaningful support in rectifying a specific error.
As the final equipment was not yet available at the time of the evaluation, the prototype was implemented with a video see-through HMD consisting of a Samsung Galaxy S6 and a Gear VR.
The prototype for the HMD was developed by Tobias Walter, the developer responsible at Ergosign GmbH, on the basis of the wireframes developed in the operating concept.
After participants initiated the troubleshooting via the prototyped desktop application of the control center, they were asked to identify the unknown error and rectify it using the AR prototype.
Visual Design
As part of the bachelor thesis, initial visual designs were developed for the interface on the head-mounted display and for the application on the smartwatch. The visual design should communicate a uniform overall concept across all devices. At the same time, an identification with the Ergosign brand should be created.
For the interface on the head-mounted display, the material design approach was used to visually support a simple yet clear structure and hierarchy of elements. Superordinate elements are lightly set off with a shadow. A subtle shadow also indicates interactive elements, such as the “Fix Error” air-tap button.
Metadata
Bachelor’s Thesis
Advisors & Reviewers
Institutions
Activities & Responsibilities
Research & Analysis
(Desk-based)
Conceptual Design
Prototyping
Usability Testing
Visual Design