Real-Time Remote Maintenance Support Based on Augmented Reality (AR)

Abstract

In the realm of the current industrial revolution, interesting innovations as well as new techniques are constantly being introduced by offering fertile ground for further investigation and improvement in the industrial engineering domain. More specifically, cutting-edge digital technologies in the field of Extended Reality (XR) have become mainstream including Augmented Reality (AR). Furthermore, Cloud Computing has enabled the provision of high-quality services, especially in the controversial field of maintenance. However, since modern machines are becoming more complex, maintenance must be carried out from experienced and well-trained personnel, while overseas support is timely and financially costly. Although AR is a back-bone technology facilitating the development of robust maintenance support tools, they are limited to the provision of predefined scenarios, covering only a limited number of scenarios. This research work aims to address this emerging challenge with the design and development of a framework, for the support of remote maintenance and repair operation based on AR, by creating suitable communication channels between the shop-floor technicians and the expert engineers who are utilizing real-time feedback from the operator’s field of view. The applicability of the developed framework is tested in vitro in a lab-based machine shop and in a real-life industrial scenario.

1. Introduction

Equipment maintenance is one of the key elements consisting of manufacturing systems. In their research work, Mourtzis et al. in Reference [1] highlight the importance equipment maintenance as part of the production lifecycle, which reaches 60%–70% of the total production cost. Therefore, being able to forecast machine maintenance operations and perform them in a short time period, can lead to successful troubleshooting and simultaneously increase machine tools availability. Additionally, since the replacement of damaged components can be as high as 70% of the total maintenance cost [1], it is one of the top priorities among manufacturing firms to discover alternative policies for cutting maintenance costs as a means of increasing their revenues. As a result, the overall production performance is optimized [2].

Besides the failure forecast, a major issue identified is the completion of accurate and error-free maintenance operations, and, thus, the reassurance of fully functional machines in the earliest time possible. Based on that concept, a considerable amount of research effort has been put on the design and development of real-time maintenance support tools and applications based on mobile devices to prevent unnecessary errors from happening [3,4,5,6]. With the evolution of Information and Communication Technologies (ICT) as well as the massive break out of augmented reality (AR), the research community extended that approach by exploiting the advantages of AR for data projection during the maintenance operations [6,7,8,9,10,11]. Generally, under the framework of Industry 4.0 in an attempt to bridge the physical world with virtual worlds, new technologies have emerged, such as Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR) [12]. These digital technologies lie under the umbrella term Extended Reality (XR). The term XR is often used by practitioners and scientists in order to describe the three immersive technologies mentioned above [13].

However, most of the research works available on the web cover basic aspects of Maintenance and Repair Operations (MRO) and provide flexible and accurate troubleshooting [3,4,5,6]. The constant innovation of cutting-edge technologies unveils new opportunities for remote maintenance support. Therefore, there is an apparent need for the development of real-time AR frameworks for the support of MRO.

More specifically, the proposed research deals with the design and development of a novel framework for real-time remote maintenance support based on AR. Inspired by the concepts of Cloud manufacturing and Product Service Systems (PSS), the framework was built in a cloud-enabled environment aimed at bridging the different working groups involved in the production lifecycle. Considering that companies seek increased operational availability, also known as availability (A) of their machines, engineers are constantly developing new maintenance policies that can ensure decreased Mean Time To Repair (MTTR) and increased Actual Machining Time (AMT). The following formula reflects the correlation between the above-mentioned variables and characteristics of actual machines.

ROTBF=MTBF−AMT

where ROTBF is the Remaining Operating Time Between Failures. MTBF is the Mean Time Between Failures. AMT is the Actual Machining Time.

A=MTBFMTTR+MTBF

where A is the machine availability. MTTR is the Mean Time To Repair.

From Equation (2), it is clear that, in order to increase the operational availability of the machine, represented by A in Equation (2), there are two possible alternatives. The first alternative is to increase the numerator, i.e., the MTBF, and the second alternative is to decrease the denominator, i.e.,(MTTR+MTBF). However, under the scope of this research work, the aim is to develop a framework that will facilitate the maintenance process of machine tools and, by extension, it will result in the minimization of MTTR. Therefore, if MTTR is decreased, then the denominator of Equation (2) will also decrease and, thereby, the operational availability of the machine tool will eventually increase.

It is stressed out that the above-mentioned indexes are the cornerstones of maintenance contracts offered either as a service by maintenance contractors or OEMs (Original Equipment Manufacturers) as part of their sales program. Although, a maintenance plan may seem attractive due to the low MTTR offered by the maintenance contractor, e.g., 24 h. The Mean Time To Recovery is another index that is highly affecting the quality of the service, as it directly affects Operational Availability of the machine. The Mean Time To Recovery is an aspect of technical maintenance, dependent on a variety of factors, such as part lead times, administrative delays, and technician transportation delays. Aimed at minimizing the MTTR and eliminating the factors mentioned above, this research work is dedicated to the modeling, development, and practical implementation of a real-time, AR-based framework for remote maintenance support.

The reminder of the paper is structured as follows. In Section 2, an extensive literature review is carried out, focusing on topics relevant to technical maintenance and AR. Then, in Section 3, the proposed system architecture is presented and, in continuation in Section 4, the steps leading to the practical implementation of the framework are discussed. Section 5 deals with the discussion of a real-life industrial scenario used for validating the developed framework. Lastly, the paper concludes in Section 6, arguing the results and the future work of this research work.

2. Literature Review 2.1. Literature Review Methodology

In order to conduct a substantial literature review, a research methodology has been adopted by Reference [14], restructured so as to meet the requirements of the current research work. For the research, the most well renowned publication databases, namely Google Scholar, ResearchGate, ScienceDirect, Scopus, and Web of Science, were used. A total of 34 research papers has been tracked and examined. The investigation of the available literature was performed in two levels based on the year of publication. More specifically, publications older than a decade have been examined in order to track down the evolution of AR throughout past years and the newer publications were examined so as to conclude on the current trends in the field of Remote Maintenance Support and, by extension, to compare similar frameworks for the proposed framework.

2.2. Maintenance as Part of the Modern Manufacturing System

As stated in the previous paragraphs, it is clear that maintenance besides being a crucial part of the production line, as it ensures the continuous and flawless operation of the physical assets, is also a very complex process. Therefore, there is a considerable amount of research work trying to cover all of its aspects. However, maintenance activities can be further categorized regarding the status of the physical assets. Concretely, the most common types of technical maintenance found are: (i) Corrective Maintenance, (ii) Preventive Maintenance, (iii) Risked-based Maintenance, and (iv) Condition-based Maintenance [15].

Throughout recent decades, many types of technical maintenance policies have been developed in an aim to cover the majority of the industrial needs. However, with the big diversity in the general policies that firms adopt, there is no optimal solution regarding the equipment maintenance plan. With that said, the maintenance policy adopted a company, and is highly dependent on the general business model and, by extension, to the general goals to be achieved. As a result, all of the above-mentioned policies are equally important for the research community and the industrial domain as well. Regardless, the scope of this research work is to present the design and development of a framework that supports field technicians mainly in corrective scenarios, where a malfunction has already occurred and, in many cases, standard procedures cannot be applied. Therefore, the research work of this paper could be classified as an enhanced corrective maintenance policy, which is based on the provision of service for on-demand guidance.

The current industrial era is characterized by the immense decentralization of the manufacturing networks. Thus, the actual manufacturing plant may be in a different region than the firm’s headquarters, which makes communication of the two departments more difficult [3,16]. However, the ongoing technological advances in the ICT have enabled the remote monitoring, remote operation, and the remote maintenance of the modern manufacturing systems [17,18]. Modernized maintenance under the term of Smart Maintenance have occurred over the last decade.

2.3. Augmented Reality Based Remote Maintenance

The concept of Digital Reality has been under the research scope during recent decades. With the advent of the fourth industrial revolution, such cutting-edge digital technologies have met an increasing advance, as a result of the technological advances in the ICT. More specifically, Augmented Reality (AR) is a very popular digital technology, which offers the advantage of enhancing the user’s perception by augmenting their physical environment with computer-generated information. In the field of AR, Azuma is considered a pioneer, with the first survey on AR being published two decades ago [19], concluding that AR back then was a very immature technology. A huge potential for future development was expected. In a more recent publication [20], the challenges that must be overcome by researchers in order to make AR a mainstream form of media are discussed. Among others, the most important challenge the research and development community have to overcome, is the creation of highly intuitive AR applications and tools.

In the available literature, there can be found many publications covering the AR-based aspect of maintenance. The majority of the works is focused on the remote assistance of technical maintenance operations through the various AR-based frameworks. A typical application of remote maintenance support is presented in Reference [21], where the technicians are capable of performing maintenance operations in robotic arm manipulators with the use of AR-projected instructions. Another unique AR framework for the support of MRO is presented in Reference [22]. In Reference [23], the authors cover another aspect of AR-based remote maintenance under the framework of Product Service Systems (PSS). Therefore, the AR tools developed can facilitate manufacturers to add value to their line of products by transitioning to the PSS philosophy.

In the research work presented in Reference [4], the authors have carried out research on the recent trends around the Smart Maintenance paradigm by investigating recent publications as well as by extracting useful empirical knowledge. In Reference [5], a collaborative remote maintenance framework has been suggested. Although the research work is inspiring and of good quality, the framework implementation is based on technologies and equipment that currently can be considered as obsolete, as the technician has limited mobility since a PC is required by their side so as to visualize the AR instructions. Moreover, the instructions are based mostly on the projection of textual information. Lastly, in this research work, the use of frame markers is not suitable for the maintenance of bigger and more complex machines. Therefore, what is needed is more robust and more compact framework enhancing the technician’s mobility when working on the machine. In a more recent paper, a similar remote maintenance support system based on AR is presented [24]. It is remarkable that the authors have put a considerable amount of effort on providing a mobile tool, based on the utilization of tablet PC. However, similar to the previous work presented, the use of frame markers is inevitable, which can highly affect the overall performance of the tool and, by extension, the performance of the shop-floor/field technician.

Although there is a continuous development of the AR-based tools proposed by the research community during the last two decades, there is a lack of added functionality. The above-mentioned development as well as the practical implementation of these frameworks in real-life industrial scenarios and the in-vitro experimentation in laboratory environments has led to the conclusion that AR has to treated as a back-bone technology. By extension, in the near future, the development of AR tools has to be further enhanced by the addition of more functionalities. A representative example is the framework presented in Reference [25], where an adaptive AR framework for machine operation as well as technical maintenance is presented. The adaptivity of the framework is based on the skill level of the technicians. Similarly, in Reference [26], an innovative framework based on a wrist-haptic tracking methodology is introduced.

2.4. Maintenance under the Framework of Industry 4.0

The modern manufacturing world is undergoing a digitalization phase under the framework of Industry 4.0. During the current industrial revolution, digital technologies have tremendously advanced, which enables the improvement of other scientific fields, including that of technical maintenance. With the recent development in the ICT, and the integration of Internet of Things (IoT), the manufacturing domain has passed in a whole new phase by relying on the utilization of data, i.e., Big Data Analytics. For the successful completion of MRO besides the technical knowledge/expertise of the stuff carrying out the operations, it is crucial that all the needed components are acquired. Therefore, companies are maintaining an inventory of components. However, under the framework of Industry 4.0, and with the utilization of digital technologies, the existing inventory practices can be further improved through proper classification of the MRO components while additive manufacturing can facilitate in reducing lead times to produce several components [6,7,8,9,10,26,27,28].

For Industry 4.0 applications, two key objectives are to ensure maximum uptime throughout the production chain, and to increase productivity while reducing production costs. In Reference [29], a systematic approach to analyze the strengths and weaknesses of current open-source Big Data and stream processing technologies and assess their use for Industry 4.0 use cases is presented. In addition, the model of Industry 4.0 is gradually being implemented in worldwide development, distribution, and marketing chains. As such, the design and integration of a stainless-steel predictive maintenance system is presented in Reference [29]. This case study uses data from machinery involved in producing high-quality steel sheets. Following the predictive maintenance field, the use of real-time detection and prediction algorithms regarding future failures has significantly benefited from the technological advances of Industry 4.0 era. There’s also growing interest in decision-making algorithms triggered by predictions of failure over the past few years. A literature review on decision making in predictive maintenance in the context of smart manufacturing is done in References [30,31]. Additionally, a review on predictive maintenance as an integrated predictive platform for production systems, focusing on maintenance approaches, methods, and tools is described in Reference [32].

Another significant issue is the estimation of the maintenance time for a new maintenance project, which is among the main maintenance offerings, is based solely on the experience and knowledge of the engineer. The research work presented in Reference [33] proposes a framework for knowledge-based calculation of maintenance time based on Key Performance Indicators (KPI) monitoring to support the capture and reuse of information in maintenance activities as well as to improve the performance of the given maintenance PSS. Next, an extendable and reusable scheduling approach in the context of predictive maintenance, which supports multiple heterogeneous inputs and outputs, is described in Reference [33]. In accordance with the current schedule, predictive maintenance indicators from the tracked equipment are used for scheduling maintenance operations. A web service architecture is implemented to accommodate highly different use scenarios such as suppliers of equipment.

3. System Architecture 3.1. General System Architecture

As discussed in the previous paragraphs, the scope of this research work is focused on the design and development of a real-time, remote maintenance assistance framework based on AR by enabling the creation of new communication channels between the expert engineers and the shop-floor technicians. Concretely, there is an opportunity to eliminate the need for preparing AR scenes, which considerably decreases the time and effort needed for creating such content. Moreover, certain limitations can be overcome as expert engineers, through the use of the proposed framework, are able to guide shop-floor technicians in less likely MRO scenarios. In order to meet the above-mentioned technical requirements, the methodology adopted by aiming at addressing the identified scientific gaps will be presented in detail in the following paragraphs. The general architecture flowchart is depicted in Figure 1.

The sequence of actions begins as long as a malfunctioning machine tool has been noticed. Then, the shop-floor technician who is carrying a device, a Head Mounted Display (HMD), contacts the expert engineer for assistance. It is stressed out that the coexistence of the two involved parties under the same premises is not required. More specifically, through the use of dedicated communication protocols, the shop-floor technicians can be remotely connected to the technical support department of the machine tool manufacturer. As long as the connection between the two parties has been established, a live video stream from the shop-floor technicians’ device is broadcast to the expert engineer. This process can be realized as a live teleconference between the two parties. Therefore, while the video stream is broadcast to the expert engineer, the shop-floor technician is capable of making annotations so as to ease the engineer to get a better understanding of the machine tool malfunction. As soon as the engineer is fully aware of the malfunction, they can create on-demand AR content, which will be projected in the shop-floor technician’s FOV (Field Of View). It is stressed out that, during the live-video casting from the technician, the device is also capable to 3D scan the space around the technician. As such, the algorithm running in the background is fully aware of the user’s surroundings. Therefore, the expert engineer can register 3D content in the technician’s FOV more easily and more successfully. The framework architecture can be realized through a platform, where the shop-floor technicians can connect and request assistance from expert engineers in real-time. From the expert engineer point-of-view, the platform serves as a tool enabling the expert to guide the shop-floor technician with the use of simple 3D tools. Therefore, it is of great importance to initially present the platform architecture. In Figure 2, the architecture of the proposed framework is depicted. From the figure, the structure of the Cloud Platform is visible and consists of services and the toolbox.

The most important component of the platform architecture is the online connection and user interconnection framework. More specifically, the platform acts as a cloud platform, where users, either expert engineers or field/shop-floor technicians, can connect to and, most importantly, they can exchange digital information. It is stressed out that there are no limitations in the type and volume of the exchanged information. Additionally, the user experience can be further improved by the support of live teleconference of the involved actors. In order to enhance the communication experience between the involved parties, the framework supports a teleconference feature. When this feature is enabled, then the user is capable of having a live video feed from the expert engineer, projected in his FoV. As such, it is expected that the communication between the two parties is facilitated, as the technician can perform MRO simultaneously. As far as the information exchange is concerned, the platform is connected with a cloud database. In the database, there is dedicated space where certain 3D geometries are saved. These geometries are imported in the AR scene only by the expert engineers. The collection of these tools can be realized as a digital toolbox of the most common tools used in MRO. However, due to increasing machine tool complexity, specialty tools might be needed. As such, the expert engineer is capable of adding a digital copy of the specialty tool through the “Create tool” Graphical User Interface (GUI).

In Figure 3, the GUI for joining an online maintenance session is depicted. On the top left corner of the GUI, the user through the provided dialog can select a session to join. In case the session has reached its end, then the user has to leave the session. On the lower left corner, the provided buttons assist the master engineer to retrieve from the Cloud platform the required tools by performing a “drag and drop” action to position them in the technician’s FOV. However, if the desired tool does not exist in the Cloud toolbox, then the engineer is capable of uploading a 3D geometry. More specifically, in Figure 3, the “plus” icon when hit triggers the “Create Tool” function, so that the engineer adds the geometry in a suitable filetype along with relevant info and a description of the tool. The filetypes accepted by the Cloud Database have been limited to the filetypes accepted by the AR tool. Therefore, since the framework is developed in the Unity 3D game engine, the supported model file formats for 3D geometries are, .fbx, .dae (Collada), .3ds, .dxf, .obj. Then the algorithm running in the background is responsible for uploading the geometry on the Cloud Database as well as to create a new record in the SQL Database. The SQL database is created and used in an ontological manner, so that a smart-searching algorithm can automatically generate suggestions to the engineer during the AR scene creation phase. In Figure 4, the set of the initial tools used for populating the Cloud Toolbox is presented. For this manner, the most common technical tools have been utilized.

3.2. Cloud Platform The proposed remote maintenance service is developed as a cloud-based platform. The Cloud platform enables the communication of the shop-floor/field technicians and the expert engineer as well as the distribution of information between the involved parties. The Cloud platform is comprised of two domains: (i) the communication domain and (ii) the data storage domain. As discussed in the previous paragraphs, the communication domain is used for matching shop floor technicians for expert engineers and for establishing the communication channels between them. On the other hand, the data storage domain in essence is the Cloud Database where all the data are saved in SQL tables as well as the 3D geometries uploaded. 3.3. Remote Maintenance Sessions As long as the shop-floor/field technicians identify a malfunctioning machine, then, with the use of a mobile device, they use the cloud platform so as to communicate with an expert engineer. In the cloud platform, this form of communication is realized through the creation of online sessions. For each session created, a communication domain is created. Upon the establishment of the communication domain, the two parties are capable of bilaterally exchanging information. 4. Implementation As far as software is concerned, the first step for the development of the framework is the development of the cloud platform. From a hardware point of view, the architecture of the framework has been developed so that a variety of platforms can be supported including PCs, HMDs, and handheld devices. In its current form, the framework has been developed so that the technician uses a Microsoft Hololens HMD and the expert engineer operates from a desktop PC. For the platform, a remote server is utilized. The server has a dedicated domain for the storage of files, where 3D geometries are also uploaded. The supported filetypes for the 3D geometries are .fbx, .dae (Collada), .3ds, .dxf, .obj, and .skp. Besides the 3D geometries, the storage domain supports any form of files, providing enough degrees of freedom to the engineers in exchanging files with the technicians during a remote maintenance session.

As far as the development of the software is concerned, the Unity 3D game engine has been utilized. Regarding the AR application for the technicians, Unity 3D in conjunction with the Vuforia and MRTK (Mixed Reality Tool Kit) libraries were used. The Vuforia library is useful for the creation and handling of the AR content, whereas the MRTK library supports advanced user interaction functionalities for the Microsoft Hololens HMD. Regarding the expert engineer application for the remote support, a Windows Form Application has been developed. In order to enable the communication of the two applications, the integrated Unity UNet API (Application Programming Interface) was used. Since the above-mentioned API is targeted for the creation of multiplayer games, several functionalities had to be adjusted so as to align with the requirements of the framework. UNet can be realized through different layers. In Figure 5, the functionalities per layer of the UNet are presented.

As far as the communication module is concerned and, more specifically, the teleconference functionality, a variety of solutions has been examined, including Skype, Agora.io [34], GIGA Video Streamer [35], and the development of a custom teleconference tool. Among the available solutions, the latter was selected, i.e., the development of a custom teleconference interface. To begin with, the integration of Skype, although it is an appealing solution, and the quality of the service itself is sufficient. It required that it was run as an external application on the Microsoft HoloLens HMD, which had a negative effect on the performance of the app, as well as being unpleasant for the end-users to switch over to two different applications including one for the communication and one for the AR remote support. On the other hand, the Agora.io and the GIGA Video Streamer are very capable SDKs (Software Development Kit) that could be integrated in the proposed framework. However, these two SDKs are not free. Therefore, the development of a custom module based on the UNet API, discussed above, was the only choice satisfying the framework’s requirements. Concretely, from the expert engineer side, the video stream from the camera and the microphone of the PC was captured and transmitted to the Microsoft Hololens HMD. Form the technician side, a pop-up window appears where the technician can view the expert engineer and, through the embedded speakers of the HMD, to hear the expert’s voice.

From a hardware point of view, for the on-site technician, a Microsoft HoloLens [36] device has been utilized in order to take advantage of its four environment-understanding cameras and the mixed reality capture in terms of sensors, whereas, for the development of the framework and the validation experiments, a laptop PC has been utilized. More specifically, the laptop PC is equipped with an Intel core i7 processor clocked at 2.20 GHz, 8 GB DDR4 RAM, and a NVIDIA GeForce 1060 GPU with 6 GB dedicated memory.

5. Case Study and Results

The applicability of the proposed framework is tested and validated in-vitro in a laboratory-based machine shop as well as in a real-life industrial scenario. The industrial scenario involves the machine tools used in an existing machine-shop. Taking into consideration the technological level of the machines, this use case is ideal since the machine shop consists of machines coming from different technological eras. Thus, it is of great importance to examine the actual impact of the proposed framework in varying machine tools. In Figure 6, the validation of the framework in the real-life scenario are presented. More specifically, in Figure 6a, the malfunctioning machine is visible along with the shop-floor technician who is wearing the HoloLens HMD in order to perform the maintenance operations, as instructed by the expert engineer. Similarly, in Figure 6b, another engineer is testing the developed framework. It is stressed out that, in the background, the expert engineers using the laptop PC in order to communicate with the technician are visible.

Similarly, in Figure 7, the practical implementation and validation of the framework in the lab-based scenario is presented. Concretely, in Figure 7a, the expert engineer, from his desktop, can view the user’s FoV, which contains the malfunctioning machine tool. As can be seen, the spatial recognition algorithm of the Microsoft HoloLens device has recognized the technician’s physical environment. Thus, the engineer based on that feature can place the 3D augmentations on the user’s physical environment. On the other hand, in Figure 7b, the shop-floor technician is presented. What is worth noticing is that, in the top right corner of that figure, a snapshot from the HoloLens head up display is presented. The shop floor technician, after requesting assistance from the engineer, the connection has been established and a video teleconference has begun. Therefore, a communication window has opened in the technician’s FoV, where the audio and video stream from the engineer are received and displayed.

Validation Strategy—Design of Experiments

In the following paragraphs, the steps followed for the validation of the proposed framework in the real-life industrial scenario will be presented. Since maintenance is a very important aspect regarding the flawless and continuous operation of manufacturing systems, the validation strategy should be carefully designed and also should provide a quantified result of the impact. Therefore, for each of the machine-tools examined in this process, two basic metrics will be used. According to the guidelines given by Chryssolouris in Reference [21] regarding the mathematical programming of manufacturing systems, the operating and maintenance costs for a machine can be calculated by the equation below.

Operating and Maintenance Costs= ∑t=1T(P/F,I,t)∑i=1N_{mit xit}

where(P/F,I,t) ≡Present a worthy interest factor for discrete compounding when the discount rate is I% per period and the discount interval is t periods._mit≡The cost of operating and maintaining a machine of type i during period t. Is the number of machines, on one hand, in the work center i for period t.

In order to quantify the outcome of the validation process for the proposed framework, each of the above-mentioned values will be calculated in order to conclude on the efficiency of the real-time remote maintenance.

Therefore, the cost for operation and maintenance for a CNC milling machine has been calculated, based on data derived from a real-life machine-shop. The cost of operation and maintenance has been calculated on an annual basis for time horizon of five years, and the results are depicted in Figure 8. Additionally, in Figure 8, a prediction of the cost is made for the oncoming five years based on statistical regression, presented by the red dot trendline. It is stressed out that, due to the non-linear relation between the individual costs of operation and maintenance of machine tools during their lifecycle [37], a 2nd degree polynomial has been calculated. The same calculations were made for the cost estimation based on the proposed methodology and, similarly, a prediction is made for future maintenance costs. As a result, with the proposed methodology, a reduction of approximately 100 k euros is feasible in a time span of 10 years, which indicates that the framework could eventually be implemented in manufacturing systems and improve their operation.

Besides the cost for operating and maintenance, another metric has been calculated and compared regarding the downtime of the above-mentioned machine. In manufacturing systems, time resources are equally important for the financial resources. Therefore, a maintenance scenario has been investigated. For this repair scenario, the machine tool breaks down at an unpredicted time. At that time, the production process for this machine stops and the expert engineer is called to come on-site, inspect the machine, and repair it. The same scenario has also been investigated, but instead of requesting an expert engineer to visit the machine shop, a remote support session is established, and the repair is conducted by the technicians available in the machine shop. In Figure 8, the distribution of time per maintenance task is displayed for each of the scenarios examined. From Figure 6, it is undeniable that a significant amount of time can be saved with the adoption of the proposed framework, as the time for travelling and inspection are considerably longer, which affects the overall time the machine tool has been off the production schedule. Through the use of Figure 9, it is intended to reflect the difference of time distribution between the two scenarios examined during the validation process of the proposed framework. Therefore, from Figure 9, it can be concluded that, in the first scenario, i.e., the “AS-IS-SITUATION”, the most time spent was on inspection and travelling, as the expert engineer had to travel on site and physically inspect the malfunctioning machine. On the contrary, in the second scenario, i.e., “PROPOSED METHODOLOGY”, a totally different time distribution has been observed with the most time spent on activities relative to the acquisition of spare parts for the machine repair. In addition to that, it can be concluded that the actual repair time, which is the black portion of the pie graph, in the second scenario is bigger. This observation can be considered normal as, in the second scenario, the maintenance operations have been carried out by a field technician, whereas, in the first scenario, the same operations have been carried out by an expert engineer with solid practical experience on such machines.

In Figure 10, a side-by-side comparison of the specific times for the two maintenance scenarios is presented. It is stressed that, with the adoption of the proposed framework, the time for inspection of the machine can be shorter as no travelling time is involved in order to reduce MTTR. However, due to the inexperience of the technician carrying out the maintenance activities, the actual repair operation time is longer, as the expert engineer had to perform additional checks in order to ensure that the repair was successful.

6. Discussion

The outcome of the current research work, is a multi-sided, multi-device application, enabling the remote maintenance of complex machine tools under the framework of Industry 4.0. The term multi-sided is used in order to describe that multiple operators can connect to the platform simultaneously and discuss on a malfunction that has occurred unexpectedly. On the other hand, the term multi-device, is used in order to describe the compatibility of the developed framework in a variety of devices. It is stressed out that the most common devices used for AR implementations are HDMs and handheld devices, including smart-phones and tablets. However, since the scope of the current research was mainly focused on the provision of a totally marker-less AR solution, the use of Microsoft’s HoloLens HMD is preferred, since the device supports spatial recognition, which, in turn, has facilitated the development of the framework. It is stressed out that the use of the HMD is initially intended only for the shop-floor technician as a means to capture their surrounding and, based on that, the expert engineer is capable of placing/registering the virtual content on the user’s physical environment. The development of the framework was performed in a set of consecutive stages. The first stage was based on the conceptualization of the system’s architecture, the discussion of the modules comprising the architecture, and the tools to be used for the final development. Then, in stage two, the basic modules were developed, utilizing the software described in Section 4. As soon as the alpha version of the framework was released, the framework has been internally tested in vitro, in the lab-based machine shop described in Section 5. During this stage, colleague engineers have been asked to test the functionalities of the developed application either from the side of the expert engineer or from the side of the shop-floor technician aimed at gathering insightful feedback. Then, based on the feedback gathered during the previous stage, the framework has been improved in certain areas, regarding the layout of the GUIs and the functioning of the communication services. Then the beta version of the developed framework has been prepared and tested with the industrial partner in their premises.

Throughout the development and validation stages, meaningful feedback has been gathered. The main strengths of the proposed framework can be summarized to the minimum development effort required by the expert engineer. More specifically, the remote maintenance frameworks are based on the development of AR scenes, which are then implemented on an AR-ready device. Therefore, this requires advanced development skills and a certain amount of preparation time. On the contrary with the proposed framework, an expert engineer is capable of creating AR content on the user’s FoV with the use of simple Drag-and-Drop actions. As such, no special skills are required and the time for AR content preparation and implementation is eliminated. Another strength of this framework is the ability to create online sessions. The online sessions are based on the Unity 3D multiplayer API offering the remote connection of multiple users. As discussed in previous paragraphs, the developed application is compatible with most devices. Thus, there is no practical limitation on the use of equipment. In comparison with other approaches, presented in References [3,4,5,6], the functionality of remote connectivity combined with the broad compatibility of the app further extends mobility of the user, i.e., the shop-floor technician in an industrial environment. More specifically, in Reference [4], the technicians have to carry a laptop computer along with a USB web camera in order to establish communication with the expert engineer and proceed with the maintenance operations. The frameworks presented in the above-mentioned publications are using frame markers for the registration of the content on the physical environment. With that said, the performance of the system relies heavily on the quality of the camera used for the detection of the marker. The user’s mobility is limited, as the camera has to maintain visibility of the marker at all times. Otherwise, the augmentations cannot be registered and updated as the user moves around. In addition to that, the proposed framework, since it utilizes newer equipment with increased computational power, the use of more vivid/realistic virtual objects is feasible. Not only that, but in previous approaches, the augmentations are limited to the display of simple schematics, e.g., arrows, text boxes, or indicate the component of interest via a wireframe model. On the contrary, the current approach can recognize the objects by the utilization of Microsoft HoloLens.

7. Conclusions In this research work, the most recent and relevant publications regarding remote maintenance support have been investigated. Through the literature review process, it became apparent that, although there are numerous approaches contributing in the field of remote maintenance support, there is a small amount of research work done on the provision of real-time support. Therefore, an opportunity for a new research work has been identified. Concretely, the design and development of a framework for the real-time remote maintenance and repair operations based on AR has been presented in detail. The framework aimed at creating interactive, intuitive, and collaborative communication channels between the shop-floor technicians and the expert engineers. Furthermore, the developed application was tested in a real-life industrial scenario in order to verify that all the functional requirements are met. The results indicate that the present research work has managed to successfully address the functional requirements discussed during the modelling phase of the problem and has succeeded in delivering a zero-time content authoring AR tool, which further minimizes the MTTR. However, there is fertile ground for future improvement. As discussed in the literature review section, one of the key elements consisting of an AR application is the content authorization methodology. Towards that end, future work could focus on integrating the developed framework to an automated methodology for content authoring, which could considerably reduce the effort and the skills needed from engineers in creating AR content.

Figure 1. Flowchart of the proposed system.

Figure 2. Proposed system architecture and steps sequence.

Figure 3. GUI for online maintenance session.

Figure 4. 3D CAD geometries of tools for AR instructions.

Figure 5. Functionalities per Layer of Unity 3D UNet API.

Figure 6.(a) Validation of the proposed framework by shop-floor technician; (b) Another technician tests the framework.

Figure 7.(a) Expert engineer visualizes video stream from technician; (b) Technician is performing maintenance tasks based on expert's instructions.

Figure 8. Cost of operation and maintenance analysis.

Figure 9. Distribution of time AS-IS situation vs. Proposed Methodology.

Figure 10. Time per maintenance action AS-IS situation vs. Proposed Methodology.

Author Contributions

All authors participated in the modelling of the research project. More specifically, D.M. Supervisor, V.S. Development and methodology, J.A. Research, conceptualization, and writing. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Acknowledgments

The H2020 EC funded project "Balancing Human and automation levels for the manufacturing workplaces of the future-MANUWORK" (GA No: 723711) partially supported this work.

Conflicts of Interest

The authors declare no conflict of interest.

References

1. Mourtzis, D.; Vlachou, E.; Milas, N.; Xanthopoulos, N. A Cloud-based Approach for Maintenance of Machine Tools and Equipment Based on Shop-floor Monitoring. In Proceedings of the 48th CIRP Conference on Manufacturing Systems, Ischia, Italy, 24-26 June 2015; pp. 655-660.

2. Predictive Maintenance: Taking Pro-Active Measures Based on Advanced Data Analytics to Predict and Avoid Machine Failure. Available online: https://www2.deloitte.com/content/dam/Deloitte/de/Documents/deloitte-analytics/Deloitte_Predictive-Maintenance_PositionPaper.pdf (accessed on 2 January 2020).

3. Wang, W.; Peter, W.T.; Lee, J. Remote machine maintenance system through Internet and mobile communication. Int. J. Adv. Manuf. Technol. 2007, 31, 783-789.

4. Wang, J.; Feng, Y.; Zeng, C.; Li, S. An augmented reality based system for remote collaborative maintenance instruction of complex products. In Proceedings of the 2014 IEEE International Conference on Automation Science and Engineering (CASE), Taipei, Taiwan, 18-22 August 2014; pp. 309-314.

5. Masoni, R.; Ferrise, F.; Bordegoni, M.; Gattullo, M.; Uva, A.E.; Fiorentino, M.; Carrabba, E.; Di Donato, M. Supporting remote maintenance in industry 4.0 through augmented reality. Procedia Manuf. 2017, 11, 1296-1302.

6. Ong, S.K.; Zhu, J. A novel maintenance system for equipment serviceability improvement. CIRP Ann. 2013, 62, 39-42.

7. Mourtzis, D.; Vlachou, A.; Zogopoulos, V. Cloud-based augmented reality remote maintenance through shop-floor monitoring: A product-service system approach. J. Manuf. Sci. Eng. 2017, 139, 061011.

8. Mourtzis, D.; Zogopoulos, V.; Katagis, I.; Lagios, P. Augmented Reality based Visualization of CAM Instructions towards Industry 4.0 paradigm: A CNC Bending Machine case study. In Proceedings of the 28th CIRP Design Conference 2018, Nantes, France, 23-25 May 2018; pp. 368-373.

9. Cardoso, S.F.L.; Mariano, Q.M.C.F.; Zorzal, R.E. A Survey of Industrial Augmented Reality. Comput. Ind. Eng. 2020, 139, 106159.

10. Palmarini, R.; Erkoyuncu, A.J.; Roy, R.; Torabmostaedi, H. A systematic review of augmented reality applications in maintenance. Robot. Comput. Integr. Manuf. 2018, 49, 215-228.

11. Fernández del Amo, I.; Erkoyuncu, A.J.; Roy, R.; Palmarini, R.; Onoufriou, D. A systematic review of Augmented Reality content-related techniques for knowledge transfer in maintenance applications. Comput. Ind. 2018, 103, 47-71.

12. Berglund, F.Å.; Gong, L.; Li, D. Testing and validating Extended Reality (xR) technologies in manufacturing. In Proceedings of the 8th Swedish Production Symposium (SPS 2018), Stokholm, Sweeden, 16-18 May 2018; pp. 31-38.

13. What Is Extended Reality Technology? A Simple Explanation for Anyone. Available online: https://www.forbes.com/sites/bernardmarr/2019/08/12/what-is-extended-reality-technology-a-simple-explanation-for-anyone/#69b6421f7249 (accessed on 16 February 2020).

14. Mourtzis, D. Simulation in the design and operation of manufacturing systems: State of the art and new trends. Int. J. Prod. Res. 2019, 1-23.

15. 4 Types of Maintenance Strategy, Which one to Choose? Available online: https://new.abb.com/medium-voltage/service/maintenance/feature-articles/4-types-of-maintenance-strategy-which-one-to-choose (accessed on 2 January 2020).

16. Mourtzis, D.; Doukas, M. Design and Planning of Manufacturing Networks for Mass Customisation and Personalisation: Challenges and Outlook. In Proceedings of the 2nd CIRP Robust Manufacturing Conference (RoMac 2014), Bremen, Germany, 7-9 July 2014; pp. 1-13.

17. Xun, X. From cloud computing to cloud manufacturing. Robot. Comput. Integr. Manuf. 2012, 28, 75-86.

18. Mourtzis, D.; Vlachou, A. A cloud-based cyber-physical system for adaptive shop-floor scheduling and condition-based maintenance. J. Manuf. Syst. 2018, 47, 179-198.

19. Azuma, T.R. A Survey of Augmented Reality. Presence Teleoperators Virtual Environ. 1997, 6, 355-385.

20. Azuma, T.R. The Most Important Challenge Facing Augmenting reality. Presence Teleoperators Virtual Environ. 2017, 25, 234-238.

21. Mourtzis, D.; Zogopoulos, B.; Vlachou, E. Augmented Reality Application to Support Remote Maintenance as a Service in the Robotics Industry. In Proceedings of the 50th CIRP conference on Manufacturing Systems, Taichung City, Taiwan, 3-5 May 2017; pp. 46-51.

22. Henderson, S.; Feiner, S. Exploring the Benefits of Augmented Reality Documentation for Maintenance and Repair. IEEE Trans. Vis. Comput. Graph. 2011, 17, 1355-1368.

23. Mourtzis, D.; Angelopoulos, J.; Boli, N. Maintenance assistance application of Engineering to Order manufacturing equipment: A Product Service System (PSS) approach. In Proceedings of the 16th IFAC Symposium on Information Control Problems in Manufacturing INCOM 2018, Bergamo, Italy, 11-13 June 2018; pp. 217-222.

24. Mourtzis, D.; Xanthi, F.; Zogopoulos, V. An Adaptive Framework for Augmented Reality Instructions Considering Workforce Skill. In Proceedings of the 52nd CIRP Conference on Manufacturing Systems (CMS), Ljubljana, Slovenia, 12-14 June 2019; pp. 363-368.

25. Siew, Y.C.; Ong, K.S.; Nee, C.Y.A. A practical augmented reality-assisted maintenance system framework for adaptive user support. Robot. Comput. Integr. Manuf. 2019, 59, 115-129.

26. Chryssolouris, G. Manufacturing Systems: Theory and Practice, 2nd ed.; Springer: New York, NY, USA, 2006.

27. Chen, J.; Gusikhin, O.; Finkenstaedt, W.; Liu, Y. Maintenance, Repair, and Operations Parts Inventory Management in the Era of Industry 4.0. In Proceedings of the 9th IFAC Conference on Manufacturing Modelling, Management and Control MIM 2019, Berlin, Germany, 28-30 August 2019; pp. 171-176.

28. Bokrantz, J.; Skoogh, A.; Berlin, C.; Wuest, T.; Stahre, J. Smart Maintenance: An empirically grounded conceptualization. Int. J. Prod. Econ. 2019, 107534.

29. Sahal, R.; Breslin, G.J.; Ali, I.M. Big data and stream processing platforms for Industry 4.0 requirements mapping for a predictive maintenance use case. J. Manuf. Syst. 2020, 54, 138-151.

30. Ruiz-Sarmiento, J.R.; Monroy, J.; Moreno, F.A.; Galindo, C.; Bonelo, J.M.; Gonzalez-Jimenez, J. A predictive model for the maintenance of industrial machinery in the context of industry 4.0. Eng. Appl. Artif. Intell. 2020, 87, 103289.

31. Bousdekis, A.; Lepenioti, K.; Apostolou, D.; Mentzas, G. Decision Making in Predictive Maintenance: Literature Review and Research Agenda for Industry 4.0. IFAC-PapersOnLine 2019, 52, 607-612.

32. Efthymiou, K.; Papakostas, N.; Mourtzis, D.; Chryssolouris, G. On a Predictive Maintenance Platform for Production Systems. Procedia CIRP 2012, 3, 221-226.

33. Nikolakis, N.; Papavasileiou, A.; Dimoulas, K.; Bourmpouchakis, K.; Makris, S. On a versatile scheduling concept of maintenance activities for increased availability of production resources. Procedia CIRP 2018, 78, 172-177.

34. Agora.io. Available online: https://www.agora.io/en/ (accessed on 26 February 2020).

35. GIGA Video Streamer. Available online: https://assetstore.unity.com/packages/tools/video/giga-video-streamer-125456?_ga=2.75845011.740357347.1582886552-1421986738.1578039987 (accessed on 26 February 2020).

36. Microsoft HoloLens Specification. Available online: https://www.windowscentral.com/hololens-hardware-specs (accessed on 26 February 2020).

37. Bengtsson, M.; Kurdve, M. Machining Equipment Life Cycle Costing Model with Dynamic Maintenance Cost. In Proceedings of the 23rd CIRP Conference on Life Cycle Engineering, Berlin, Germany, 22-24 May 2016; pp. 102-107.