Cloud Hosting

This section describes the data collection process. It starts by describing the design of the experiment and the setup, including a description of the acquisition and processing elements of the methodology.

The experiment was conducted using a machine in which multiple operators interacted through the same HMI to perform a mixture creation task. In this scenario, an industrial mixing machine from the food sector was utilized, which offers the advantage of being regularly used throughout the day by several users across two working shifts. Each time a mixture was ordered, the operator carried out a series of individual interactions with the HMI. These interactions were related to adjusting various parameters, including additive quantity, mixture type, and the use of containers. These parameters directly influenced the properties of the final product.

Users interacted with the machine through a mobile app that was specifically designed for the experiment. Operators accessed the app by scanning a QR code, after which they proceeded to select the required mixture. The captured interactions included two key components: (i) the order and sequence of steps the user followed, and (ii) the time interval in which the user interacted with the machine.

Twenty-seven volunteer operators, aged between 23 and 45 years, participated in the experiment. Each operator granted formal consent to have their daily interactions recorded through the app. In total, 10,608 interactions were captured over a period of 151 days. All data was anonymized and does not contain sensitive user information.

Figure1 illustrates the methodology for data acquisition, which begins with the preparation stage. This stage encompasses two steps: firstly, the user interface (UI) is formally described using a user interface description language (UIDL), which consists of a mark-up language that describes the entire HMI12. In this study, the JSON format was employed to represent each visual element in the HMI, with each element assigned a unique alphanumeric identifier.To provide an example of the UIDL utilized in this study, Fig.2 displays a representation of the UI alongside its corresponding UIDL.

Data acquisition methodology.

UIDL JSON description example of a UI.

The HMI was implemented using Next.js, a React framework and Chakra UI. A dedicated function was created to programmatically generate the HMI using the user interface descriptor. The interface is designed to be responsive and can be used on tactile devices.

Next, the interaction process representation required to prepare a mixture in the machine is described as a Finite State Machine (FSM), which is a model consisting of states, transitions, and inputs used to represent processes or systems. In this process, the user adjusts the parameters of a mixture until the values are considered correct (Fig.3).

Interaction process representation (FSM).

During the active phase of the experiment, when users access the machine using the application, a non-intrusive layer captures the interactions and stores them in a database (capture interactions). The information captured includes the user identity, the timestamp of the interaction in EPOCH format, and the identification of the interacted element (store raw interactions) (see Table1). Once this information is collected, the data processing step generates the sequences.

The goal of this step of the methodology is to generate valid sequences of interactions for each user. Perer & Wang13 define a sequence of events (E=langle {e}_{1},{e}_{2},...,{e}_{m}rangle ) (ei D) as an ordered list of events ei, where D is a set of events known and the order is defined by i. This means that the event ei occurs before the event ei+1. Additionally, in this process is considered that E must contain at least two events e to be accepted as a sequence9.

Using this definition and taking as input the raw interactions, it is possible to define valid interaction sequences as ({s}_{i}=left[{e}_{begin},{e}_{1}^{i},ldots ,{e}_{k}^{i},{e}_{end}right]) where si is a set of events and:

The events ebegin and eend are known, determining the beginning and the ending of the interaction sequence

The variable l determines the length of the interaction sequence and its value should be >=2

The sequences are extracted using the Valid sequences extractor algorithm presented by Reguera-Bakhache et al.9. As demonstrated in the FSM (Fig.3), the interaction process initializes when an interaction occurs in any of the elements that allow the parametrization of the mixture and finalizes when the user clicks the button BTN1OK.

From the 10,608 interactions recorded, 1358 valid sequence interactions were generated. The composition of each interaction sequence is described in the following section.

See original here:

Structured dataset of human-machine interactions enabling adaptive ... - Nature.com