(1) Real vehicle experiment (Experiment 1)

In regard to the three levels of motion sickness, there were 85 cases of 1 (No sickness), 31 cases of 2 (Slight Sickness), and 1 case of 3 (High Sickness). The transition of motion sickness for each time interval is shown in Figure 3 on Experiment 1. In the plots, the horizontal axis indicates the time interval, and the vertical axis indicates the average of motion sickness in all participants. Significant differences were seen in one-way repeated measures ANOVA with time interval (F(12,96) = 2.216, p = 0.017, and η2 = 0.109). Shaffer post-hoc paired test with time factor was conducted but there were no significant differences.

(2) Stereoscopic visual simulation

In regard to the three levels of motion sickness, there were 456 cases of 1 (No sickness), 196 cases of 2 (Slight Sickness), and 24 cases of 3 (High Sickness). The transition for each time interval is shown in Fig. 3 on Experiment 2. In the plots, the horizontal axis indicates the time interval, and the vertical axis indicates the average of motion sickness in all participants. Significant differences were seen in one-way repeated measures ANOVA with time interval (F(12, 612) = 11.736, p < 0.001, and η2 = 0.090). Shaffer post-hoc paired test with time factor showed significant differences in Time 0-10 sec (T1)–T9, T10, T13, T2–T7, T8, T9, T10, T11, T12, T13, T3–T7, T8, T9, T10, T13, T4–T9, T10, T13, T5–T10, T13, T6–T10, and T13 (Table A1 and Table A2).

To judge the correspondence with the subjective indicators, we calculated the mean value of each indicator, using 10 s as an interval. Additionally, the objective indicators used were EDA, pulse rate, and LF/HF, and each participant was normalized from −1 by MinMaxScaler to see the variation.

For the model to predict motion sickness, a deep learning model based on one-dimensional convolutional neural network (1DCNN) was used. The prediction was performed using a recurrent neural network (RNN); however, as the 1DCNN model had higher precision, we reported the 1DCNN model results. The model had three layers–convolution, pooling, and dropout–and two fully connected layers. In the first fully connected layer, we input the predicted value for motion sickness in the previous interval. Leaky ReLU was used as the activation function, and Softmax was used as the output layer. The categorical cross-entropy loss function and Adam optimization algorithm were used. An image of the model is shown in Figure 7.

Data with missing values were excluded, so datasets were for 9 people in Experiment 1 and 52 in Experiment 2 and the data were divided every 10 s to predict motion sickness levels. For objective indicators, EDA, pulse rate, and LF/HF were used. Each indicator was normalized in the MinMaxScaler as −1–1 for each participant. When predicting motion sickness, as there were few cases of 3 (High Sickness), it was considered to be 0 (No sickness) as 1 (No Sickness), and 1 (Sickness) as 2 (Slight Sickness) and 3 (High Sickness). So that the data of Experiment 1 is based on 85 cases of 0, 32 cases of 1, and the data of Experiment 2 is based on 456 cases of 0, 220 cases of 1.

For accuracy verification, the average value of the results in four-fold cross-validation randomly in person and time was used. A summary of the confusion matrix of the four sessions is shown in Table II. The mean accuracy of the four sessions was 0.833, with the F1-score for “Nothing” being 0.883 and F1-score for “Sickness” being 0.684.

For accuracy verification, the average value of the results in the four-fold cross-validation randomly in person and time was used. A summary of the confusion matrix of the four sessions is shown in Table III. The mean accuracy for the four sessions was 0.844, with the F1-score for “Nothing” being 0.885 and F1-score for “Sickness” being 0.745.

To validate the effect of expanding the training data from Experiment 2, a four-fold cross-validation was performed on Experiment 1 data. The data from the stereoscopic video experiment was also added to the training data (Figure 8). A summary of the confusion matrix for the four sessions is shown in Table IV. The mean accuracy for the four sessions was 0.866, with the F1-score for “Nothing” being 0.896 and F1-score for “Sickness” being 0.754.

It is known that the user experience changes with viewing time in VR [22]. Thus, the effect of time and somatosensory information was analyzed by comparing Experiment 1 and Experiment 2. The results of the actual vehicle experiment demonstrated a tendency for motion sickness to increase, with a peak at the 110-s interval. The results of the stereoscopic visual experiment demonstrated a tendency for motion sickness to increase, with a peak at the 100- and 130-s intervals. Where each experiment demonstrated an increasing trend, the fact that the position of the peak differs is thought to be due to the fact that the degree of sensory conflict differs in the various cases of vehicle sickness and simulation sickness. Particularly, it is affected by the presence or absence of somatosensory information being presented.

Compared to the real vehicle experiment, each indicator in the stereoscopic visual experiment tended to vary by only a small degree. In the results of ANOVA for the real vehicle experiment, a large effect size was seen for EDA and HR, whereas a small effect size was seen for LF/HF bands. Furthermore, in the results of ANOVA for the stereoscopic visual experiment, a small degree of change was seen for all participants. This suggests that the response of the objective index was greater for the real vehicle than for the stereoscopic visuals.

If we compare the peak positions for the objective and subjective indicators, we can see that the peak position does not match for any participants. This suggests that evaluating the response to specific objective indicators and subjective indicators is difficult.

The cross-validation accuracy of the real vehicle test was 0.833, and the cross-validation accuracy of the stereoscopic visual experiment was 0.844. When the learning data used in the stereoscopic visual experiment was extended, the cross-validation accuracy was 0.866. In a previous study, predicting the motion sickness levels through a self-organizing neural fuzzy inference network (SONFIN) using EEG signals, an overall accuracy of about 82% through experiments was achieved [23]. When predicting motion sickness levels through a three-dimensional convolutional neural network (3DCNN) using 3D image information, the correlation between a simulator sickness questionnaire (SSQ) [24] and prediction score was 0.845 [25]. When predicting the VR sickness levels through a Deep Long Short Term Memory Model (LSTM) using posture instability signal, the correlation between a SSQ and prediction score was 0.89 [26]. These results suggest that the accuracy for predicting motion sickness within the respective tests could be predicted with constant accuracy. Therefore, it is thought that the objective indicators obtained in this study can contribute to predicting motion sickness to some extent. Additionally, by adding the stereoscopic visual simulation data to the real vehicle experiment, it was confirmed that the accuracy for predicting motion sickness in the real vehicle experiment could be improved. Thus, even when data is obtained under different environments, the data capture common responses within the scope of sensory conflict.