Parkinson's Disease (PD) is a severe and long-term neurodegenerative disorder of central nervous system. It is characterized by Freezing of Gait (FOG) as one of the major symptoms, with an increasing incidence among elderly individuals. Existing FOG assessments through clinical examinations and scales by clinicians are subjective and unreliable. The quality of PD care is gradually being improved by sensor-based evaluation technologies that measure the symptoms objectively and enhance PD diagnosis and therapeutic intervention. An objective system for assessing FOG in patients with Parkinson's disease is proposed in this study. The proposed system uses multiple motion capture sensors to provide precise and reliable data to healthcare professionals regarding FOG and the progression of the disease. Seven healthy subjects trained to simulate the FOG condition were volunteered for the study. They are instructed to perform walking in a FOG-provoking path with several turning events. The proposed objective method visualizes the number of FOG occurrences and their duration. Experimental findings demonstrate the accuracy of the proposed system for FOG prediction in the conducted experiments. In the future, the reliability of the method will be studied with significantly larger datasets.
Cadence; Edge computing; Festation; Freezing of gait; Gait analysis; Kinect sensor, Real-time motion capture; Rehabilitation
Parkinson’s Disease (PD) is a progressive neurodegenerative disorder associated with severe motor and non-motor complications [1], that affects millions of people worldwide especially elderly people. Due to the breakdown of certain brain tissues, the motor symptoms of PD include Freezing Of Gait (FOG), gait imbalance, postural instability, bradykinesia, rest tremor and rigidity and non-motor manifestations include cognitive impairment, fatigue, mood disorders and so on [1]. The symptoms usually increase with disease progression. Due to the nature of the disease, PD patients exhibit aberrant walking patterns; as a result, they are more likely to trip or feel disoriented. FOG is the one of the major challenging symptoms of PD, especially in advanced stage of the disease.
FOG refers to “the state of being frozen against the intention to walk”. The prevalence rate of FOG is reported as 50.6% [2]. FOG increases the risk of falls, fall-related injuries and loss of independence [3,4]. Stability of an individual affected with PD could be a highly targeted therapeutic goal for the clinicians. For improved symptom management, rehabilitation therapy might be combined with medication or surgery [5,6]. From the perspective of clinicians and researchers, determining the severity of PD is crucial for initiating, titrating, and monitoring treatment; this helps to validate any effective treatment. Parkinsonian Gait is observed in patients in the later stages of the disease, which severely impacts their quality of life as it restricts their motor capabilities, therefore early detection and appropriate treatment can help improve their lives.
The Unified Parkinson's Disease Rating Scale and other specialised questionnaires like Freezing of Gait Questionnaire [7] and New Freezing of Gait Questionnaire [8] are generally utilized among the subjective assessment methods of FOG that are typically inaccurate because of factors like different perspectives of medical professionals, inconsistent questionnaire responses, and patient and carer ignorance of the disease. Meanwhile, objective assessment of PD using video data [9] or video-based sensors is beneficial without the results being affected by recall bias. It is therefore essential to calculate the measurement accuracy of a motion capture system for FOG scoring in order to ensure clinical interpretability before it can be used for FOG assessment in research and clinical practice.
In the context of healthcare industry, cloud computing plays a crucial role in providing remote access to patient information, enabling efficient data collection and analysis for diagnostic and therapeutic decision-making [10]. It is beneficial in the case of Parkinson's disease severity assessment, where qualified doctors may not always be able to reach rural patients due to technological limitations. By utilizing cloud computing, healthcare providers can store and aggregate patient data in a cost-effective manner, allowing for better diagnostic decision-making and remote consultations with specialists [11]. However, by moving computational load towards edge nodes, the latency and dependence on communication infrastructure can be reduced. The seamless collaboration between cloud and edge computing is increasingly important in time-sensitive scenarios. Furthermore, it can enhance the patient experience by providing seamless and uninterrupted access to healthcare services, regardless of geographical limitations [12].
In this context, this study aims to propose a non-invasive edge computing approach using real-time motion data captured by multi-Kinect sensors to assess PD based on motor deficits. A mathematical model is also proposed to detect the occurrence of FOG events, which are periodically exhibited in PD patients, and to classify them. It detects the FOG occurrence based on the significant spatiotemporal and kinematic gait characteristics extracted through the Kinect setup. A web application framework is also proposed to acquire gait data of the subject and provides insightful visualizations to assist the medical experts to provide accurate diagnosis. The early detection of the disease can guarantee proper intervention and improve the patient's quality of life.
The major contributions of this paper are as follows.
This section describes the state-of-the-art related works of utilization of Microsoft Kinect sensor for the assessment techniques of the progression of Current methods for evaluating PD by experienced clinicians or physical therapists rely on the physical abilities of patients and may be prone to subjectivity and unreliability as a result of human error and variations in symptom manifestation. Thus, it can lead to inaccurate assessments and delayed diagnosis, causing a burden on patients who have to undergo frequent visits to the doctor for physical tests.
In recent years, sensor technology has gained popularity for motion-capturing systems leading to better clinical assessment and more efficient therapeutic decisions. However, motor analysis using wearable sensors is more affected by environmental noise and undesirable movements. Hence, an alternative method is adapted to utilize non-intrusive sensors like Microsoft Kinect which captures motion data and enable objective assessment of the progression of the disease. Many researchers extend their research on non-invasive assessment methods for PD using Kinect as it is cost affordable. A study [13] classified, assessed and managed gait disorders of parkinsonian patients from festination of gait to freezing of gait. Another work [14] suggested that the presence of abnormal gait usually indicates cognitive impairment, increasing their risk of falling and getting injured and reducing their quality of life. Thus, it is crucial to understand that how abnormal gait plays an inductive role in the assessment of the disease, and it is measured using laboratory-based and ambulatory-based gait analysis technology.
The accuracy of Microsoft Kinect V2 Depth sensor was measured in [15] using the Time-of-Flight method where active sensors measured the distance of a surface by calculating the round-trip time of a pulse of light. A multi-Kinect setup was suggested that would illustrate variance in depth calculated using the trilateration method. Therefore, it increases the quality of the depth images captured from the sensor. This study demonstrated that a multi-Kinect setup showed a decrease in the average error score.
A study on the correlation between the accuracy and performance of three-dimensional motion capture systems for gait analysis such as the Kinect V2 and the effect of camera viewing angles on their joint angle tracing performance was presented [15]. It evaluated the accuracies of three depth sensors: Azure Kinect, Kinect v2, and Orbbec Astra for tracking kinematic gait patterns during treadmill walking at different camera viewing angles. The results concluded that the Kinect sensor had better Root Mean Squared Error (RMSE) scores when placed at a frontal viewing angle.
A method to detect FOG and reduce false positive results by identifying the difference between standing still/resting and freezing of movement was suggested in [16]. The similarity between resting and FOG episodes is found using the X coordinates of the feet. If the patient is standing still or resting, his/her X coordinate will be constant. Additionally, it highlights the fact that, if FOG is observed, the angle between the ankle and foot joint will be constantly changing. Standing still or resting mode can be differentiated from FOG by finding the number of high-value peaks in the regions of interest. The area with high-value peaks with respect to a threshold is related to FOG and the area without the high-value peaks shows the resting mode.
Another study [17] fused data gathered from multiple Kinect sensors for clinical gait analysis. A five Kinect configuration was suggested that eradicates the problems of inaccurate skeleton poses caused due to occlusions or tracking failures. Two methods were used to estimate states of such discrete-time linear systems: the first being classical Kalman filtering where state disturbances are assumed to be gaussian white noises, secondly, they follow the Set-Membership filtering approach where the noise is a measurement of the multivariate gaussian probability distribution function model. The results proved the latter approach would be the most suitable for gait analysis. A gait recognition method based on a 3D skeleton [18] was proposed which utilized spatiotemporal changes in relative angles among skeletal joints concerning a reference point. It calculated a complete human gait cycle represented by a sequence of joint relative angles between two skeletal joints providing an intuitive representation of the motion patterns. The joint relative angle sequences of each skeletal joint pair represented the significant gait descriptors. The proposed method proved human gait recognition better than that of existing Kinect-based systems.
Another study [19] to monitor the behaviour of subjects' gait cycle and the number of footsteps each time to estimate the occurrence of Freezing of Gait. The inferences are provided on how to consider a foot-off event and foot contact, foot-off-event is observed when the knee angle has decreased. A non-invasive method was suggested in [20] with the help of data mining algorithms to categorize PD cases and controls. Various data mining techniques have been compared to evaluate the most reliable method concerning the changes in the gait descriptors of different people. The results demonstrated the accuracy of predictive analytics of the model.
A study is presented in [21] on which Machine Learning classification model would be more suitable in providing accurate results by comparing the performance of models like decision trees, Bayesian networks, neural networks (multilayer perceptron algorithm), and K-nearest neighbors classifier. The best-performing classifier was found to be a Naïve Bayesian classifier. The classification model achieved a high accuracy of 93.4% with the help of the correlation-based feature selection technique, the significant features include movement and position of the left arm.
From the literature study, the various aspects of implementing a FOG detection system were analysed. The current systems employ only a single depth sensor for FOG detection which has a very limited range. The current systems are designed in a way that only people possessing a considerable amount of technical knowledge can set up and use the system, there is no user-friendly plug-and-play software that has been discussed or ideated. Based on the literature review and the currently existing systems, our proposed solution is to develop a user-friendly solution that uses a multi-Kinect setup to track patient joints and identifies FOG events. Additionally, the solution uses a web application to provide a user-friendly way of generating reports on when and how long FOG events occurred along with informational statistical visualizations.
In this section, a FOG assessment system for PD patients is proposed based on the obtained motion data from the multi-Kinect Sensors setup environment.
Data acquisition
The Microsoft Kinect V2 is a non-wearable motion capture sensor that captures the 3D motion of a person. It consists of a 3D depth sensor, an RGB camera, and multi-array microphones, through which it enables interactions with body movements, voice, and images. The RGB camera can identify 25 human skeletal joints and capture videos at 30fps. In our study, a multi-Kinect setup has been used to improve the accuracy of the camera and trail path coverage area of the sensors. The proposed system optimizes the multi-Kinect setup to widen the field of vision, remove dead spots, and collect data in order to utilize it to record gait and identify significant gait metrics of the subject. The data from 7 subjects are captured. The subjects are healthy without any movement disability. The average age of the subjects is 21.5±1.2. The subjects have different heights and walking speeds and are trained with video on how FOG occurs for PD patients at different stages of the disease. The general information about the subjects is presented in table 1.
Characteristics |
Subjects |
Total Sample, n |
10 |
Gender, M:F |
7:3 |
Age, y |
25.3 ± 4.6 |
Height, cm |
160.78 ± 6.22 |
Weight, Kg |
65.26 ± 9.64 |
Table 1: Characteristics of study population.
Data are presented as mean ± standard deviation.
In clinical practice and research setting, the testing environment plays a crucial role to provoke FOG. In patients with freezing histories, FOG episodes are objectively induced by walking in a small area and performing full rapid turns of 180 degrees. So, in our research, a FOG-provoking path is designed with a straight-line path and a full rapid turning event. The subjects are asked to walk along the path from the beginning point to the ending point and then turn and return to the starting point, named as Walk with Turn (WWT). The multi-Kinect sensors are placed on a table with a height of 1.2 meters. The path should be designed on non-reflective floors. The distance from the starting point to the ending point is 5.9 meters and the straight-line path is set at 3.8 meters away from the sensor, shown in figure 1. The movement data is captured for each subject with WWT experiment. The inclusion criteria for the volunteers were as follows. The participant should not have walking-related, neurological, or orthopedic injuries or illnesses that would affect lower limb movement. Each subject is asked to repeat the experiments if the walking pattern is not like a real FOG phenomenon to make sure the data is captured correctly. For measuring the actual severity of PD, the simulated walking sessions of all the subjects are validated by the clinicians in observing the FOG (Figure 2).
Figure 1: Multi-Kinect setup in a testing environment.
Figure 2: Walk with Turn (WWT) - FOG provoking path.
Data processing
The body's skeletal joints are extracted from the video data captured by the Kinect sensors. Since it attains depth data, each joint is represented by a list of 3D positions (X, Y and Z) and the length of the list is equal to the number of frames of the video. Concerning the position of the Kinect sensors setup as shown in figure 1, the subject is walking along the X axis of the Kinect sensor and thus the X trajectory data is selected for our FOG analysis.
Proposed edge classification framework
This subsection describes the proposed edge-based body motion capture system for PD severity assessment application by detecting the occurrence of FOG events. A three-tier edge architecture, shown in figure 3 is proposed which consists of edge, cloud and application layers.
Figure 3: An edge-based Architecture for PD severity assessment.
The edge computing layer contributes to data collection from the Kinect sensors, data preprocessing by the Brekel body v3 software and data classification using proposed mathematical model which will be discussed in the next section. The Brekel body v3 software is utilized in conjunction with the Kinect setup to combine the data from the sensors to generate CSV file. The Brekel body v3 software enables real-time motion capture which can be beneficial for immediate feedback and interactive software development. It can export motion capture data to various formats, making it compatible with different development software. The patient data is stored in the back-end database server SQLite that will be exploited by the proposed web application.
Furthermore, the cloud computing serves as a central layer that provides services for long-term data management, processing, and storage. This enables healthcare organizations to make data-driven decisions, optimize processes and improve overall efficiency.
The application layer is intended to employ the PD severity assessment service where a web application is developed using Bootstrap, an open-source front-end framework to be accessed by the different stakeholders such as doctors, clinicians, patients. Stakeholders can visualize the report generated and shared data among them using Power BI dashboards. It enables enhanced data visualization of complex datasets for the stakeholders who can easily share insights and collaborate on report findings due to the cloud-based nature of Power BI dashboards. The Interface between the application and backend server is implemented using open-source python-based web framework called Django.
As illustrated in figure 4a, authorised users like medical practitioners or clinicians are enabled to access the web application, where they can enter the patient's personal and medical details in the form, shown in figure 4b. Then the subject is instructed to perform walking session on the predefined FOG-provoking path. The respective human skeletal joint data can be collected from the Kinect setup and they are combined to produce CSV file by the Brekel body v3 software. The file will be uploaded to the application by the clinician. The recordings of the walking experiment will be analysed through mathematical models to detect the occurrence of FOG events. Finally, the clinician and the patient are able to visualise the analysis reports that have been generated in the power BI dashboard.
Figure 4: a) Proposed web application framework and b) Patient’s data form.
The details of all the patients and their files uploaded through the framework will be stored with the respective patient primary IDs and managed in the database server. The data in the database can be altered using the administrator access login.
FOG classification model
As mentioned before, FOG refers to an inability to move irrespective of the wish to move forward. As walking involves the joints of the leg such as the hip, knee, ankle, and foot, the proposed classification model uses the ankle and foot joints to assess FOG. The WWT experiment needs both left and right feet X trajectories while walking in the forward and backward directions to detect all the FOGs. So, the left foot trajectory is used before turning and the right foot trajectory is used after turning. Finally, the results of both feet are fused to obtain the final FOG assessment for the experiments conducted.
Figure 5 illustrates the plot of the walking pattern of the healthy control and PD patients at various progressive stages of the disease, as a result of conducted WWT experiment. It shows that the gait pattern of the control subject seems to be quite steady.
Figure 5: Plot of the walking pattern for (a) Control subject, (b) Low PD subject, (c) Moderate PD subject, and (d) High PD subject.
Figure 6 shows the plot of the X trajectory samples of the foot joint of the subjects at different phases of the PD. The x-axis of the plots represents frame IDs and the y-axis shows the displacements of the foot joint in the x-axis. The X trajectory of the left foot is plotted as the decreasing curve before the turning phase and that of the right foot is plotted as the increasing curve after the turning phase. The turning process is indicated in the curve at the global minimum.
Figure 6: Plot of the X trajectory samples for (a) Control subject, (b) Low PD subject, (c) Moderate PD subject, and (d) High PD subject.
As FOG makes the patient unable to move for a period, the X trajectory of the foot joint during FOG episodes would be flat. Hence, these episodes have derivatives equal to or close to zero. These zero or near-to-zero regions are extracted as Regions of Interest in FOG (FOG-ROIs).The derivative of the X trajectory is represented as f(x) and is computed based on Equation (1).
where g(x) shows the derivative of the X trajectory signal. The term f(x_i) represents the displacement value in the x-axis at the current frame ‘i’ and f(x_(i-1)) represent the same at the previous frame 'i-1'. Assume 'n' denotes the total number of movement frames captured by Kinect sensor for a subject. The ROIs that might represent FOG episodes of the subject, can be identified using Equation (2).
As shown in figure 7, all the FOG-ROIs are extracted from the raw X trajectory signal. However, not all the regions are classified as FOG episodes as some of them are too short. Figure 8 plots the h(x) regions which is 1 for the area with a derivative close to zero and 0 for other areas. If h(x) has a group of consecutive 1's and its length is greater than a threshold then the region is classified as a real FOG region. All the short zero derivative areas are ignored.
Figure 7: The plot of X trajectory signal from the walking pattern (Figure 3). The Regions of interest regions for (a) Control subject, (b) Low PD subject, (c) Moderate PD subject, and (d) High PD subject.
Figure 8: The derivative g(x) plot in blue and the h(x) plot in red. The Refined Region of interest regions for (a) Control subject, (b) Low PD subject, (c) Moderate PD subject, and (d) High PD subject.
The classification model helps filter out FOG episodes from resting or standing periods from the FOG-ROIs that were extracted earlier. This can be performed by determining the angle between the ankle and foot joint and between the foot joint and the projection point of the ankle joint on the ground line. During a FOG episode, the angle will be constantly changing whereas it would be stable when the patient is in standing or resting mode It can be calculated using Equation (3).
where v1 indicates the vector between the ankle and foot joint and v2 indicates the vector between the foot joint and the projection point of the ankle joint on the ground line. The value a_AG can be computed for each frame resulting in a list of a_AG as shown in Equation 3.
Next, the identified FOG episodes from the preceding stage are evaluated to eliminate the standing mode regions. The displacement of the angle list AFG_list is utilized to ascertain the degree of change in angle over time for the subject under observation. The displacement can be determined by Equation (4).
where n is the number of frames. a_AGi and a_(AG(i-1))represents the angle in ‘i’th frame and ‘(i-1)’th frame respectively.
Step detection
When the knee angle falls below a predetermined threshold, a foot-off event is deemed to have occurred. After repeated trial and error analysis, the knee angle was determined to be 170 degrees. Additionally, foot contact occurs when the knee angle of the same foot returns to its initial value (170 > 180) after more than 200 milliseconds. Due to some irregularities and noise in the Kinect data readings, the 200ms timing barrier was imposed to prevent false positive signals. But since this need not always be the case as shuffling of steps occurs in PD patients, there may not be a foot-on-foot-off event with the knee angle going below 170 degrees. So, the knee angle formula is not a viable option for calculating the steps.
Since the path is restricted on the z-axis, we can calculate the step occurrence by calculating the gradient of the foot’s x trajectory with 5 frames as the difference. This lets us know the timestamps at which the patient has taken a stride, using which we can calculate the stride length and cadence for the Cadence based FOG Detection formula using Equation (5).
where g(x) shows the derivative of the trajectory signal. f(x_i) represents the displacement value of the current frame ‘i’ and f(x_(i-5))- displacement value of the 5th previous frame from ‘i’.
The multi-Kinect sensors are set up and the volunteers are asked to perform the experiments. The proposed system generates the list of FOG episodes based on the gradient displacement of the foot trajectory. In order to reduce false positives from resting mode, akin to the characteristics of FOG modes, it also employs the gradient displacement of the angle between the ankle and foot joints and the ground. Furthermore, the method under consideration guarantees that the system provides users with an accurate count of FOGs.
Figure 9 shows that the FOG occurrence duration plotted along the timestamp and number of FOGs occurred for the subjects at different stages of the disease. The regions marked in red depict the regions where FOG occurred. Figure 9a reports for the control subject for whom there was no FOG occurred. Figure 9b and 9c report that few FOG episodes are experienced by subjects with low and moderate PD. Figure 9d reports that the subject with high PD had more FOG episodes during the experiment conducted.
Figure 9: FOG occurrence in the red plot for (a) Control subject, (b) Low PD subject, (c) Moderate PD subject, and (d) High PD subject.
Figure 10 shows the FOG occurrence and festation regions in red and green plots respectively for the subjects at different stages of the disease. The regions marked in green depict the festation regions where the subjects experience shuffled steps during walking experiments. Figure 10a reports for the control subject for whom there were some shuffled steps experienced. Figure 10b and 10c report that some festation gait is experienced by subjects with low and moderate PD. Figure 10d reports that the subject with high PD experiences more shuffled steps along with FOG episodes during the experiment conducted.
Figure 10: FOG occurrence in the red plot and festation in the green plot for (a) Control subject, (b) Low PD subject, (c) Moderate PD subject, and (d) High PD subject.
Our proposed FOG assessment system is evaluated for healthy subjects simulating the PD patients at different stages of the disease. However, in future work, it will be evaluated with significantly larger number of real PD patients at disease progression.
In the proposed work, we have developed an edge-based real-time motion capture system for PD Severity Assessment Scenario using multi-Kinect Sensors setup. It captures the gait data of PD patients which can process the recorded skeletal joint data of the subjects in a 3D space through the proposed mathematical model to identify the occurrence of FOG events. Based on the identified FOG events, the Regions of Interest (ROI) are extracted and refined to draw up informative visualizations and generate the required reports.
The proposed system offers its services through a web application deployed on the intranet for easy access from any system connected to the internet. The web application makes it easy for practitioners to upload data and obtain the required inferences and reports. Therefore, the non-invasive sensor-based assessment technologies have the potential to enable precise evaluation of motor manifestations in PD, resulting in 1) better and early diagnostic accuracy, 2) more sensitive monitoring and 3) more precise adjustments to medical therapies. Overall, the system is a software package that can help the practitioners identify FOG events that would help them provide more patient-specific treatment that would in turn improve their quality of life.
The authors thank every volunteer that participated in the study.
The authors declare that there are no conflicts of interest.
Citation: Madhana K, Jayashree LS, Manoj KN (2024) Severity Assessment Framework for Parkinson’s Disease with Real-Time Bio-Kinematic Movement Tracking Using Multi-Kinect System. J Gerontol Geriatr Med 10: 237.
Copyright: © 2024 Madhana K, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.