Incremental Learning in Time-series Data using Reinforcement Learning

Mustafa Shuqair, Joohi Jimenez-Shahed, Behnaz Ghoraani

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Scopus citations


System monitoring has become an area of interest with the increasing growth in wearable sensors and continuous monitoring tools. However, the generalizability of the classification models to unseen incoming data remains challenging. This paper proposes a novel architecture based on reinforcement learning (RL) to incre-mentally learn patterns of time-series data and detect changes in the system state. Our rationale is that RL's ability to learn from past experiences can help increase the performance and generalizability of classification models in time-series monitoring applications. Our novel definition of the environment consists of a set of one-class anomaly detectors to define environment states based on the dynamics of the incoming data and a reward function to reward the RL agent according to its actions. A deep RL agent incrementally learns to perform continuous, binary classification predictions according to the environment states and the received reward. We applied the proposed model for detecting response to medication (ON or OFF) in patients with Parkinson's disease (PD). The PD dataset consisted of 170 minutes of time-series movement signals collected from 12 patients using two wearable sensors. Our proposed model, with a testing accuracy of 77.95%, outperformed Adaptive Boosting, Multi-layer Perceptron, and Support Vector Machines with 53.10%, 44.92%, and 52.70% testing accuracy, respectively. The proposed model had a slight decline in the F-score, decreasing from 88.15% validation score to 78.42% in testing, a significantly slight decline compared to the other three models. These evidence the potential of the proposed RL-based classifier in time-series monitoring applications as a highly generalizable model for unseen incoming data.

Original languageEnglish
Title of host publicationProceedings - 22nd IEEE International Conference on Data Mining Workshops, ICDMW 2022
EditorsK. Selcuk Candan, Thang N. Dinh, My T. Thai, Takashi Washio
PublisherIEEE Computer Society
Number of pages8
ISBN (Electronic)9798350346091
StatePublished - 2022
Event22nd IEEE International Conference on Data Mining Workshops, ICDMW 2022 - Orlando, United States
Duration: 28 Nov 20221 Dec 2022

Publication series

NameIEEE International Conference on Data Mining Workshops, ICDMW
ISSN (Print)2375-9232
ISSN (Electronic)2375-9259


Conference22nd IEEE International Conference on Data Mining Workshops, ICDMW 2022
Country/TerritoryUnited States


  • Deep q-learning
  • deep reinforcement learning
  • incre-mental learning
  • reinforcement learning
  • system monitoring
  • time-series data


Dive into the research topics of 'Incremental Learning in Time-series Data using Reinforcement Learning'. Together they form a unique fingerprint.

Cite this