Abstract:
Humans can continuously accumulate, develop, and transfer knowledge and skills
throughout their lifetimes, giving rise to continual learning principles. However, Isolated learning is currently the most used paradigm in machine learning, where the
model experiences catastrophic forgetting or interference. Catastrophic forgetting occurs when an artificial neural network tends to completely forget information learnt
previously as it learns new information. Catastrophic interference occurs because
many of the weights that contain older information are changed as new information
is learnt, hence, forgetting the previously stored knowledge.
Further, continual learning is more challenging for NLP, because language is vague
and its meaning depends on context. Continual learning is crucial in real-world
natural language processing applications, where computer systems must interact with
ongoing streams of data and language across time. When forced to adapt to new tasks
and inputs, language models experience catastrophic forgetting
