HiProIBM: unsupervised continual learning through hierarchical prototypical cross-level discrimination along with information bottleneck subnetwork masking

Catastrophic Forgetting (CF) occurs when a machine learning model forgets the experience of previous tasks while learning new tasks due to inadequate retention mechanisms. Unsupervised continual learning (UCL) addresses this by enabling the model to adapt to new tasks using unlabeled data while retaining past knowledge. To mitigate CF in UCL, we use a parameter isolation technique to mask sub-networks dedicated to each task, thus preventing interference with previous tasks. However, relying solely on weight magnitude for constructing these sub-networks can result in the retention of irrelevant weights and the creation of redundant sub-networks. This approach also risks capacity saturation and information suppression for tasks encountered later in the sequence. To overcome this, we use masked sub-networks, inspired by the information bottleneck (IB) concept. It accumulates valuable information into essential weights to construct redundancy-free sub-networks which effectively mitigates CF and enables the new task training. The IB subnetwork masking faces challenges in balancing input compression with meaningful pattern preservation without labels. It risks overcompression and loss of crucial latent structures, which degrades model performance. We address this by learning multiple semantic hierarchies present in the data using unsupervised contrastive learning. However traditional contrastive learning techniques learn meaningful representations by contrasting similar and dissimilar data points. These approaches lack adequate representational power for modeling datasets with multiple semantic hierarchies. The inherent hierarchical semantic structures in datasets are necessary to integrate semantically related clusters into larger, coarser-grained clusters, but existing contrastive learning methods often overlook this and limit semantic understanding. We address this by constructing and updating hierarchical prototypes with cross-level group discrimination to represent semantic structures in the latent space. Our experiments on four standard datasets show performance improvements over SOTA baselines for varying task-sequences from 5 to 100, with nearly-zero forgetting.