MULTI-SOURCE DOMAIN GENERALIZATION FOR TRANSFORMER-BASED SENTIMENT CLASSIFICATION WITH LIGHTWEIGHT REPRESENTATION REFINEMENT

Abstract

Fine-tuned encoders based on the transformer architecture produce extremely accurate results when testing review texts are drawn from the same product category as training texts. Nevertheless, the models that are applied to review texts from another product category show lower accuracy, inconsistency in predictions between the initial states of the model, and unpredictable behavior. Domain adaptation addresses this issue by adding samples belonging to the target product category into the training process; however, a practical sentiment analysis system that can be deployed in production lacks access to target category samples. There might simply be no category to review yet at the moment when the model is being developed. The more complicated problem, namely, training on any labelled categories and later producing good results on an unseen one, is called domain generalization. This is the topic of my thesis. Below is the empirical study performed on multi-source domain generalization of sentiment classification task using transformer-based neural networks on Amazon Multi-Domain Sentiment dataset. In all experiments, the strict rule of no-target access will be maintained: no labels and even unlabelled texts from the target domain will be used during the training phase. The DistilBERT base uncased architecture was chosen as the encoder in my experiment, and I compare four setups: (i) empirical risk minimization (ERM) using single source – training of the model separately for each category; (ii) ERM using multiple sources – merging three categories into one dataset; (iii) DANN baseline for multiple sources; and (iv) the approach suggested in my thesis with adding a tiny projection-residual module on top of the DistilBERT CLS vector while keeping the encoder fully fine-tuned. This additional module is based on the two-layer bottleneck architecture with a skip connection. It takes roughly 0.6% of total model parameters, and initially serves as an identity function. With leave-one-domain-out on the four domains (Books, DVD, Kitchen, and Electronics), our proposed model reaches an average cross-domain accuracy of 86.56%. This number exceeds the single-source model baseline by 1.40%, beats the multi-source ERM baseline by 0.69%, and surpasses the DANN baseline by 0.50%. The most notable gains are made in target domains Books and Kitchen, where the proposed approach outperforms the multi source ERM baseline in all random seed configurations. In the challenging target domain Electronics, the projection-residual module does not contribute much to the average, but almost halves the standard deviation (0.54 to 0.31) through a more stable performance v pattern—a finding verified through a second residual-connection ablation analysis. With another ablation test, we found out that the projection-residual module is not particularly sensitive to the bottleneck dimension size r. We experimented with bottleneck dimensions r = {64, 128, 256, 512}, and observed that all four options yielded similar average accuracies within one percentage point of each other, except for r = 64, which performed better in the Electronics target domain. This research concludes that a simple combination of all-source category aggregation, using cross-entropy fine-tuning, and using an additional refinement layer on top of the encoder's classification token is sufficient for obtaining significant and reproducible performance gains in cross-domain sentiment classification. It is not necessary to use any adversarial training or invariance regularization, nor any modifications to the encoder's structure.