Naive Bayes vs Logistic Regression: Theory, Implementation and Experimental Validation

This article presents the theoretical derivation as well as practical steps for implementing Naive Bayes (NB) and Logistic Regression (LR) classifiers. A generative learning under Gaussian Naive Bayes assumption and two discriminative learning techniques based on gradient ascent and Newton-Raphson methods are described to estimate the parameters of LR. Some limitation of learning techniques and implementation issues are discussed as well. A set of experiments are performed for both the classifiers under different learning circumstances and their performances are compared. From the experiments, it is observed that LR learning with gradient ascent technique outperforms general NB classifier. However, under Gaussian Naive Bayes assumption, both classifiers NB and LR perform similar.