Margin is a significant algorithmic parameter that has an impact on the performance of margin-based machine learning algorithms. Margin setting algorithm (MSA) is a novel margin-based learning algorithm. However, there is no comprehensive study concerning the impact of setting margin on enhancing the performance of MSA. In this article, we studied the impact of margin on performances of MSA by comparing it to another popular margin-based algorithm, the support vector machine (SVM). This comparison comprehensively analyzes and compares how margin affects training performance and generalization, both theoretically and experimentally. In our theoretical analysis, margin definition and margin impacts are comprehensively discussed by demonstrating how they affect the decision boundary. Experimental analysis is performed on two-dimensional Gaussian data sets and benchmark data sets. The experimental results support our theoretical analysis, revealing that with an increased margin, training performance gets worse and generalization tends to improve within a certain range.