Speaker recognition systems are susceptible to model inversion attacks that can reconstruct sensitive user data from model outputs, posing significant privacy risks. This study investigates the key factors influencing the success of such attacks and explores effective defense strategies. Our research is divided into two core components: implementing model inversion attacks and evaluating defense mechanisms. For the attack analysis, we examine how input data representations, feature extraction techniques (e.g., Mel-frequency cepstral coefficients), dataset characteristics, and model architectural complexity impact attack performance. Our findings indicate that more complex model architectures present greater challenges for attackers. In evaluating potential defenses, we demonstrate the efficacy of differential privacy, particularly for raw waveform and SincNet acoustic models. Furthermore, we show that adopting targeted feature extraction methods and refining model architectures can significantly enhance the resilience of speaker recognition systems against model inversion attacks. Our study provides valuable insights into the vulnerabilities of speaker recognition systems and proposes robust defense strategies to mitigate privacy threats posed by model inversion attacks in communication technologies.