TL;DR: Only program in voice channels if your scanner requires it.
For older scanners where you have to enter the control channel and the voice freqs there is no alternative--you have to enter it all so the speed is what it is.
On newer control channel only scanners, you'd actually slow it down by entering the voice freqs. The control channel just announces channel numbers for active talkgroups; the scanner knows* what the channel numbers are.
When you program in the frequencies, the scanner is going to listen for something that sounds like a control channel. If it can't find one or if it's too weak it will move on to the next programmed freq. This is why you want to program any alternate control channels. If you program in voice channels alongside the control channel(s) the scanner will stop on those freqs even though there's never going to be an active control channel. Of course if the RSSI is below a certain threshold it's not going to stop for more than a few milliseconds, but on a good signal it might pause for the entire length of an analog voice call, or possibly skip after a few hundred milliseconds of not being able to decode it as a control channel.
*For a rebanded system you have to set up a custom band plan because the frequencies are now assigned to different channel numbers.