The reason such comparisons are helpful is that, unless hear your processors back to back, it’s hard to notice the subtle differences in their sound. They all use different algorithms, so just because two brands have the same type of processor, it doesn't mean they'll sound identical. You might have a general sense that two compressors both add some character, but you will be able to better understand the differences in their sounds, and which you like better, after you compare them.
Apples to apples to oranges
Within each effect category, there are two different types of comparisons you can go for: comparing the same types to each other, for instance two plate reverbs, two 1176 clones, etc. Alternatively, you could test two that are in the same effects category, but are emulating different flavors of the effect, such as a plate and a hall reverb, or an optical and a tube compressor.
In the case of the former, you can listen for which one one sounds better. In the latter case, A/Bing will help you better understand the sonic differences. In both cases, the process of doing the comparison will have the added benefit of making you more familiar with the controls on the plug-ins.
A/B setup tips
There are several different ways to do an A/B comparison. The fastest to setup involves inserting both plug-ins on the same channel, and bypassing one at a time. It’s helpful to either loop a section of audio on the track, or set your DAW up to return to the same spot after you hit stop (most DAWs have a function like that). That way, you’re hearing the same section of audio for both A and B.
You’re best off stopping the transport after you listen to A, then bypassing A and activating B before you turn the transport back on to listen to B. Otherwise, you’ll hear a section of unprocessed audio after you bypass A and before you activate B, and that can confuse the comparison. It’s much better to hear A right after B with no other audio in between. For that same reason, you might not want to compare more than two plug-ins at a time.
Although it takes a little bit more setup, a more efficient way to go is to duplicate the track with the audio you’re using for the comparison, and insert effect A on one track and effect B on the other. Then slide the audio of B to the end of A. It’s best to keep the examples short, so when you loop it it will move pretty quickly from A to B. The best part is once you hit play, there's no button pushing required.
Fair is fair
If you want your comparisons to be of value, you need to make sure that the parameter settings on each processor are as close to the same as possible. Let’s say you were comparing compressors, you’d want each to have the same ratio, same threshold, same attack and release times. You’d also want to make sure you’re getting the same amount of gain reduction (on most compressors, this is shown in the gain reduction mode of the meter).
In some cases, you’ll have to do a little math, because some plug-ins express certain parameters differently. For example, one EQ might express upper mid and high frequencies in Kilohertz while the other is always in Hertz.
For some types of processors, reverbs especially, there are so many possible parameter types, that it can be hard to match the sounds totally. In that case, just do the best you can to get there settings as close as possible, and make sure, at least that you’re using the same type of algorithm (plate, hall, etc.,) unless you're comparing those against each other. Some key reverb parameters are reverb time, pre-delay, diffusion and wet/dry mix.
In any comparison, it’s crucial that A and B play back at the same volume. That’s because otherwise; the louder one will invariably sound better, which will mess up the whole exercise. Your ears are the best judge of this, but also check on the meters of the channels (or on the master fader if you have one setup) and adjust the volumes of the plug-ins so that they match as closely as possible. Make sure each plug-in is getting a similar amount of input level, as well, because, especially with compressors, this can change the character of the processing.
As to what settings to use to start with, one way to go is to open up a preset on plug-in A that’s designed for the kind of source material you’re testing on, and then match the parameters of B to it.
In cases where you're not noticing much difference you might want to exaggerate the settings a little. You might boost or cut that EQ band by a couple of more dB than you’d likely do on a real project (don’t go crazy, though) or set the compressor threshold a little lower than you might normally, or set the wet/dry a little wetter on an reverb or delay.
Once you’ve gotten things setup and the parameters as close as possible, start listening back and forth between A and B. Compare the sound quality, both in terms of clean reproduction and added character. Some processors will add a sonic signature that can be pleasing and alter the tone. Others (especially in the case of EQs and some compressors), achieve their results transparently. On reverbs, compare how the tails decay. Is one smoother than the other, or more rich and full? If you're comparing different flavors of an effect, make a mental note of the sonic differences.
On delays, especially analog or tape delay emulations, pay attention to the taps. How does the sound quality change on each successive one? It's best to compare the processors on more than one type of source. Compare them on drums, vocals, guitars, etc., so you can more fully grasp the affect of those processors on various types of tracks.
The goal here is to better understand your plug-ins, so that you can make more informed decisions about which one to use in what situation. You’ll be surprised at the subtleties you’ll discover. What's more, with all the mucking about you’ll do in the parameters when setting up the comparisons, you'll get a better grasp of the various controls, functions, and capabilities of your plug-ins. Happy testing!