You won’t really hear any difference with the impedance switch. But maybe you have that wired incorrectly. The round eyelets should be what you are switching (green, yellow or white) and the odd shaped one (on the same side as one of the round eyelets) should go to the output jack. I think thats how it goes if I recall correctly.
Feel free to post a picture and we can see if it looks right. Also, make sure that the coax cables that you use to connect across the amp have the shielding ground at only one side.