Sounds like a weak connection somewhere, check the solder joints, tube sockets, grounds.
It’s probably not the speaker. I know you triple checked everything already (we all do) and that all that all the solder joints are perfect (we all do that too), but 9 times out of 10….. Maybe more like 9.9 times out of 10, it’s going to be weak joint, marginal ground, tube socket or something like that. Of course, it could be a lot of things…. bad resister, pot, bad speaker, bad tube. Did you try the chop stick test to see if you an make it happen?