I've never thought any of STP's catalog to be particular "difficult" to listen to due to the production/mixing. No. 4 certainly has the poorest mix/production out of their whole catalog, though. Core is a close second though, maybe even tied. Tiny Music isn't far behind... TM is like a blend between the production styles of Purple and No. 4. Purple was far and away the best-- perfect tone throughout that entire record. Core would have been good, but way too much reverb and the treble is too bright. Ever hear the isolated vocals from Core, WITHOUT the reverb/effects applied? Man, talk about ass kicking vocal power.
So while I think No. 4 is definitely the worst, I think Core stands to benefit the most from a remix, if that makes any sense. They should do a Core Redux just like they did with PJ's Ten album-- the production problems with Core are almost identical to the problems the original pressing of Ten has (too much reverb, too bright).