Two possible causes:
1. Yummy tries Passive mode first, which fails, Yummy waits for a while then tries Active mode. Try switching to Active mode and see if that makes any difference. (See the Advanced tab of the Bookmark settings for the 'Use passive FTP' checkbox). If this makes no difference, switch back to Passive mode.
2. Your sites have a lot of link files which Yummy tries to resolve before displaying the listing. Try unchecking the 'Resolve symlinks' checkbox in the Server Options tab of the Preferences. If this makes no difference, re-enable the checkbox
If neither of these fix the problem, please send me an FTP Transcript of a listing that takes a long time - select Save FTP Transcript from the File menu and send the resulting file to me: support at yummysoftware dot com