Yes, of course Yummy works with an IP address. That is the basis of how the Internet works

The message you're receiving is not from Yummy - it's from the server itself, so this means that you got connected to the server successfully using the IP address, but the login details you provided (Username & Password) were not accepted.
So, I would check with your hoster to see whether your username and password are correct.
The issue about Passive FTP would only come into play if you had successfully logged into the server. Clearly you haven't even got that far, so that isn't the problem.