Yes, unless you're an Admin, you need to know the exact email address or login of anyone you want to add to your project. There is currently no way to search for users.
That's the easiest way for us to enforce privacy.
We do want to improve this, but it will take time. Ultimately we'd like to group users and project into organisations and then provide more privileges/visibility within those organisations.