Yes, there are a number of ways in which this can happen. For example, if you allow someone to act as if they are your agent, you may well create an agency relationship even without an explicit agreement.
The safest option in such circumstances is to create a written agreement. This will clarify what the relationship is and what each party's rights and responsibilities are. Otherwise, you might find that you have unwittingly given the agent various rights, or the authority to act for you in ways you did not intend.
Note that self-employed agents (any agent who isn't your employee) performing their duties within the EU in relation to goods have certain statutory rights. Some of these can be contracted out of, some of them cannot (but it may be appropriate to address them in other ways) and all of them need to be considered.
