You could be held liable for any losses suffered by the vendor. These will include the vendor's legal costs and so on. But they could also include any subsequent fall in the value of the premises, and the costs the vendor has finding a new purchaser.
Use your solicitor to negotiate an agreement if you want to pull out of the purchase. Make sure that your liabilities are clearly agreed and limited.
