If you have a match with your purchase order to an invoice in the Invoice module, you can remove this match simply by removing (deleting the information, or clicking on the 'x' to clear the field) the PO number from the PO field box.
The PO number can be found in either the header, or the line items table.

If you would like to match multiple POs to one invoice, turn on the PO# in the Line Items table by selecting 'Display PO Column' on the right side of the line items table.

If you have already selected the PO in the header fields, you will need to delete it as the PO cannot be in both the line items table and the header table.

If your PO fields are not accessible, this is because the invoice is not editable or you are not the owner. In the first case, you will first need to reset your invoice in order to remove the PO number or assign it to yourself.