Correct, an empty wallet cannot use an own input for the fee, so it will not be a payjoin and as such the direction of the transaction is more easy to guess.
As far as I understood, payjoin is not jet implemented in the contract branch even though you would expect it to be since fees are paid by both sender and receiver.