The former Health Secretary has been disinvited from a top travel conference after WhatsApp leaks