After he puts the phone down he starts to think that maybe he has been too polite. She has obviously failed to understand his refusal. When he had said, laughing, that his participation was "unlikely", she had apparently failed to realise he had really been saying "No. Not in a hundred years. Not if you went down on your knees and begged me . . ."