"Hardly," he scoffs. "I don't ally myself with dull, goalless lunatics."

Riddler doesn't touch the word friend. Nothing like that exists for villains, after all.

"How do you know me?" Not that he expects a proper answer, but there might be clues.