Justin Martyr was keen for Christianity to be recognise as legal in the empire. But his argument is that Christianity is not criminal, not that no criminals are Christians. In fact, he says some criminals will be Christian (but the Christian bit shouldn’t be a crime). Do you get what I’m saying he’s saying?
“Some have ere now been arrested and convicted as evil-doers. For you condemn many, many a time, after inquiring into the life of each of the accused severally, but not on account of those of whom we have been speaking. And this we acknowledge, that as among the Greeks those who teach such theories as please themselves are all called by the one name “Philosopher,” though their doctrines be diverse, so also among the Barbarians this name on which accusations are accumulated is the common property of those who are and those who seem wise. For all are called Christians. Wherefore we demand that the deeds of all those who are accused to you be judged, in order that each one who is convicted may be punished as an evil-doer, and not as a Christian; and if it is clear that any one is blameless, that he may be acquitted, since by the mere fact of his being a Christian he does no wrong. “