The two concepts are intermingled. Ad fraud detection is specialist software, that analyzes users of a website or advertisement, to look for the signals that indicate fraudulent activity. Analytics, measures customers engagement and so in turn can also pick up the signs of fraudulent activity. However it is fair to say analytics is more of a general solution, and ad fraud detection is a specialist solution.
There is also no harm in using analytics and ad fraud software, to help use each tool for their specific purpose and often they two can help triangulate similar trends.