Codecov vs Coverity Scan: What are the differences?
Codecov: Hosted coverage reports with awesome features to enhance your CI workflow. Our patrons rave about our elegant coverage reports, integrated pull request comments, interactive commit graphs, our Chrome plugin and security; Coverity Scan: Find and fix defects in your Java, C/C++ or C# open source project for free. Coverity's implementation of static analysis can follow all the possible paths of execution through source code (including interprocedurally) and find defects and vulnerabilities caused by the conjunction of statements that are not errors independent of each other.
Codecov belongs to "Code Coverage" category of the tech stack, while Coverity Scan can be primarily classified under "Code Review".
Some of the features offered by Codecov are:
- Beautiful Reports
- Pull Request Comments
- Interactive Commit Graphs
On the other hand, Coverity Scan provides the following key features:
- Test every line of code and potential execution path.
- The root cause of each defect is clearly explained, making it easy to fix bugs
- Integrates with GitHub and Travis CI
What is Codecov?
What is Coverity Scan?
Need advice about which tool to choose?Ask the StackShare community!
Why do developers choose Coverity Scan?
Sign up to add, upvote and see more prosMake informed product decisions
What are the cons of using Coverity Scan?
Sign up to get full access to all the companiesMake informed product decisions
Sign up to get full access to all the tool integrationsMake informed product decisions
We use Codecov because it's a lot better than Coveralls. Both of them provide the useful feature of having nice web-accessible reports of which files have what level of test coverage (though every coverage tool produces reasonably nice HTML in a directory on the local filesystem), and can report on PRs cases where significant new code was added without test coverage.
That said, I'm pretty unhappy with both of them for our use case. The fundamental problem with both of them is that they don't handle the ~1% probability situations with missing data due to networking flakiness well. The reason I think our use case is relevant is that we submit coverage data from multiple jobs (one that runs our frontend test suite and another that runs our backend test suite), and the coverage provider is responsible for combining that data together.
I think the problem is if a test suite runs successfully but due to some operational/networking error between Travis/CircleCI and Codecov the coverage data for part of the codebase doesn't get submitted, Codecov will report a huge coverage drop in a way that is very confusing for our contributors (because they experience it as "why did the coverage drop 12%, all I did was added a test").
We migrated from Coveralls to Codecov because empirically this sort of breakage happened 10x less on Codecov, but it still happens way more often than I'd like.
I wish they put more effort in their retry mechanism and/or providing clearer debugging information (E.g. a big "Missing data" banner) so that one didn't need to be specifically told to ignore Codecov/Coveralls when it reports a giant coverage drop.