Hi Sathish,
Straightforward responding you: Will the shift from Datadog to Grafana be a wise move? R: Maybe, specially in the long run.
and flawless? R: It depends, but I don't think so. And it also won't be so smooth. You'll reach a point of Datatog deprec where your team will be supporting both, and at this peak will be when you'll think it was a bad move, because you'll be paying for Datadog and Prometheus/Grafana servers, plus your team will have to work on new system metrics and fine tuning metric queries and graphs on Grafana, so you can expect a delay on business deliveries. Your team will feel overburdened.
So bear in mind that basically you'll be moving from an out of the box monitoring solution to your own. The analogy I'd do is to moving out from a cloud provider to your own self managed servers, though way less complex of course.
Grafana itself won't do the trick. You'll need a data scrapper system like Prometheus to collect metrics/data from your app, or use a middleware Pushgateway to receive these data so Prometheus can scrape them. To do this you'll need some backend work to expose metrics data, instead of HTTP post your metrics data to Datadog, so you can expect a little re-architecture or engineering on that, depending on how your system is designed.
In the long run, you'll probably need a team focused only on that: to take care of the Prometheus/Grafana updates as well as their servers (after all they're still systems that need to be managed).
I think all of this will depend on the size of your company now and your teams (which I advise you to include them on that decision), and your budget plus rush for business deliveries. In the long run it'll pay, for your company will be more specialized on this stack, and you can have a team focused on that plus developer support.
If you wanna go with it, I'd suggest to do a small PoC (which still will need a good amount of work). Select a project (or even an endpoint/ mobile feature) to have both integrations: Datadog plus Grafana, and do the same graphs you have on Datadog to Grafana. I think this way you can have a better picture of the reality on how that will be for your company.
