We chose Kubernetes initially for the same reasons as everyone else -- it was the future.
One thing we realized really quickly was it would introduce a massive amount of complexity into our deployments. The most difficult constraint to overcome was deploying it as the main orchestrator for our on-prem instances. At first, we ran into huge blockers, where there were incompatibilities with operating systems, configuration / network complexity and even mysterious failure states (bad for an orchestration tool!) . The worst of all is the installation for simple rollouts became overcomplicated and unreliable -- often failing due to a setup issue rather than an application issue.
Over the past 2 years the kubernetes landscape has changed dramatically. Installation issues have been knocked away while the improvements in deployment reliability and the amount of middleware available has dramatically made our lives easier. Now k8s is our main on-prem installer / orchestrator and works out of the box wonderfully. The shared deployment stories between cloud and on prem now feel like a reality.
