If you want framework-agnostic code, Django is really not the right framework for you. Its large ORM layer, heavy use of middleware, and reliance on class inheritance, will make migrating to another framework very painful. For more freedom and control I would recommend Flask, or even better FastAPI.

At the same time, it seems DDD has nothing to do with using other libraries or not.

Either way, whatever framework you choose, remember the golden rule: Don’t Fight The Framework.