Alternatives to Pandasql

Hello everybody! I want to create an easy to use website (and simplified) builder. Users create a site, add their domain and edit Pages inside an Admin Area inside predefined "Blocks" like with Wagtail.

There should be several default templates/themes and layouts. Choosing a different theme might change both the layout and styling. And I'm looking for SSR or SSG with a focus on the end-user's Page performance.

Current Stack:

• GraphQL for the dynamic part (for example data to Pricing might not be required on the homepage, or images might go into gallery in one theme, and on the homepage in another theme)

• FastAPI and SQLAlchemy to store Data. I have prior experience with Wagtail and Django and want to stay with Python and learn Fastapi

• NGINX to handle domain-specific things (the how is not yet refined)

I want a scalable architecture and especially a fast frontend. I'm tending to use SvelteKit, but I'm afraid that the dynamic part won't make me happy after having hundreds of sites, as the bundler cannot load themes dynamically. sveltekit - dynamic import

I want to handle the scenario: Admin-User chooses a theme and other options for the site (domain) and the End-User Frontend/Session/Cookie will have to fetch and store data about the theme and load subsequent calls to prerendered sites.

Also the "hundreds of sites" might not be all covered by 5 Themes, because users may purchase a custom-made theme, and I would have to manage one big project having lots of different Styleguides and flows. I even thought of statically building the site whenever the user makes a change, maybe using Hugo (or VitePress?). Then I would have to take more care with the page recreation and it would be easily routable instead of nesting every component with logic in the frontend... I think. But this I haven't thought through yet.

Can anyone please help?

Merging Django ORM with SQLAlchemy for Easier Data Analysis (Video Inside)

Development of products with Django framework is usually easy and straightforward; great documentation, many tools out of the box, plenty of open source libraries and big community. Django ORM takes full control about SQL layer protecting you from mistakes, and underlying details of queries so you can spend more time on designing and building your application structure in Python code. However, sometimes such behavior may hurt - for example, when you’re building a project related to data analysis. Building advanced queries with Django is not very easy; it’s hard to read (in Python) and hard to understand what’s going on in SQL-level without logging or printing generated SQL queries somewhere. Moreover, such queries could not be efficient enough, so this will hit you back when you load more data into DB to play with. In one moment, you can find yourself doing too much raw SQL through Django cursor, and this is the moment when you should do a break and take a look on another interesting tool, which is placed right between ORM layer and the layer of raw SQL queries.

Server side

We decided to use Python for our backend because it is one of the industry standard languages for data analysis and machine learning. It also has a lot of support due to its large user base.

• Web Server: We chose Flask because we want to keep our machine learning / data analysis and the web server in the same language. Flask is easy to use and we all have experience with it. Postman will be used for creating and testing APIs due to its convenience.

• Machine Learning: We decided to go with PyTorch for machine learning since it is one of the most popular libraries. It is also known to have an easier learning curve than other popular libraries such as Tensorflow. This is important because our team lacks ML experience and learning the tool as fast as possible would increase productivity.

• Data Analysis: Some common Python libraries will be used to analyze our data. These include NumPy, Pandas , and matplotlib. These tools combined will help us learn the properties and characteristics of our data. Jupyter notebook will be used to help organize the data analysis process, and improve the code readability.

Client side

• UI: We decided to use React for the UI because it helps organize the data and variables of the application into components, making it very convenient to maintain our dashboard. Since React is one of the most popular front end frameworks right now, there will be a lot of support for it as well as a lot of potential new hires that are familiar with the framework. CSS 3 and HTML5 will be used for the basic styling and structure of the web app, as they are the most widely used front end languages.

• State Management: We decided to use Redux to manage the state of the application since it works naturally to React. Our team also already has experience working with Redux which gave it a slight edge over the other state management libraries.

• Data Visualization: We decided to use the React-based library Victory to visualize the data. They have very user friendly documentation on their official website which we find easy to learn from.

Cache

• Caching: We decided between Redis and memcached because they are two of the most popular open-source cache engines. We ultimately decided to use Redis to improve our web app performance mainly due to the extra functionalities it provides such as fine-tuning cache contents and durability.

Database

• Database: We decided to use a NoSQL database over a relational database because of its flexibility from not having a predefined schema. The user behavior analytics has to be flexible since the data we plan to store may change frequently. We decided on MongoDB because it is lightweight and we can easily host the database with MongoDB Atlas . Everyone on our team also has experience working with MongoDB.

Infrastructure

• Deployment: We decided to use Heroku over AWS, Azure, Google Cloud because it is free. Although there are advantages to the other cloud services, Heroku makes the most sense to our team because our primary goal is to build an MVP.

Other Tools

• Communication Slack will be used as the primary source of communication. It provides all the features needed for basic discussions. In terms of more interactive meetings, Zoom will be used for its video calls and screen sharing capabilities.

• Source Control The project will be stored on GitHub and all code changes will be done though pull requests. This will help us keep the codebase clean and make it easy to revert changes when we need to.

Should I continue learning Django or take this Spring opportunity? I have been coding in python for about 2 years. I am currently learning Django and I am enjoying it. I also have some knowledge of data science libraries (Pandas, NumPy, scikit-learn, PyTorch). I am currently enhancing my web development and software engineering skills and may shift later into data science since I came from a medical background. The issue is that I am offered now a very trustworthy 9 months program teaching Java/Spring. The graduates of this program work directly in well know tech companies. Although I have been planning to continue with my Python, the other opportunity makes me hesitant since it will put me to work in a specific roadmap with deadlines and mentors. I also found on glassdoor that Spring jobs are way more than Django. Should I apply for this program or continue my journey?

I need some advice to choose an engine for generation web pages from the Spring Boot app. Which technology is the best solution today? 1) JSP + JSTL 2) Apache FreeMarker 3) Thymeleaf Or you can suggest even other perspective tools. I am using Spring Boot, Spring Web, Spring Data, Spring Security, PostgreSQL, Apache Tomcat in my project. I have already tried to generate pages using jsp, jstl, and it went well. However, I had huge problems via carrying already created static pages, to jsp format, because of syntax. Thanks.

Which tools are preferred if I choose to work on more data side? Which one is good if I decide to work on web development? I'm using DBeaver and am now considering a move to AzureDataStudio to break the monotony while working. I would like to hear your opinion. Which one are you using, and what are the things you are missing in dbeaver or data studio.

We have a 138 row, 1700 column database likely to grow at least a row and a column every week. We are mostly concerned with how user-friendly the graphical management tools are. I understand MySQL has MySQL WorkBench, and Microsoft SQL Server has Microsoft SQL Server Management Studio. We have about 6 months to migrate our Excel database to one of these DBMS, and continue (hopefully manually) importing excel files from then on. Any tips appreciated!