Python helps us automate the tedious and has the gold standard Natural Language Processing library. Python
I really love Django because it is really fast to create a web application from scratch and it has a lot a facilities like the ORM or the Admin module ! The Python language is really easy to read and powerful, that's why I prefer Django over Symfony.
I use Django at work to make tools for the technicians but I also use it for me to build my personal website which I host on PythonAnywhere, and with a domain name bought on Namecheap.
Context: I wanted to create an end to end IoT data pipeline simulation in Google Cloud IoT Core and other GCP services. I never touched Terraform meaningfully until working on this project, and it's one of the best explorations in my development career. The documentation and syntax is incredibly human-readable and friendly. I'm used to building infrastructure through the google apis via Python , but I'm so glad past Sung did not make that decision. I was tempted to use Google Cloud Deployment Manager, but the templates were a bit convoluted by first impression. I'm glad past Sung did not make this decision either.
Solution: Leveraging Google Cloud Build Google Cloud Run Google Cloud Bigtable Google BigQuery Google Cloud Storage Google Compute Engine along with some other fun tools, I can deploy over 40 GCP resources using Terraform!
Check Out My Architecture: CLICK ME
Check out the GitHub repo attached
Relly like the architectural artefact, what did you use to produce it?
I used draw.io. Free and easy to use!
Architecture looks great! Just out of curiosity, did you have unit and integration testing components for Terraform e.g. Terratest, Kitchen Terraform, Sentinel?
I did NOT. This was to demo the mechanics. Not something for robust production. Would love if you can make a pull request for some of those tests ;)
We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.
To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas
To build #Webapps we decided to use Angular 2 with RxJS
#Devops - GitHub , Travis CI , Terraform , Docker , Serverless
I'm working as one of the engineering leads in RunaHR. As our platform is a Saas, we thought It'd be good to have an API (We chose Ruby and Rails for this) and a SPA (built with React and Redux ) connected. We started the SPA with Create React App since It's pretty easy to start.
We use Jest as the testing framework and react-testing-library to test React components. In Rails we make tests using RSpec.
Our main database is PostgreSQL, but we also use MongoDB to store some type of data. We started to use Redis for cache and other time sensitive operations.
We have a couple of extra projects: One is an Employee app built with React Native and the other is an internal back office dashboard built with Next.js for the client and Python in the backend side.
Python is actually the first decision we made in our stack selection process. The entire dev team is familiar with the language and more importantly, it is the language of choice for most of the leading machine learning research and applications. Another thing that we considered is that using python allows us to more easily hire developers in the future. Python is generally the kind of language in which it is really easy to get something started with minimal effort, which is ideal for us given our short timeline
I need your help in choosing either Spring Boot or ASP.NET Core. Both frameworks seem to have mature ecosystems. I would like to hear your thoughts on the following points:
- Difficulty level of both frameworks
- Level of community support
- Career prospects i.e do Spring based jobs pay more or vice versa
- which one will be helpful if I decide to transition towards a more specialized field like data engineering.
I am asking this because it is something that I am also exploring in parallel. I know that Python and #SQL play a huge role in big data.
I'll preface this by saying I haven't ever done work on the Microsoft stack, so take this with a grain of salt.
Community support: Java tends to have a more active community (and much more diverse universe of new open source projects) than .Net (related to below).
Difficulty level: I'd say the Java/Spring stack is more difficult because Java developers tend to use more community projects and have to know which ones to choose in which circumstances (e.g. what logging framework to use? What database connection pooling library to use? What testing framework to use). That being said, most decisions you have to make have choices that are vastly more popular than others. My impression is that .NET guys use stuff from the standard library almost exclusively. If it's not provided there, it's a problem they never thought to solve or maybe write their own (usually the former).
Career prospects: I'll get hate for this probably, but Java/Spring has more jobs than .Net. .NET Core even more so because it's new. Most .NET jobs out there are going to be not Core, and not on anything other than Windows. LinkedIn search for jobs in United States shows 97,103 for Java and 36,448 for C#. That's not to say some individual city might not have more .NET than Java, but by and large Java is bigger. Also, .NET is not dying at the same pace as say Ruby on Rails (sorry Rails fans). I'd say it's ticking very slowly lower, maybe even holding steady. I wouldn't say you're screwed career-wise if you choose .NET. C# is also pretty similar to Java from what I've seen, and I know professionals that have transitioned to Java (though interestingly, I don't know any that did the reverse). Several companies have basically no .NET footprint, for example, Amazon, Google, Facebook, LinkedIn, Twitter all have Java, but basically no C#.
Specialized fields: Neither Spring or .Net are going to be a thing in a field like data science. That's exclusively Python (some R) for the most part. The only exception being that there is some Java in Hadoop, and Scala in Spark (which runs on the JVM). But I think those are generally more for data products being created, and not data science work. I'm also under the impression this is increasingly less the case than historically. Some other specialties might make Java useful, for example Android development.
In short, Java is the COBOL of the modern enterprise (which is both a good and bad thing). I recommend it to anyone over .NET, but not for technical reasons. It's for reasons related to the questions you asked. There are actually reasons I think CLR and C# are actually better from a technical perspective than Java (unsurprisingly, since they had the benefit of hindsight). But that's not what you were asking about...
Hearing you say " Java is the COBOL of the modern enterprise" sounds kind of depressing....
What I mean by that is that it's ubiquitous and stable. And that I'm maybe a touch cynical after being a Java guy for over a decade.
Innovation has gotten better under Oracle. Sun kinda let the language languish from a lack of new features (e.g. functional programming). But the last several years have picked up and I think has made companies that were looking at Scala or whatever alternative languages decide that the updated Java was good enough and they no longer needed an alternative.
Let's be honest though, all these languages (Python, C#, Java) aren't exciting. They've been around a really long time and are stable workhorses. But that's not a bad thing. A language that has a lot of exciting new features is just gonna break my shit more likely than not.
I'd also add that Python is chipping away at Java's dominance it seems (although it's not happening in the city I live in). So good on you for having that in your toolbelt.
Python for sure has been dominating for the several years. However, I've heard that when it comes to the field of data engineering, Java has more edge. Perhaps a lot of tools like Apache Spark and Kafka are based on it and it also the ability to handle large scale data, but I am not sure about it. Another reason of me asking for Spring/ASP.NET Core is the nature of Java and C#. Knowing a statically typed language (and the relevant frameworks) forces you
to think about your code in more cautious manner. They give a different perspective about writing code. But again, these are just my thoughts.
That is true. Java/Scala are pretty big in data intensive applications, though this is a fairly highly specialized area. Since most data science is done in Python, I think fairly often folks just run their stuff in Python and even though it's not all that performant, it's often good enough. Though there's definitely overlap in that problem-space.
It is useful to understand how both dynamic and statically typed languages work. But I wouldn't say a statically typed language makes you have to be more cautious, I'd say it eliminates a category of problems (things not being the type you expect at runtime -- you always know exactly what types you're dealing with). Java is rather verbose in its syntax, but that's not strictly related to it's lack of dynamic types. Scala is much less verbose, but also lacks dynamic types (but it does have type inference, which is one of the things that reduces the amount of ceremony). Static typing also have better tooling (it's easier for an IDE to autocomplete stuff, etc), and better performance (in theory, someone probably could create a dynamically typed language that performs as well as statically typed languages, but it has never happened yet).
Spring boot helps you creating microservices in hours, not days and there is a very active community around it with amazing integrations. Check one of the tutorials maybe. At least here in Germany, the job market will be better for Spring Boot as well, there are a lot more companies using Java then C#.
Objective: I am trying to build a custom service that will create VMs in Azure, based on inputs taken from a web interface. I want the backend code that interacts with Azure to be PowerShell.
Ask: Hoping to find help with deciding the simplest architecture of tools to achieve this.
What I have so far with my Limited Knowledge: I am new to Azure and Jenkins. I arrived at Jenkins coz it can run PowerShell and has API that can be called to trigger a job. Although integrating with it over the web seems problematic since its on-prem network. I hear it is possible using the VPN. For the Web, I hope to use Azure Web App with Python/Node.js that I can manage to make API calls to Jenkins.
Is there a better way? I just need help getting the right directions; I will walk the way.
If you want to use Azure Web Apps and want to keep it all in one ecosystem then easiest would be to use Azure Devops for the pipelines and Azure Functions to run the PowerShell code. Azure DevOps is a CI/CD platform(and more) kind of like Jenkins but integrated into Azure. And Azure Functions is a serverless thing that can run your code(your PowerShell) on demand without the need to have it hosted on your server. Jenkins can be nice but you will have to host it somewhere. If you want you could use Azure to set up a VM that hosts Jenkins and the website Frontend. Technically if you have the website, it's backend language should be able to execute scripts that are on the same machine so you wouldn't even need Jenkins or another pipeline solution. It would be recommended though for easier management.
I would have a look at "infrastructure as code" solutions like Hashicorp Terraform or Ansible from Redhat that support declarative and/or imperative approaches in the provisioning of cloud resources.
Going from your original requirement, that you want to build a web interface to provision new Azure VMs, your stack seems to be overly complicated. Why would you need to use Jenkins and Powershell at all? First Jenkins is a Continous Integration Server, you don‘t need it. Second Powershell is a Shell-Script, you can interact with Azure through Powershell but you don‘t need to. As others said, don‘t build your backend service in Powershell, if you can make it work, it will not be maintainable.
You mention you want to build the web interface itself on Python or Node.js, both are fine, just use the one that works best for you. Both allow to you to create a backend service that can make the API-calls to the Azure-API to create your VMs.
You must have gone through multiple thoughts on how to handle the requirement, but the biggest problem that I am able to see here is a dependency on a certain platform for executing Powershell scripts. Next is being a developer will never suggest having Powershell or bash as a backend of your system until it is related to some IoT Stuff. In place of it, I would rather suggest you use Infrastructure as a Code that comes with terraform or Ansible or as your application is based on Azure so u can use Azure Resource Manager too.
Adding to it for making config or attaching a trigger to these infrastructure controllers, you can use lightweight stateful APIs with the help of flask or express or go-mux, to keep track on the progress of deployments.
Let me tell you this: There is never a wrong time to start leaning. So, when you have time I highly recommend that you download the module "TKInter". You can install it through pip. This module will help you in developing basic GUI's.
But wait! If you want to develop apps on android or iOS, python won't help. For android, you need to learn Kotlin or Java, for iOS you need to learn Swift. But don't worry, with python you can make apps for windows. When you learn how to make a good GUI and eventually make a python app, you can easily convert it into a .exe file for windows compatibility. Learn how to convert it here https://www.geeksforgeeks.org/convert-python-script-to-exe-file/