I have got a small radio service running on Node.js. Front end is written with React and packed with Webpack . I use Docker for my #DeploymentWorkflow along with Docker Swarm and GitLab CI on a single Google Compute Engine instance, which is also a runner itself. Pretty unscalable decision but it works great for tiny projects. The project is available on https://fridgefm.com
We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.
To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas
To build #Webapps we decided to use Angular 2 with RxJS
#Devops - GitHub , Travis CI , Terraform , Docker , Serverless
We are planning to choose Docker since it will allow us to build and install libraries and dependencies with ease. Its extensive use in the world will be helpful to provide us with useful discussion boards. This will be the first time any member of the dev team will be using Docker as part of their application. Given the limited readings, we have been able to do about it in the time we had, we a really excited to get to work with it. It seems to have a lot of potential that we would like to explore as a team. Another reason is that our dev team currently only has access to Windows machines and we want our application to be system agnostic. Using Docker will also help us limit the number of CI minutes our application requires.
We use GraphQL for the communication between our Minecraft-Proxies/Load-Balancers and our global Minecraft-Orchestration-Service JCOverseer.
This connection proved to be especially challenging, as there were so many available options and very specific requirements and we tried our hardest to put as little complexity into this interface as possible.
Initially we considered designing our very own Netty based Packet-Protocol. While the performance of this approach probably would've been noteworthy, we would have had to write a lot of packets as the individual payloads would differ a lot and for the protocol specification a new project would've been needed, so we scrapped that idea.
Our second idea was to use a combination of Redis Key/Value store (in particular the ability to write whole, complex sets as the values of keys) for existing data, Redis Pub-Sub for the synchronization of new/changed/deleted data and a Vert.x based REST API for the mutation requests of the clients. While this would certainly have been possible, we decided against it, as redis offers no real other data types than strings and typing was important to us.
So we finally settled for GraphQL as it would allow us to define dynamic queries and mutations and additionally has subscriptions in store, so we would only need one component instead of three separate. The proxies register as subscribers to the server changes channel and fetch the current data set in advance. If they need to request changes, this is done through a mutation in GraphQL aswell.
The status of the invidiual servers is fetched through Docker healthchecks and a Docker client in the orchestration service, that subscribes to changed HEALTHINESS values in docker. If a service becomes unhealthy it is unregistered and synchronized through GraphQL. The healthcheck is comparable to a ping packet that expects a response in a given time frame.
I'm open to anything. just want something that break less and doesn't need me to pay for it, and can be hosted on Docker. our scripting language is powershell core. so it's better to support it. also we are building dotnet core in our pipeline, so if they have anything related that helps with the CI would be nice.
Our whole DevOps stack consists of the following tools:
- GitHub (incl. GitHub Pages/Markdown for Documentation, GettingStarted and HowTo's) for collaborative review and code management tool
- Respectively Git as revision control system
- SourceTree as Git GUI
- Visual Studio Code as IDE
- CircleCI for continuous integration (automatize development process)
- Prettier / TSLint / ESLint as code linter
- SonarQube as quality gate
- Docker as container management (incl. Docker Compose for multi-container application management)
- VirtualBox for operating system simulation tests
- Kubernetes as cluster management for docker containers
- Heroku for deploying in test environments
- nginx as web server (preferably used as facade server in production environment)
- SSLMate (using OpenSSL) for certificate management
- Amazon EC2 (incl. Amazon S3) for deploying in stage (production-like) and production environments
- PostgreSQL as preferred database system
- Redis as preferred in-memory database/store (great for caching)
The main reason we have chosen Kubernetes over Docker Swarm is related to the following artifacts:
- Key features: Easy and flexible installation, Clear dashboard, Great scaling operations, Monitoring is an integral part, Great load balancing concepts, Monitors the condition and ensures compensation in the event of failure.
- Applications: An application can be deployed using a combination of pods, deployments, and services (or micro-services).
- Functionality: Kubernetes as a complex installation and setup process, but it not as limited as Docker Swarm.
- Monitoring: It supports multiple versions of logging and monitoring when the services are deployed within the cluster (Elasticsearch/Kibana (ELK), Heapster/Grafana, Sysdig cloud integration).
- Scalability: All-in-one framework for distributed systems.
- Other Benefits: Kubernetes is backed by the Cloud Native Computing Foundation (CNCF), huge community among container orchestration tools, it is an open source and modular tool that works with any OS.
So why is your deployment different for your (Heroku) test/dev and your stage/production?
When it comes to testing our web app we do not demand great computational resources and need a very simple, convenient and fast PaaS solution for deploying the app to our testers. In production though, the demand of great computational resources can rise very fast. With Amazon we are able to control that in better way.
We are planning to migrate web application with the same UI AngularJS to #AWS cloud with AWS Elastic Load Balancing (ELB), Jenkins, Docker stack, to check its performance for 200 users. Any suggestions for alternative technologies?
What is the infra required?
Thanks in advance.
I only know Java and so thinking of building a web application in the following order. I need some help on what alternatives I can choose. Open to replace components, services, or infrastructure.
- Frontend: AngularJS, Bootstrap
- Web Framework: Spring Boot
- Database: Amazon DynamoDB
- Authentication: Auth0
- Deployment: Amazon EC2 Container Service
- Local Testing: Docker
- Marketing: Mailchimp (Separately Export from Auth0)
- Website Domain: GoDaddy
- Routing: Amazon Route 53
PS: Open to exploring options of going completely native ( AWS Lambda, AWS Security but have to learn all)
I would recommend to upgrade your stack and consider Angular.
Also, if you are working with docker, instead of manually managing your EC2 and docker inside it, switch to ECS as its free of cost and hassle free way to deploy and keep running your containers efficiently.
Instead of Docker , no doubt its great but it has vulnerabilitis and restricitions with dameon and root thread. I would pickup Podman. Also Ambasador is a culmination of Gateway LB and ServiceMesh on istio and Envoy. Great for both East-west and North south microservices communication, policy managment and security with Istio. Spring Boot is not a WebFW. For platform web fw one can use Reactive like SPring WebFlow rather than Spring MVC. For java experience, Spring provides great assets.
I will switch to using Kubernetes whether managed or custom depends on several factors rather than AWS ecs. For LB Amabassador is a great alternative on AWS. One can simply use this on top of ECS clusters. Instead of running in to different frameworks one can simply use one FW at both client and server side for consuming and SSE. I believe one can look at Lot of it depends what you need a full FW or a light librarry like React to be part of V in your MVC. Whether you need a SPA , on Mobile etc... in that case KOTLIN is also another option on Java. Dont go with Android. Best luck. Swapnil S
We would like to connect a number of (about 25) video streams, from an Amazon S3 bucket containing video data to endpoints accessible to a Docker image, which, when run, will process the input video streams and emit some JSON statistics.
The 25 video streams should be synchronized. Could people share their experiences with a similar scenario and perhaps offer advice about which is better (Wowza, Amazon Kinesis Video Streams) for this kind of problem, or why they chose one technology over the other?
The video stream duration will be quite long (about 8 hours each x 25 camera sources). The 25 video streams will have no audio component. If you worked with a similar problem, what was your experience with scaling, latency, resource requirements, config, etc.?
I have different experience with processing video files that I'll describe below. It might be helpful or at least make you think a bit diffferent about the problem. What I did (part of it is a mistake): To increase the level of parallelism at the time consuming step which was the video upload, using a custom cmd tool written in Python, I splitted the input videos to much smaller chunks (without losing their ordering - just file name labeling with timestamp) . It then uploaded the chunks to S3. That triggered a few Lambdas that each first pulled a chunked video, did the processing with ffmpeg (the Lambdas were the mistake - at that time the local Lambda storage was up to 512MB so lots of chunks and lots of Lambdas had to be in place, also Lambda are hell to debug), later called Rekognition and later using AWS Elemental MediaConvert to rebuild the full length video. I would use some sort of ECS deployment where processing is triggered by S3 event, and scale the number of Fargate nodes dependent on the number of chucks/videos. Then each processor pulls its video (not stream) to its local storage (local EBS drive) and works. I failed to understand why are you trying to stream videos that are basically static, as a file, or that putting the files on S3 is a current limitation (while your input videos are 'live' and streaming) that you're trying to remove ?