Stack Advice

Avatar of jigarshah2811
Security Engineer at Pinterest

We would like to detect unusual config changes that can potentially cause production outage.

Such as, SecurityGroup new allow/deny rule, AuthZ policy change, Secret key/certificate rotation, IP subnet add/drop. The problem is the source of all of these activities is different, i.e., AWS IAM, Amazon EC2, internal prod services, envoy sidecar, etc.

Which of the technology would be best suitable to detect only IMP events (not all activity) from various sources all workload running on AWS and also Splunk Cloud?

READ MORE
7 upvotes26.2K views
Replies (5)
Avatar of natiabebe12
Cloud Architect at AWS
Recommends
AWS Config

For continuous monitoring and detecting unusual configuration changes, I would suggest you look into AWS Config.

AWS Config enables you to assess, audit, and evaluate the configurations of your AWS resources. Config continuously monitors and records your AWS resource configurations and allows you to automate the evaluation of recorded configurations against desired configurations. Here is a list of supported AWS resources types and resource relationships with AWS Config https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html

Also as of Nov, 2019 - AWS Config launches support for third-party resources. You can now publish the configuration of third-party resources, such as GitHub repositories, Microsoft Active Directory resources, or any on-premises server into AWS Config using the new API. Here is more detail: https://docs.aws.amazon.com/config/latest/developerguide/customresources.html

If you have multiple AWS Account in your organization and want to detect changes there: https://docs.aws.amazon.com/config/latest/developerguide/aggregate-data.html

Lastly, if you already use Splunk Cloud in your enterprise and are looking for a consolidated view then, AWS Config is supported by Splunk Cloud as per their documentation too. https://aws.amazon.com/marketplace/pp/Splunk-Inc-Splunk-Cloud/B06XK299KV https://aws.amazon.com/marketplace/pp/Splunk-Inc-Splunk-Cloud/B06XK299KV

READ MORE
9 upvotes15.8K views
Avatar of isaacpovey
Casual Software Engineer at Skedulo
Recommends
Terraform

While it won't detect events as they happen a good stop gap would be to define your infrastructure config using terraform. You can then periodically run the terraform config against your environment and alert if there are any changes.

READ MORE
6 upvotes1 comment17.7K views
View all (5)
Needs advice
on
React
PostgreSQL
and
ExpressJS

We're a non-profit looking to rebuild our system.

Hey everyone - this seems like a little bigger project then I've seen on here! I tried to give a concise description of the project. We haven't started, but this is the general idea for our tech stack and what we're planning!

Description:

  • We handle all sorts of features within the medical supply chain, from country-wide warehouses to front-line health facilities.
  • We are in around 30 developing countries, and it would be essential to migrate our clients, who currently have their 4D relational databases, to our new system.

Our current system has:

  • A single central server, with many clients/servers scattered throughout a country. Client servers are in hospitals, pharmacies, health centers, small regional warehouses. The main server is in a country's main warehouse.
  • Countries are independent and run their own system.
  • Countries operate on what can only be described as .. extremely poor internet (assume 56k modem in the middle of the Amazon)
  • Our system is a transactional system run all through the same 4D database, which has a lot of extra features:
    • Allows clients to create orders to other members of the system - i.e., hospital->warehouse.
    • Allows dispensing medicines to patients.
    • Allows managing where stock is in a facility.
    • Manage expiry dates, stocktakes, inventory adjustments
    • Reporting on stock movements through the single 4D database

Main requirements are:

  • Extensibility. Due to being in so many countries and expanding, we have a lot of customizations. We'd most likely be wanting to look into some sort of plugin system.
  • Eventual consistency & availability. Due to operating in developing countries, we use offline-first filtered replication throughout the system and require each of the client servers to have (close to!) 100% availability of data, regardless of their network connection. We need eventual consistency in the main server so that we can report country-wide.
  • Scalability. We don't need to scale to massive amounts. Our maximum capacity looks to be a country around 200 million. That's about.. 1000 million records every year syncing into the main server.

Current thoughts: We're currently looking to run a PostgreSQL instance on each of the client servers. Each client server running an ExpressJS server. React as a front end. We are looking to roll our own sync system through most likely -a RESTful API.

This is a pretty vague and small description. I was wondering if anyone had any thoughts on a tech stack?

READ MORE
4 upvotes10.9K views
Replies (4)
Recommends
PostgreSQL

I think your choice of technologies are fine, but it you are starting new development I think your choice of architecture is most important. You can implement the desired architecture with many different technologies. Separating the layers of the application is important to allow each technology to be replaced or evolve independently. For example, I would write the code in a way that I could replace PostgreSQL with a different DBMS fairly easily in case a better alternative came along. The same goes for the React front end. If you have a good REST API you could replace the React front any with any framework, you could easily allow different locations to use Angular, Vue, or other front end technologies as long as the REST API is not designed to specifically target the React front end. I think the architecture of the code behind the REST layer, that would run in Express (the tool/language you are considering) is the decision that you will be stuck with for the longest time and that all other aspects of the application need to work around; so you want to get that architecture right. I think you should consider CQRS and Event Sourcing. https://docs.microsoft.com/en-us/azure/architecture/patterns/event-sourcing https://dzone.com/articles/cqrs-and-event-sourcing-intro-for-developers

READ MORE
8 upvotes7.7K views
Avatar of kit9496
CTO at Recordsure
Recommends
Amazon DynamoDB

I would really avoid building all of the scaffolding you need for the global server-server replication, eventual consistency and offline tools.

As long as your application architecture can fits a Key/Value pair type store, and you want master-master replication, then I'd start with AWS DynamoDB configured in Global Tables mode: https://aws.amazon.com/dynamodb/global-tables/

Then for offline-first access from client devices, use AWS Amplify, where (when used right) it's datastore feature will do most of the online/offline sync for you and let the developers focus on the mobile or web app. https://aws.amazon.com/amplify/features/#DataStore

I've had great success in using Amplify over very spotty connections (moving vehicles), using React and React Native for Web and Mobile applications respectively.

If your back-end data architecture really must be more RDBMS/SQL like, then consider whether you can use a central database in one geography, with all updates finding their way back there, then with read-replicas in other geographies. A good article on this, and suitable cloud-first databases to use, is here: https://read.acloud.guru/why-and-how-do-we-build-a-multi-region-active-active-architecture-6d81acb7d208

READ MORE
4 upvotes5.6K views
View all (4)
Avatar of dhAlcojor
Frontend Developer at atSistemas
Needs advice
on
UIkIt
Bulma
and
Bootstrap

I'm building, from scratch, a webapp. It's going to be a dashboard to check on our apps in New Relic and update the Apdex from the webapp. I have just chosen Next.js as our framework because we use React already, and after going through the tutorial, I just loved the latest changes they have implemented.

But we have to decide on a CSS framework for the UI. I'm partial to Bulma because I love that it's all about CSS (and you can use SCSS from the start), that it's rather lightweight and that it doesn't come with JavaScript clutter. One of the things I hate about Bootstrap is that you depend on jQuery to use the JavaScript part. My boss loves UIkIt, but when I've used it in the past, I didn't like it.

What do you think we should use? Maybe you have another suggestion?

READ MORE
8 upvotes91.8K views
Replies (6)
Avatar of DamienLucchese
Front End Web Developer
Recommends
Bulma

Been checking out Bulma, myself, and really dig it. I like that it's a great base level jumping off point. You can get a layout going with it, pretty quickly, and then customize as you want. It definitely sounds like it's the one you're leaning towards but a big factor would be who will be using it most? Your boss, yourself, others? Whichever you like best, you'll prob be most productive with but if in the end your boss says it has to be UIkit, then best to be open-minded and give it another shot. Sometimes you may not jive with new tools in your stack, at first, but then they can become tools you learn to love. Best to you in your decision! Take care & keep safe.

READ MORE
6 upvotes68.7K views
Recommends
UIkIt

I have used bulma in several projects. We could not customize with the websites very well. Also when we need "quick solutions" Bulma is not suitable (I mean basic animations, to-top buttons, transparent navbar solutions etc. For these solutions, you need extra js codes).

Everybody knows about Bootstrap (heavy but popular).

Now we start a new project with UI kit, I like it. Pros: It is fast and lightweight and imho it has very good UI. Cons: Small community. Documentation.

Check this link for kick-off. https://github.com/zzseba78/Kick-Off

Maybe it is helpful.

READ MORE
6 upvotes68.9K views
View all (6)
Avatar of prithvi97singh7095
Application Developer at Montaigne Smart Business Solutions
Needs advice
on
PostgreSQL
MySQL
and
MongoDB

I am going to work on a real estate project and have to decide on a database. Now, SQL databases can be very efficient if appropriately designed. More relations between the data and less redundancy. But with a #NoSQL database, the development time is reduced, and it is easy to query. Since this is my first time working on the real estate domain, I would like to pick a database that would be efficient in the long run.

READ MORE
8 upvotes66.8K views
Replies (4)
Avatar of aricfedida
Founder, CTO at ASK Technologies Inc
Recommends
PostgreSQL

I recommend PostgreSQL as it鈥檚 the most powerful out of the 3 databases you mentioned. It supports JSON objects so you can mimic the MongoDB functionality, but I would also argue that SQL is actually quite powerful and in many cases significantly easier to work with than with NoSQL databases.

Stay away from foreign keys, keep it fast and simple. Define your data structures well in advance. Try to model your data structures based on your system鈥檚 vision; based on where it鈥檚 going and not based solely on what you currently need it to do. This will help you avoid drastic changes to your database after your system is launched. Populate the database with fake data and run tests. PostgreSQL allows you to create Views from multiple tables. Try to create those views and make sure you can easily create useful views from multiple tables. Run an Explain on those view queries to make sure you created your indexes correctly. Make sure it鈥檚 fast!

READ MORE
15 upvotes58.7K views
Recommends
PostgreSQL

Any of those three databases are going to be efficient, scalable, and reliable in the long term if you configure and use them correctly. They all also have solid hosting solutions.

All things being equal, I would agree with other posters that Postgres is my preference among the three, but there are caveats.

MongoDB and MySQL have better support for mutli-region replication in your big three cloud environments. Azure recently bought Citus Data, which was a best-in-class Postgres replication solution, so they might be the only one I trust to provide cross-region replication at the moment.

If you have a single region deployment and are on AWS, I can't recommend Aurora Postgres highly enough. It's a very good implementation and extremely performant.

READ MORE
5 upvotes56K views
View all (4)
Avatar of christyciss7625
Program Manager
Needs advice
on
PyCharm
IntelliJ IDEA
and
Eclipse

UPDATE: Thanks for the great response. I am going to start with VSCode based on the open source and free version that will allow me to grow into other languages, but not cost me a license ..yet.

I have been working with software development for 12 years, but I am just beginning my journey to learn to code. I am starting with Python following the suggestion of some of my coworkers. They are split between Eclipse and IntelliJ IDEA for IDEs that they use and PyCharm is new to me. Which IDE would you suggest for a beginner that will allow expansion to Java, JavaScript, and eventually AngularJS and possibly mobile applications?

READ MORE
9 upvotes402.6K views
Replies (12)
Recommends
PyCharm

The Community version of PyCharm is free and should give you what you need to get started with Python. Both PyCharm and IntelliJ are made by JetBrains. IntelliJ is initially focused on Java but you can get plugins for lots of other things. I subscribe to JetBrains' Toolbox: https://www.jetbrains.com/toolbox-app/ and have access to all of their great tools.

READ MORE
13 upvotes71.2K views
Avatar of vvvlad423985
Dev at Intel

Pycharm is great for python development, but can feel sometimes slow and community version has Somme very annoying restrictions (like they disabled jupyter notebooks plugin and made it premium feature). I personally started looking into VS Code as an alternative, and it has some very good potential. I suggest you take it into account.

READ MORE
12 upvotes71.2K views
View all (12)
Avatar of prateek964
Senior Associate at PwC
Needs advice
on
React
Node.js
and
MongoDB
in

Hi, I'm very new to Web Dev, and I'm building a simple text-based website with multiple pages using the MERN stack. But when I switch from one page to another, the load time is slow. I came across this website of JS Library Reveal (https://revealjs.com/installation/), which has similar functionality of switching from one page to another, but it renders the webpages at an amazing speed. Please suggest that how can I improve my page rendering speed?

READ MORE
4 upvotes3.5K views
Replies (4)
Avatar of killshot13
Full Stack Web Developer at Safe This Home, LLC
Recommends
Next.js

Since you are using a MERN stack, you could consider integrating Next.js into your frontend and rendering your pages on the server-side. This does require a slightly more advanced configuration, but with worthwhile benefits. With server-side rendering, you will obtain quicker load times, less configuration, and components with built-in routing.

Next.js is fully supported by React; the framework is even featured in React's list of recommended toolchains for building server-side PWA's with Node. I recommend at least having a look at their documentation and/or toying with some of the examples on GitHub to better understand the framework and decide if it will work for you.

READ MORE
4 upvotes241 views
Recommends
Postman

If you're working locally, the issue may be that your React app is entering an "infinite request loop" when loading a new page. When I started with React, it took me a while to figure out how to properly use the UseEffect hook, and so some pages would make an endless series of requests to the APi... causing the app to load slowly and sometimes crash.

To check whether or not this is the problem, open your developer console in Chrome, go to the "Network" tab, and make sure that the client isn't making an endless series of requests to the backend. Refresh the page, with the Network tab open, to check the requests your client is making to the API.

To check whether or not the issue is with your API, you can test your endpoints using a tool like Postman... see if some specific request is causing you problems.

If the site is working fine locally, but has slowed down after deployment, then the issue is probably your hosting as Andrew has pointed out.

READ MORE
3 upvotes922 views
View all (4)
Needs advice
on
Stylelint
Sass Lint
and
ESLint

Scenario: I want to integrate Prettier in our code base which is currently using ESLint (for .js and .scss both). The project is using gulp.

It doesn't feel quite right to me to use ESLint, I wonder if it would be better to use Stylelint or Sass Lint instead.

I completed integrating ESLint + Prettier, Planning to do the same with [ Stylelint || Sasslint || EsLint] + Prettier.

And have gulp 'fix' on file save (Watcher).

Any recommendation is appreciated.

READ MORE
7 upvotes30.4K views
Replies (3)
Avatar of amalv
Senior Frontend Developer at Landbot.io
Recommends
ESLint

In the case of .js files I would recommend using both Eslint and Prettier.

You can set up Prettier as an Eslint rule using the following plugin:

https://github.com/prettier/eslint-plugin-prettier

And in order to avoid conflicts between Prettier and Eslint, you can use this config:

https://github.com/prettier/eslint-config-prettier

Which turns off all Eslint rules that are unnecessary or might conflict with Prettier.

READ MORE
6 upvotes1 comment16.4K views
Avatar of Alexisvt
Software Engineer at SpeedUrWeb
Recommends
Stylelint

Pura vida! Well, I had a similar issue and at the end I decided to use Stylelint + Prettier for that job, in our case, we wanted that our linting process includes the SCSS files and not only the JS file, base on that we concluded that using only ESLint to do both things wasn't the best option, so, we integrated prettier with Stylelint, and for that we used a neat plugin that allowed us to use Prettier inside Stylelint here is the link, https://github.com/prettier/stylelint-prettier#recommended-configuration, I hope that this can help you, hasta pronto!, :)

READ MORE
5 upvotes1 comment16.4K views
View all (3)
Needs advice
on
slate
and
draft-js

draft-js is backed by FB, while slate is maintained by a person. While the stars are the same, I want some suggestions.

READ MORE
2 upvotes60 views
Replies (3)
Avatar of jdspugh
Software Engineer / Project Manager / Technical Architect

I haven't used slate or draft-js but I have used many other tools including those made by big companies such as Facebook, and by individuals. What I find is the tools made by big companies are designed for big company workflows i.e. big teams where each person needs an abstraction layer for separation from the other team members so they can work on their portion of the project independently. As a consequence the resulting applications tend to be more heavy and during development they have many parts that need to be learned. Which is fine if you have a team and each person only needs to learn their portion of the system.

Tools made by individuals tend to target solo developers. The tools are usually more compact and may require the developer to have a wider variety of skills but the learning curve will be lower than learning all the layers of a big company's tool. The resulting application is likely to be lighter also.

So I would say that if you're a solo developer, go for the tools made by individual developers. If you are a team, or planning to be a team, where each person specialises in a particular skillset, use tools made by big companies.

READ MORE
2 upvotes44 views
Avatar of cwray-tech
Owner, Developer at Soltech LLC
Recommends
slate
at

From what I can see, slate would be more for people that want a built, ready-made solution.

draft-js would be more for organizations that have more unique needs and want to have complete control over what they do.

If I was just interested in popping in a text editor for my project, I would go with slate.

READ MORE
2 upvotes2 comments41 views
View all (3)
Needs advice
on
Redis
RabbitMQ
and
Kafka

We are going to develop a microservices-based application. It consists of AngularJS, ASP.NET Core, and MSSQL.

We have 3 types of microservices. Emailservice, Filemanagementservice, Filevalidationservice

I am a beginner in microservices. But I have read about RabbitMQ, but come to know that there are Redis and Kafka also in the market. So, I want to know which is best.

READ MORE
9 upvotes100.1K views
Replies (4)
Avatar of alurimahee
Lead Architect
Recommends
Kafka

Kafka is an Enterprise Messaging Framework whereas Redis is an Enterprise Cache Broker, in-memory database and high performance database.Both are having their own advantages, but they are different in usage and implementation. Now if you are creating microservices check the user consumption volumes, its generating logs, scalability, systems to be integrated and so on. I feel for your scenario initially you can go with KAFKA bu as the throughput, consumption and other factors are scaling then gradually you can add Redis accordingly.

READ MORE
6 upvotes62.9K views
Avatar of amitmormail7948
Software Architect at Payoneer
Recommends
Kafka

I think something is missing here and you should consider answering it to yourself. You are building a couple of services. Why are you considering event-sourcing architecture using Message Brokers such as the above? Won't a simple REST service based arch suffice? Read about CQRS and the problems it entails (state vs command impedance for example). Do you need Pub/Sub or Push/Pull? Is queuing of messages enough or would you need querying or filtering of messages before consumption? Also, someone would have to manage these brokers (unless using managed, cloud provider based solution), automate their deployment, someone would need to take care of backups, clustering if needed, disaster recovery, etc. I have a good past experience in terms of manageability/devops of the above options with Kafka and Redis, not so much with RabbitMQ. Both are very performant. But also note that Redis is not a pure message broker (at time of writing) but more of a general purpose in-memory key-value store. Kafka nowadays is much more than a distributed message broker. Long story short. In my taste, you should go with a minialistic approach and try to avoid either of them if you can, especially if your architecture does not fall nicely into event sourcing. If not I'd examine Kafka. If you need more capabilities than I'd consider Redis and use it for all sorts of other things such as a cache.

READ MORE
3 upvotes58.7K views
View all (4)

Hello everyone,

I am currently on an internship, and I am a new intern in an SME. My first mission is to choose the right tool for predictive sales analysis (management of the quantity in stock). I found several tools (paying and open source), and the company leaves the choice of tools to me (even paying). They suggest SAP Analytics Cloud as a first attempt (since we want a tool on the cloud too). I would like to have your proposals since I'm new to the business.

PS: I code in Python !! thank you in advance.

READ MORE
6 upvotes4.7K views
Replies (3)
Avatar of CircleOnCircles
Director at Codustry

In sales analysis, you might need some sort of timeseries prediction. I would recommend the sagemaker DeepAR. where you could co-op the seasonal effect into the model.

READ MORE
4 upvotes4.5K views
Avatar of pthiran
Research & Technology & Innovation | Software & Data & Cloud | Professor in Computer Science

Hello Amina, You need first to clearly identify the input data type (e.g. temporal data or not? seasonality or not?) and the analysis type (e.g., time series?, categories?, etc.). If you can answer these questions, that would be easier to help you identify the right tools (or Python libraries). If time series and Python, you have choice between Pendas/Statsmodels/Serima(x) (if seasonality) or deep learning techniques with Keras.

Good work, Philippe

READ MORE
3 upvotes884 views
View all (3)