Feed powered byStream Blue Logo Copy 5
MongoDB

MongoDB

Application and Data / Data Stores / Databases

Decision at FundsCorner about Zappa, AWS Lambda, SQLAlchemy, Python, Amazon SQS, Node.js, MongoDB Stitch, PostgreSQL, MongoDB

Avatar of jeyabalajis
CTO at FundsCorner
ZappaZappa
AWS LambdaAWS Lambda
SQLAlchemySQLAlchemy
PythonPython
Amazon SQSAmazon SQS
Node.jsNode.js
MongoDB StitchMongoDB Stitch
PostgreSQLPostgreSQL
MongoDBMongoDB

Recently we were looking at a few robust and cost-effective ways of replicating the data that resides in our production MongoDB to a PostgreSQL database for data warehousing and business intelligence.

We set ourselves the following criteria for the optimal tool that would do this job: - The data replication must be near real-time, yet it should NOT impact the production database - The data replication must be horizontally scalable (based on the load), asynchronous & crash-resilient

Based on the above criteria, we selected the following tools to perform the end to end data replication:

We chose MongoDB Stitch for picking up the changes in the source database. It is the serverless platform from MongoDB. One of the services offered by MongoDB Stitch is Stitch Triggers. Using stitch triggers, you can execute a serverless function (in Node.js) in real time in response to changes in the database. When there are a lot of database changes, Stitch automatically "feeds forward" these changes through an asynchronous queue.

We chose Amazon SQS as the pipe / message backbone for communicating the changes from MongoDB to our own replication service. Interestingly enough, MongoDB stitch offers integration with AWS services.

In the Node.js function, we wrote minimal functionality to communicate the database changes (insert / update / delete / replace) to Amazon SQS.

Next we wrote a minimal micro-service in Python to listen to the message events on SQS, pickup the data payload & mirror the DB changes on to the target Data warehouse. We implemented source data to target data translation by modelling target table structures through SQLAlchemy . We deployed this micro-service as AWS Lambda with Zappa. With Zappa, deploying your services as event-driven & horizontally scalable Lambda service is dumb-easy.

In the end, we got to implement a highly scalable near realtime Change Data Replication service that "works" and deployed to production in a matter of few days!

17 upvotes6.1K views

Decision about MongoDB, GraphQL, Node.js

Avatar of juank11memphis
MongoDBMongoDB
GraphQLGraphQL
Node.jsNode.js

I just finished the very first version of my new hobby project: #MovieGeeks. It is a minimalist online movie catalog for you to save the movies you want to see and for rating the movies you already saw. This is just the beginning as I am planning to add more features on the lines of sharing and discovery

For the #BackEnd I decided to use Node.js , GraphQL and MongoDB:

  1. Node.js has a huge community so it will always be a safe choice in terms of libraries and finding solutions to problems you may have

  2. GraphQL because I needed to improve my skills with it and because I was never comfortable with the usual REST approach. I believe GraphQL is a better option as it feels more natural to write apis, it improves the development velocity, by definition it fixes the over-fetching and under-fetching problem that is so common on REST apis, and on top of that, the community is getting bigger and bigger.

  3. MongoDB was my choice for the database as I already have a lot of experience working on it and because, despite of some bad reputation it has acquired in the last months, I still believe it is a powerful database for at least a very long list of use cases such as the one I needed for my website

10 upvotes6 comments2.6K views

Decision at FundsCorner about Amazon SQS, Sentry, GitLab CI, Slack, Google Compute Engine, Netlify, AWS Lambda, Zappa, vuex, Vuetify, Vue.js, Swagger UI, MongoDB, Flask, Python

Avatar of jeyabalajis
CTO at FundsCorner
Amazon SQSAmazon SQS
SentrySentry
GitLab CIGitLab CI
SlackSlack
Google Compute EngineGoogle Compute Engine
NetlifyNetlify
AWS LambdaAWS Lambda
ZappaZappa
vuexvuex
VuetifyVuetify
Vue.jsVue.js
Swagger UISwagger UI
MongoDBMongoDB
FlaskFlask
PythonPython

At FundsCorner, we are on a mission to enable fast accessible credit to India鈥檚 Kirana Stores. We are an early stage startup with an ultra small Engineering team. All the tech decisions we have made until now are based on our core philosophy: "Build usable products fast".

Based on the above fundamentals, we chose Python as our base language for all our APIs and micro-services. It is ultra easy to start with, yet provides great libraries even for the most complex of use cases. Our entire backend stack runs on Python and we cannot be more happy with it! If you are looking to deploy your API as server-less, Python provides one of the least cold start times.

We build our APIs with Flask. For backend database, our natural choice was MongoDB. It frees up our time from complex database specifications - we instead use our time in doing sensible data modelling & once we finalize the data model, we integrate it into Flask using Swagger UI. Mongo supports complex queries to cull out difficult data through aggregation framework & we have even built an internal framework called "Poetry", for aggregation queries.

Our web apps are built on Vue.js , Vuetify and vuex. Initially we debated a lot around choosing Vue.js or React , but finally settled with Vue.js, mainly because of the ease of use, fast development cycles & awesome set of libraries and utilities backing Vue.

You simply cannot go wrong with Vue.js . Great documentation, the library is ultra compact & is blazing fast. Choosing Vue.js was one of the critical decisions made, which enabled us to launch our web app in under a month (which otherwise would have taken 3 months easily). For those folks who are looking for big names, Adobe, and Alibaba and Gitlab are using Vue.

By choosing Vuetify, we saved thousands of person hours in designing the CSS files. Vuetify contains all key material components for designing a smooth User experience & it just works! It's an awesome framework. All of us at FundsCorner are now lifelong fanboys of Vue.js and Vuetify.

On the infrastructure side, all our API services and backend services are deployed as server less micro-services through Zappa. Zappa makes your life super easy by packaging everything that is required to deploy your code as AWS Lambda. We are now addicted to the single - click deploys / updates through Zappa. Try it out & you will convert!

Also, if you are using Zappa, you can greatly simplify your CI / CD pipelines. Do try it! It's just awesome! and... you will be astonished by the savings you have made on AWS bills at end of the month.

Our CI / CD pipelines are built using GitLab CI. The documentation is very good & it enables you to go from from concept to production in minimal time frame.

We use Sentry for all crash reporting and resolution. Pro tip, they do have handlers for AWS Lambda , which made our integration super easy.

All our micro-services including APIs are event-driven. Our background micro-services are message oriented & we use Amazon SQS as our message pipe. We have our own in-house workflow manager to orchestrate across micro - services.

We host our static websites on Netlify. One of the cool things about Netlify is the automated CI / CD on git push. You just do a git push to deploy! Again, it is super simple to use and it just works. We were dogmatic about going server less even on static web sites & you can go server less on Netlify in a few minutes. It's just a few clicks away.

We use Google Compute Engine, especially Google Vision for our AI experiments.

For Ops automation, we use Slack. Slack provides a super-rich API (through Slack App) through which you can weave magical automation on boring ops tasks.

9 upvotes5.5K views

Decision at Kokoen GmbH about ExpressJS, Node.js, JavaScript, MongoDB, Go, MySQL, Laravel, PHP

Avatar of ASkenny
CEO at Kokoen GmbH
ExpressJSExpressJS
Node.jsNode.js
JavaScriptJavaScript
MongoDBMongoDB
GoGo
MySQLMySQL
LaravelLaravel
PHPPHP

Back at the start of 2017, we decided to create a web-based tool for the SEO OnPage analysis of our clients' websites. We had over 2.000 websites to analyze, so we had to perform thousands of requests to get every single page from those websites, process the information and save the big amounts of data somewhere.

Very soon we realized that the initial chosen script language and database, PHP, Laravel and MySQL, was not going to be able to cope efficiently with such a task.

By that time, we were doing some experiments for other projects with a language we had recently get to know, Go , so we decided to get a try and code the crawler using it. It was fantastic, we could process much more data with way less CPU power and in less time. By using the concurrency abilites that the language has to offers, we could also do more Http requests in less time.

Unfortunately, I have no comparison numbers to show about the performance differences between Go and PHP since the difference was so clear from the beginning and that we didn't feel the need to do further comparison tests nor document it. We just switched fully to Go.

There was still a problem: despite the big amount of Data we were generating, MySQL was performing very well, but as we were adding more and more features to the software and with those features more and more different type of data to save, it was a nightmare for the database architects to structure everything correctly on the database, so it was clear what we had to do next: switch to a NoSQL database. So we switched to MongoDB, and it was also fantastic: we were expending almost zero time in thinking how to structure the Database and the performance also seemed to be better, but again, I have no comparison numbers to show due to the lack of time.

We also decided to switch the website from PHP and Laravel to JavaScript and Node.js and ExpressJS since working with the JSON Data that we were saving now in the Database would be easier.

As of now, we don't only use the tool intern but we also opened it for everyone to use for free: https://tool-seo.com

8 upvotes12.3K views

Decision at ReadMe.io about Compose, MongoLab, MongoDB Atlas, PostgreSQL, MySQL, MongoDB

Avatar of gkoberger
ComposeCompose
MongoLabMongoLab
MongoDB AtlasMongoDB Atlas
PostgreSQLPostgreSQL
MySQLMySQL
MongoDBMongoDB

We went with MongoDB , almost by mistake. I had never used it before, but I knew I wanted the *EAN part of the MEAN stack, so why not go all in. I come from a background of SQL (first MySQL , then PostgreSQL ), so I definitely abused Mongo at first... by trying to turn it into something more relational than it should be. But hey, data is supposed to be relational, so there wasn't really any way to get around that.

There's a lot I love about MongoDB, and a lot I hate. I still don't know if we made the right decision. We've been able to build much quicker, but we also have had some growing pains. We host our databases on MongoDB Atlas , and I can't say enough good things about it. We had tried MongoLab and Compose before it, and with MongoDB Atlas I finally feel like things are in a good place. I don't know if I'd use it for a one-off small project, but for a large product Atlas has given us a ton more control, stability and trust.

8 upvotes10.8K views

Decision at SalesAutopilot Kft. about AWS CodePipeline, Jenkins, Docker, vuex, Vuetify, Vue.js, jQuery UI, Redis, MongoDB, MySQL, Amazon Route 53, Amazon CloudFront, Amazon SNS, Amazon CloudWatch, GitHub

Avatar of gykhauth
CTO at SalesAutopilot Kft.
AWS CodePipelineAWS CodePipeline
JenkinsJenkins
DockerDocker
vuexvuex
VuetifyVuetify
Vue.jsVue.js
jQuery UIjQuery UI
RedisRedis
MongoDBMongoDB
MySQLMySQL
Amazon Route 53Amazon Route 53
Amazon CloudFrontAmazon CloudFront
Amazon SNSAmazon SNS
Amazon CloudWatchAmazon CloudWatch
GitHubGitHub

I'm the CTO of a marketing automation SaaS. Because of the continuously increasing load we moved to the AWSCloud. We are using more and more features of AWS: Amazon CloudWatch, Amazon SNS, Amazon CloudFront, Amazon Route 53 and so on.

Our main Database is MySQL but for the hundreds of GB document data we use MongoDB more and more. We started to use Redis for cache and other time sensitive operations.

On the front-end we use jQuery UI + Smarty but now we refactor our app to use Vue.js with Vuetify. Because our app is relatively complex we need to use vuex as well.

On the development side we use GitHub as our main repo, Docker for local and server environment and Jenkins and AWS CodePipeline for Continuous Integration.

7 upvotes4.5K views

Decision about Yarn, Redux.js, React, jQuery, vuex, Vue.js, MongoDB, Redis, PostgreSQL, Sidekiq, Rails, Font-awesome, Bulma.io

Avatar of cyrusstoller
YarnYarn
Redux.jsRedux.js
ReactReact
jQueryjQuery
vuexvuex
Vue.jsVue.js
MongoDBMongoDB
RedisRedis
PostgreSQLPostgreSQL
SidekiqSidekiq
RailsRails
#Font-awesome
#Bulma.io

I'm building a new process management tool. I decided to build with Rails as my backend, using Sidekiq for background jobs. I chose to work with these tools because I've worked with them before and know that they're able to get the job done. They may not be the sexiest tools, but they work and are reliable, which is what I was optimizing for. For data stores, I opted for PostgreSQL and Redis. Because I'm planning on offering dashboards, I wanted a SQL database instead of something like MongoDB that might work early on, but be difficult to use as soon as I want to facilitate aggregate queries.

On the front-end I'm using Vue.js and vuex in combination with #Turbolinks. In effect, I want to render most pages on the server side without key interactions being managed by Vue.js . This is the first project I'm working on where I've explicitly decided not to include jQuery . I have found React and Redux.js more confusing to setup. I appreciate the opinionated approach from the Vue.js community and that things just work together the way that I'd expect. To manage my javascript dependencies, I'm using Yarn .

For CSS frameworks, I'm using #Bulma.io. I really appreciate it's minimal nature and that there are no hard javascript dependencies. And to add a little spice, I'm using #font-awesome.

5 upvotes5.2K views

Decision at updown.io about updown.io, MongoDB, Rails

Avatar of adrienjarthon
Founder at updown.io
updown.ioupdown.io
MongoDBMongoDB
RailsRails

I chose Rails and MongoDB to build updown.io because it's very convenient and already in my skill set. This allows me to save on development and research time. The schema-less design of MongoDB allows for easier migrations and the replicaset feature for easier ops.

5 upvotes4.3K views

Decision at FundsCorner about MongoDB Atlas, MongoDB, PostgreSQL

Avatar of jeyabalajis
CTO at FundsCorner
MongoDB AtlasMongoDB Atlas
MongoDBMongoDB
PostgreSQLPostgreSQL

Database is at the heart of any technology stack. It is no wonder we spend a lot of time choosing the right database before we dive deep into product building.

When we were faced with the question of what database to choose, we set the following criteria: The database must (1) Have a very high transaction throughput. We wanted to err on the side of "reads" but not on the "writes". (2) be flexible. I.e. be adaptive enough to take - in data variations. Since we are an early-stage start-up, not everything is set in stone. (3) Fast & easy to work with (4) Cloud Native. We did not want to spend our time in "ANY" infrastructure management.

Based on the above, we picked PostgreSQL and MongoDB for evaluation. We tried a few iterations on hardening the data model with PostgreSQL, but realised that we can move much faster by loosely defining the schema (with just a few fundamental principles intact).

Thus we switched to MongoDB. Before diving in, we validated a few core principles such as: (1) Transaction guarantee. Until 3.6, MongoDB supports Transaction guarantee at Document level. From 4.0 onwards, you can achieve transaction guarantee across multiple documents.

(2) Primary Keys & Indexing: Like any RDBMS, MongoDB supports unique keys & indexes to ensure data integrity & search ability

(3) Ability to join data across data sets: MongoDB offers a super-rich aggregate framework that enables one to filter and group data

(4) Concurrency handling: MongoDB offers specific operations (such as findOneAndUpdate), which when coupled with Optimistic Locking, can be used to achieve concurrency.

Above all, MongoDB offers a complete no-frills Cloud Database as a service - MongoDB Atlas. This kind of sealed the deal for us.

Looking back, choosing MongoDB with MongoDB Atlas was one of the best decisions we took and it is serving us well. My only gripe is that there must be a way to scale-up or scale-down the Atlas configuration at different parts of the day with minimal downtime.

4 upvotes2 comments1.8K views

Decision about Koa, ES6, JavaScript, Ionic, Vue.js, MongoDB, ExpressJS, Node.js

Avatar of akobashikawa
Web developer | Blogger | Freelancer at Rulo Kobashikawa
KoaKoa
ES6ES6
JavaScriptJavaScript
IonicIonic
Vue.jsVue.js
MongoDBMongoDB
ExpressJSExpressJS
Node.jsNode.js

We are using Node.js and ExpressJS to build a REST services that is middleware of a legacy system. MongoDB as database. Vue.js helps us to make rapid UI to test use cases. Frontend is build for mobile with Ionic . We like using JavaScript and ES6 .

I think next step could be to use Koa but I am not sure.

3 upvotes4 comments7.5K views