Alternatives to JSON Server

Recently we were looking at a few robust and cost-effective ways of replicating the data that resides in our production MongoDB to a PostgreSQL database for data warehousing and business intelligence.

We set ourselves the following criteria for the optimal tool that would do this job: - The data replication must be near real-time, yet it should NOT impact the production database - The data replication must be horizontally scalable (based on the load), asynchronous & crash-resilient

Based on the above criteria, we selected the following tools to perform the end to end data replication:

We chose MongoDB Stitch for picking up the changes in the source database. It is the serverless platform from MongoDB. One of the services offered by MongoDB Stitch is Stitch Triggers. Using stitch triggers, you can execute a serverless function (in Node.js) in real time in response to changes in the database. When there are a lot of database changes, Stitch automatically "feeds forward" these changes through an asynchronous queue.

We chose Amazon SQS as the pipe / message backbone for communicating the changes from MongoDB to our own replication service. Interestingly enough, MongoDB stitch offers integration with AWS services.

In the Node.js function, we wrote minimal functionality to communicate the database changes (insert / update / delete / replace) to Amazon SQS.

Next we wrote a minimal micro-service in Python to listen to the message events on SQS, pickup the data payload & mirror the DB changes on to the target Data warehouse. We implemented source data to target data translation by modelling target table structures through SQLAlchemy . We deployed this micro-service as AWS Lambda with Zappa. With Zappa, deploying your services as event-driven & horizontally scalable Lambda service is dumb-easy.

In the end, we got to implement a highly scalable near realtime Change Data Replication service that "works" and deployed to production in a matter of few days!

We use MongoDB as our primary #datastore. Mongo's approach to replica sets enables some fantastic patterns for operations like maintenance, backups, and #ETL.

As we pull #microservices from our #monolith, we are taking the opportunity to build them with their own datastores using PostgreSQL. We also use Redis to cache data we’d never store permanently, and to rate-limit our requests to partners’ APIs (like GitHub).

When we’re dealing with large blobs of immutable data (logs, artifacts, and test results), we store them in Amazon S3. We handle any side-effects of S3’s eventual consistency model within our own code. This ensures that we deal with user requests correctly while writes are in process.

Application and Data: Since my personal website ( https://alisoueidan.com ) is a SPA I've chosen to use Vue.js, as a framework to create it. After a short skeptical phase I immediately felt in love with the single file component concept! I also used vuex for state management, which makes working with several components, which are communicating with each other even more fun and convenient to use. Of course, using Vue requires using JavaScript as well, since it is the basis of it.

For markup and style, I used Pug and Sass, since they’re the perfect match to me. I love the clean and strict syntax of both of them and even more that their structure is almost similar. Also, both of them come with an expanded functionality such as mixins, loops and so on related to their “siblings” (HTML and CSS). Both of them require nesting and prevent untidy code, which can be a huge advantage when working in teams. I used JSON to store data (since the data quantity on my website is moderate) – JSON works also good in combo with Pug, using for loops, based on the JSON Objects for example.

To send my contact form I used PHP, since sending emails using PHP is still relatively convenient, simple and easy done.

DevOps: Of course, I used Git to do my version management (which I even do in smaller projects like my website just have an additional backup of my code). On top of that I used GitHub since it now supports private repository for free accounts (which I am using for my own). I use Babel to use ES6 functionality such as arrow functions and so on, and still don’t losing cross browser compatibility.

Side note: I used npm for package management. 🎉

*Business Tools: * I use Asana to organize my project. This is a big advantage to me, even if I work alone, since “private” projects can get interrupted for some time. By using Asana I still know (even after month of not touching a project) what I’ve done, on which task I was at last working on and what still is to do. Working in Teams (for enterprise I’d take on Jira instead) of course Asana is a Tool which I really love to use as well. All the graphics on my website are SVG which I have created with Adobe Illustrator and adjusted within the SVG code or by using JavaScript or CSS (SASS).

I use Visual Studio Code because at this time is a mature software and I can do practically everything using it.

• It's free and open source: The project is hosted on GitHub and it’s free to download, fork, modify and contribute to the project.

• Multi-platform: You can download binaries for different platforms, included Windows (x64), MacOS and Linux (.rpm and .deb packages)

• LightWeight: It runs smoothly in different devices. It has an average memory and CPU usage. Starts almost immediately and it’s very stable.

• Extended language support: Supports by default the majority of the most used languages and syntax like JavaScript, HTML, C#, Swift, Java, PHP, Python and others. Also, VS Code supports different file types associated to projects like .ini, .properties, XML and JSON files.

• Integrated tools: Includes an integrated terminal, debugger, problem list and console output inspector. The project navigator sidebar is simple and powerful: you can manage your files and folders with ease. The command palette helps you find commands by text. The search widget has a powerful auto-complete feature to search and find your files.

• Extensible and configurable: There are many extensions available for every language supported, including syntax highlighters, IntelliSense and code completion, and debuggers. There are also extension to manage application configuration and architecture like Docker and Jenkins.

• Integrated with Git: You can visually manage your project repositories, pull, commit and push your changes, and easy conflict resolution.( there is support for SVN (Subversion) users by plugin)

Fontumi focuses on the development of telecommunications solutions. We have opted for technologies that allow agile development and great scalability.

Firebase and Node.js + FeathersJS are technologies that we have used on the server side. Vue.js is our main framework for clients.

Our latest products launched have been focused on the integration of AI systems for enriched conversations. Google Compute Engine , along with Dialogflow and Cloud Firestore have been important tools for this work.

Git + GitHub + Visual Studio Code is a killer stack.

In #Aliadoc, we're exploring the crowdfunding option to get traction before launch. We are building a SaaS platform for website design customization.

For the Admin UI and website editor we use React and we're currently transitioning from a Create React App setup to a custom one because our needs have become more specific. We use CloudFlare as much as possible, it's a great service.

For routing dynamic resources and proxy tasks to feed websites to the editor we leverage CloudFlare Workers for improved responsiveness. We use Firebase for our hosting needs and user authentication while also using several Cloud Functions for Firebase to interact with other services along with Google App Engine and Google Cloud Storage, but also the Real Time Database is on the radar for collaborative website editing.

We generally hate configuration but honestly because of the stage of our project we lack resources for doing heavy sysops work. So we are basically just relying on Serverless technologies as much as we can to do all server side processing.

Visual Studio Code definitively makes programming a much easier and enjoyable task, we just love it. We combine it with Bitbucket for our source code control needs.

We just launched the Segment Config API (try it out for yourself here) — a set of public REST APIs that enable you to manage your Segment configuration. A public API is only as good as its #documentation. For the API reference doc we are using Postman.

Postman is an “API development environment”. You download the desktop app, and build API requests by URL and payload. Over time you can build up a set of requests and organize them into a “Postman Collection”. You can generalize a collection with “collection variables”. This allows you to parameterize things like username, password and workspace_name so a user can fill their own values in before making an API call. This makes it possible to use Postman for one-off API tasks instead of writing code.

Then you can add Markdown content to the entire collection, a folder of related methods, and/or every API method to explain how the APIs work. You can publish a collection and easily share it with a URL.

This turns Postman from a personal #API utility to full-blown public interactive API documentation. The result is a great looking web page with all the API calls, docs and sample requests and responses in one place. Check out the results here.

Postman’s powers don’t end here. You can automate Postman with “test scripts” and have it periodically run a collection scripts as “monitors”. We now have #QA around all the APIs in public docs to make sure they are always correct

Along the way we tried other techniques for documenting APIs like ReadMe.io or Swagger UI. These required a lot of effort to customize.

Writing and maintaining a Postman collection takes some work, but the resulting documentation site, interactivity and API testing tools are well worth it.

Secure Membership Web API backed by SQL Server. This is the backing API to store additional profile and complex membership metadata outside of an Azure AD B2C provider. The front-end using the Azure AD B2C to allow 3rd party trusted identity providers to authenticate. This API provides a way to add and manage more complex permission structures than can easily be maintained in Azure AD.

We have .Net developers and an Azure infrastructure environment using server-less functions, logic apps and SaaS where ever possible. For this service I opted to keep it as a classic WebAPI project and deployed to AppService.

• Trusted Authentication Provider: @AzureActiveDirectoryB2C
• Frameworks: .NET Core
• Language: C# , Microsoft SQL Server , JavaScript
• IDEs: Visual Studio Code , Visual Studio
• Libraries: jQuery @EntityFramework, @AutoMapper, @FeatureToggle , @Swashbuckle
• Database: @SqlAzure
• Source Control: Git
• Build and Release Pipelines: Azure DevOps
• Test tools: Postman , Newman
• Test framework: @nUnit, @moq
• Infrastructure: @AzureAppService, @AzureAPIManagement

We've tried a couple REST clients over the years, and Insomnia REST Client has won us over the most. Here's what we like about it compared to other contenders in this category:

• Uncluttered UI. Things are only in your face when you need them, and the app is visually organized in an intuitive manner.
• Native Mac app. We wanted the look and feel to be on par with other apps in our OS rather than a web app / Electron app (cough Postman).
• Easy team sync. Other apps have this too, but Insomnia's model best sets the "set and forget" mentality. Syncs are near instant and I'm always assured that I'm working on the latest version of API endpoints. Apps like Paw use a git-based approach to revision history, but I think this actually over-complicates the sync feature. For ensuring I'm always working on the latest version of something, I'd rather have the sync model be closer to Dropbox's than git's, and Insomnia is closer to Dropbox in that regard.

Some features like automatic public-facing documentation aren't supported, but we currently don't have any public APIs, so this didn't matter to us.

We've tried a couple REST clients over the years, and Insomnia REST Client has won us over the most. Here's what we like about it compared to other contenders in this category:

• Uncluttered UI. Things are only in your face when you need them, and the app is visually organized in an intuitive manner.
• Native Mac app. We wanted the look and feel to be on par with other apps in our OS rather than a web app / Electron app (cough Postman).
• Easy team sync. Other apps have this too, but Insomnia's model best sets the "set and forget" mentality. Syncs are near instant and I'm always assured that I'm working on the latest version of API endpoints. Apps like Paw use a git-based approach to revision history, but I think this actually over-complicates the sync feature. For ensuring I'm always working on the latest version of something, I'd rather have the sync model be closer to Dropbox's than git's, and Insomnia is closer to Dropbox in that regard.

Some features like automatic public-facing documentation aren't supported, but we currently don't have any public APIs, so this didn't matter to us.