Shaw Academy

Multiple systems means there is a requirement to cart data across them.

Started off with Talend scripts. This was great as what we initially had were PHP/Python script - allowed for a more systematic approach to ETL.

But ended up with a massive repository of scripts, complex crontab entries and regular failures due to memory issues.

Using Stitch or similar services is a better approach: - no need to worry about the infrastructure needed for the ETL processes - a more formal mapping of data from source to destination as opposed to script developer doing his/her voodoo magic - lot of common sources and destination integrations are already builtin and out of the box Stitch

Multiple systems means there is a requirement to cart data across them.

Started off with Talend scripts. This was great as what we initially had were PHP/Python script - allowed for a more systematic approach to ETL.

But ended up with a massive repository of scripts, complex crontab entries and regular failures due to memory issues.

Using Stitch or similar services is a better approach: - no need to worry about the infrastructure needed for the ETL processes - a more formal mapping of data from source to destination as opposed to script developer doing his/her voodoo magic - lot of common sources and destination integrations are already builtin and out of the box

Running a subscription service with just direct calls to Stripe or similar payment gateways is possible but also needs dedicated person(s) for decent amount of development and maintenance.

Plus features like updating card details, invoice history - all these can be built. Again, more dev work and resources.

Use of subscription platform like Chargebee or Recurly is definitely a great help here.

Chargebee offered a simple pay-as-you-go transparent pricing and almost trivial signup process.

#Paymentgatewayintegration

Multiple systems means there is a requirement to cart data across them.

Started off with Talend scripts. This was great as what we initially had were PHP/Python script - allowed for a more systematic approach to ETL.

But ended up with a massive repository of scripts, complex crontab entries and regular failures due to memory issues.

Using Stitch or similar services is a better approach: - no need to worry about the infrastructure needed for the ETL processes - a more formal mapping of data from source to destination as opposed to script developer doing his/her voodoo magic - lot of common sources and destination integrations are already builtin and out of the box

etl @{etlasaservice}|topic:1323|

Initial storage was traditional MySQL. The pace of changes during a startup mode made it very difficult to have a clean and consistent schema. Large portions ended up as unstructured data stuffed into CLOBs and BLOBs.

Moving to MongoDB definitely made this part much easier.

Accessing data for analysis is a little bit of a challenge - especially for people coming from the world of SQL Workbench. But with tools like Exploratory this is becoming less of a problem.

#NosqlDatabaseAsAService

Using a #cloud infrastructure and architecting for horizontal scaling is great way to handle changing traffic loads. Combining this with a caching layer can make things lot more efficient.

We use it as a layer for data access that does not change very often.

For example, student's profile information does not change frequently. Also, helps insulate students taking classes from any load and traffic spikes in new registrations.

Startup and not want to be bothered about physical servers and related headaches? You need to signup with #AWS or one of the other cloud providers.

Even when we had physical servers it was not an on-premise device. Leased from a provider. But not build around cloud principles so no elasticity etc.

One summer month when in-bound traffic went up 3x was sufficient to quickly realize the value of moving to Amazon EC2 instances.

Using Firebase in 2 ways: - App configuration and mobile backend aspects not handled by the main application. - Realtime database features to build a webinar+chat system for classroom sessions