Monitoring MEAN stack applications with Datadog

240
Datadog
Datadog is a monitoring service for IT, Operations and Development teams who write and run applications at scale, and want to turn the massive amounts of data produced by their apps, tools and services into actionable insight.

This post is by Mallory Mooney of Datadog

The MEAN stack has become an increasingly popular choice for developing dynamic applications. Similar to other development stacks like LAMP and Ruby on Rails, MEAN bundles together complementary technologies to provide a ready-made framework for developing and serving applications. There are variations to this stack, but it primarily consists of the following components:

  1. MongoDB: an open source database that stores data in JSON-style documents
  2. Express: a Node.js framework for building web application backends and APIs
  3. Angular: a frontend development framework for designing mobile and web applications
  4. Node.js: an open source, JavaScript runtime environment for creating server-side frameworks and applications

Monitoring the MEAN stack

As your application grows in complexity, debugging issues across its various components becomes increasingly difficult. In this guide, we’ll show you how to get comprehensive insights into your MEAN stack application and its underlying infrastructure with Datadog. Along the way, we will:

  1. Install and configure the Datadog Agent
  2. Enable Datadog’s integrations to collect data from your MEAN stack application
  3. Instrument your MEAN stack to get deeper insights into application performance

The basic structure of a MEAN application consists of Angular components and application server file(s) (e.g., server.js, index.js, or app.js). These files can include configurations for serving Angular build files, creating Express routes for an API backend, connecting to the MongoDB database, and executing database commands (i.e., CRUD operations). In this post, we will walk through configuring some of these components as well as show how to provide visibility across all the pieces of your stack with custom dashboards.

If you don’t already have a Datadog account, you can sign up for a free trial if you would like to follow along with the steps in this guide.

Installing and configuring the Datadog Agent

In order to begin collecting data from your MEAN stack application, you will need to download and install the Datadog Agent on your application server. This step requires your Datadog API key, which you can find in your account. For Linux hosts, run the following script to install the Agent:

DD_API_KEY=<YOUR_API_KEY> bash -c "$(curl -L https://raw.githubusercontent.com/DataDog/datadog-agent/master/cmd/agent/install_script.sh)"

You can run the Agent’s status command to verify that it was successfully installed. The Agent will automatically forward system-level metrics such as CPU usage, memory breakdowns, and load averages to Datadog, so you can immediately visualize and alert on host metrics. Before we enable specific integrations, we first need to configure the Agent to collect logs and traces.

Configuring the Agent to collect request traces and logs

The Agent includes parameters for log and trace collection in its datadog.yaml configuration file, located in the /etc/datadog-agent/directory on Linux hosts; to find the correct path for another platform, consult the documentation. Tracing is enabled by default, but you can add an environment tag in the Agent’s APM settings to better categorize the data coming from your application:

# Trace Agent Specific Settings
apm_config:
  enabled: true
  env: staging

Tags give you the ability to filter and aggregate metrics in Datadog. This is useful if you monitor applications across multiple environments, or need to view metrics from a specific group of applications.

To monitor application logs, you can enable log collection in the same datadog.yaml file:

# Logs agent
logs_enabled: true

Save the file and restart the Agent to apply your recent changes:

sudo service datadog-agent restart

With the Agent prepped, you can configure it to begin collecting metrics and logs from the technologies in your stack. In the next section, we will walk through enabling Datadog’s MongoDB, Express, and Node.js integrations.

Enabling Datadog integrations for the MEAN stack

To configure the Agent integrations for each component of your MEAN stack, you can create a conf.yaml file in the appropriate subdirectory of the Agent’s configuration directory (/etc/datadog-agent/conf.d/ on Linux hosts). The Agent will automatically apply any metric and log configurations you place in those files, including any custom configurations you want to apply to your stack.

Configuring Datadog’s MongoDB integration

Datadog’s MongoDB integration keeps you informed of uptime and replication state changes in addition to collecting metrics from your nodes. In order to collect data from MongoDB, you will first need to create a new datadog user (in a mongo shell) with read and clusterMonitor roles in the admin database. You’ll also need to grant the datadog user the read role on your application database:

# MongoDB 3.x
use admin
db.auth("admin", "<ADMIN_PASSWORD>")

db.createUser({
  "user":"datadog",
  "pwd": "<UNIQUE_PASSWORD>",
  "roles" : [
    {role: 'read', db: 'admin' },
    {role: 'clusterMonitor', db: 'admin'},
    {role: 'read', db: '<YOUR_DB>'}
  ]
})

This will grant the Agent read-only access to your database collections and MongoDB’s own cluster monitoring tools, such as the top command for retrieving usage statistics for your collections. You can view these commands in more detail and generate a unique password in the MongoDB integration tile of your Datadog account. Note that the commands to create a new user will differ slightly for MongoDB 2.x.

Next, edit the Agent’s MongoDB integration configuration file, located in the /etc/datadog-agent/conf.d/mongo.d/ directory on Linux hosts. Within this directory, you’ll find a conf.yaml.example file which you can copy to conf.yaml. This example file includes all available configuration options, though you do not need every option to begin forwarding metrics. To get started, the integration only requires your MongoDB URI:

init_config:

instances:
  - server: mongodb://datadog:<UNIQUE_PASSWORD>@localhost:27017/<YOUR_DB>?authSource=admin

The authSource parameter delegates authentication for your application’s database to the admin database, where the datadog user is stored.

To begin forwarding logs to Datadog, include the path to your MongoDB log file in the logs section of the same conf.yaml configuration file:

## Log section (Available for Agent >=6.0)

logs:
  - type: file
    path: /var/log/mongodb/mongodb.log
    service: <YOUR_SERVICE_NAME>
    source: mongodb

The service attribute links these logs to request traces and application performance metrics that come from the same service—we’ll show you how to instrument your application to collect this data in a later section. The source attribute instructs Datadog to process these logs with the built-in MongoDB log integration pipeline, which automatically extracts useful attributes such as timestamps and operations so you can use them as facets to easily search and analyze your database logs.

Restart the Agent to apply the new changes and run its status command:

sudo service datadog-agent restart
sudo datadog-agent status

In the output, you should see that the Agent ran a monitoring check on your MongoDB server:

mongo (1.7.0)
    -------------
      Instance ID: mongo:92ade187167a3df2 [OK]
      Total Runs: 1
      Metric Samples: Last Run: 145, Total: 145
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 1
      Average Execution Time : 32ms

You can verify which metrics the Agent collects by running its service check:

sudo -u dd-agent -- datadog-agent check mongo

This command will provide a breakdown for each MongoDB metric the Agent is collecting. The following excerpt shows information about the metric type, host, and tags associated with the mongodb.connections.available metric, which measures the number of available connections in the database:

{
      "metric": "mongodb.connections.available",
      "points": [
        [
          1548449540,
          816
        ]
      ],
      "tags": [
        "server:mongodb://datadog:*****@localhost:27017/mean-db2?authSource=admin"
      ],
      "host": "meanbox",
      "type": "gauge",
      "interval": 0,
      "source_type_name": "System"
    }

By default, Datadog collects a sample of metrics using MongoDB database commands such as dbStats and serverStatus, with minimal configuration. But you can check out our documentation if you want to collect additional types of metrics, such as the total number of objects in a specific collection.

Now that the Agent is collecting your MongoDB metrics and logs, you can view them in detail in Datadog. The out-of-the-box MongoDB dashboard provides a high-level overview of database activity and resource utilization, including memory usage, connection counts, and host availability.

You can clone this dashboard and then customize it by adding graphs from other components of your stack such as Node.js or Express. You can also click on a graph and then select “View related logs” if you want to view logs collected from the MongoDB server(s) shown.

The screenshot below shows a sample MongoDB log, with key attributes that have been automatically parsed by the integration pipeline.

Collecting Express metrics from your stack

Datadog’s Express integration uses the connect-datadog middleware to forward server response metrics to Datadog. In order for the Agent to begin collecting Express metrics, you will first need to install the npm module:

npm install connect-datadog

Next, you’ll need to modify your application server file to include the Datadog middleware (connect-datadog):

// server.js
var dd_options = {
  'response_code':true,
  'tags': ['env:staging']
}

var connect_datadog = require('connect-datadog')(dd_options);

// Add the datadog-middleware before your router
app.use(connect_datadog);
app.use(router);

The middleware package includes multiple options for fine-tuning metric collection for Express. In the dd_options section of the example configuration above, we’ve enabled collecting HTTP response code metrics with response_code:true and organized the metrics using the tags option. When you start your application, the Agent will immediately begin forwarding Express metrics to Datadog so you can gain deeper insights in the health of your server. For example, you can create a custom graph to monitor Express server response counts by route, as seen in the image below.

Creating Node.js custom metrics

Datadog’s Node.js integration uses the hot-shots DogStatsD client to collect custom metrics. Once you install the library (npm install hot-shots), you can track metrics such as the number of page views for a specific route by adding the following to your application server file:

// server.js
var StatsD = require('hot-shots');
var dogstatsd = new StatsD();

app.get("/contacts", function(req, res) {
  // Increment a counter.
  dogstatsd.increment('node.page.views', ['method:GET', 'route:contacts']);
  [...]
});

In the example above, we are using node to categorize our custom page views metric; the full metric name will appear in Datadog as node.page.views. This allows us to group custom metrics and link related logs. After you restart your application to apply the new changes, you can use the Metrics Explorer in your Datadog account to create graphs of your custom metrics, which you can then export to your dashboard.

Log collection for Node.js and Express

Datadog recommends using the Winston library to collect logs from Node.js applications. Winston is a comprehensive logging utility that allows you to implement a customized logger for your application, whether it runs on pure Node.js or uses a framework like Express.

You can install the library with npm:

npm install --save winston

Next, configure Winston to log data by creating a custom logger:

// config/logger.js
const { createLogger, format, transports } = require('winston');

const logger = createLogger({
  level: 'info',
  format: format.combine(
    format.timestamp({
      format: 'YYYY-MM-DD HH:mm:ss'
    }),
    format.json()
  ),
  transports: [
    new transports.File({ filename: 'combined.log' })
  ]
});

The code snippet above configures Winston to write all log messages that have the info log level priority (includes warn and error logs) to the same combined.log file in JSON format. You can then implement your logger within your Express routes and database functions:

// server.js
// import logger
var logger = require('./config/logger');

[...]

app.post("/api/contacts", function(req, res) {
  logger.log('info', 'create new contact');


  db.collection("contacts").insertOne(newContact, function(err, doc) {
      if (err) {
        logger.log('error', 'Failed to create a new contact');
      } 
      [...]
  });
});

Each log will include the log level, message, and timestamp in the combined.log file:

{"level":"info","message":"GET /api/contacts","timestamp":"2019-01-09 13:55:55"}
{"level":"error","message":"Failed to create a new contact","timestamp":"2019-01-09 15:23:25"}

To forward these logs to Datadog, you can create a /nodejs.d/conf.yaml configuration file in your host’s /etc/datadog-agent/conf.d/ directory and include the location of your application’s log file(s):

init_config:

instances:


##Log section
logs:

    ## - type : file (mandatory) type of log input source (tcp / udp / file)
    ##   port / path : (mandatory) Set port if type is tcp or udp. Set path if type is file
    ##   service : (mandatory) name of the service owning the log
    ##   source : (mandatory) attribute that defines which integration is sending the logs
    ##   sourcecategory : (optional) Multiple value attribute. Can be used to refine the source attribute
    ##   tags: (optional) add tags to each logs collected

  - type: file
    path: /path/to/your/app/combined.log
    service: <YOUR_SERVICE_NAME>
    source: node

Since Datadog parses your application’s JSON logs automatically, you can use a custom name for the source attribute of your Node.js logs. In the example above, we use node for the source to associate these logs with the node.page.views custom metric. The service attribute makes it easier to associate logs with related request traces and application performance metrics in Datadog APM, which we will cover in detail in the next section.

Instrumenting your MEAN stack application with Datadog APM

You can get even greater context around your metrics and logs by instrumenting your application with Datadog APM. Datadog APM provides detailed overviews of throughput, latency, and error counts by tracing requests across every service in your application. Datadog’s open source Node.js tracing library is OpenTracing-compatible and includes plugins for Express and MongoDB, so you can easily visualize distributed request traces and performance metrics from the services in your stack.

To instrument your Node.js application, first install the npm module:

npm install --save dd-trace

Datadog’s JavaScript tracer is enabled to auto-instrument your Express and MongoDB integrations by default, which means the tracer can automatically assign service names and capture traces from your server and database. To use the tracer’s out-of-the-box instrumentation, include the following code snippet at the top of your application server file:

// server.js
const tracer = require('dd-trace').init();

You can customize your tracer’s options to suit your use case. For example, you can configure the Express and MongoDB integrations to assign a specific service name and link these services to their corresponding logs:

// server.js
const tracer = require('dd-trace').init()

// configure express integration
tracer.use('express', {
  service: '<YOUR_SERVICE_NAME>'
})

Once you restart your application, the Datadog Agent will forward traces from your stack to your Datadog account. On the APM services list page, you can view information about each service from your instrumented application.

You can also filter by the type of service (web, database, cache, or custom) and select a specific service to view more detailed information about its performance, including a list of individual request traces. Once you select an individual trace, you can inspect its flame graph to see a breakdown of spans. Spans are units of work within a trace and represent activity such as HTTP requests or database queries. When you click on a span, you can see its metadata such as the exact database call or the stack trace of a MongoDB-related error.

As seen in the example above, Datadog will capture application errors, along with useful metadata about the error, so that you can quickly diagnose issues. You can navigate to the “Host Info” tab to correlate system-level metrics with the time of the trace. This gives you a better view into the health of your host and can serve as a starting point for debugging your application. Since your application consists of many different components, being able to correlate data across your stack is integral to performance monitoring.

Exploring your application data and logs

Now that your MEAN stack application is sending metrics, request traces, and logs to Datadog, you can seamlessly navigate across all of these sources of data. For example, when viewing an individual request trace, you can click on the “Logs” tab to quickly jump to related logs generated by your application. Earlier, we configured our Node.js application to log messages in JSON format, so Datadog will automatically parse them and create structured attributes.

You can use these attributes to create facets and measures that are useful for searching and filtering your logs. You can also use measures to create custom graphs that you can then export to any dashboard to view your log analytics data in the same place as metrics and APM data.

In the example dashboard above, we can see that there is an issue with our database based on MongoDB’s “CRIT” status and the noticeable drop in database activity (in the “MongoDB executed commands” graph). For more details about this problem, we can look in the log stream, which captured an error log from the database:

`E STORAGE [initandlisten] Failed to set up listener: SocketException: Address already in use`

Dashboards give you the ability to correlate key metrics with logs from your stack in one place, so you can be aware of issues as they happen.

Get started with MEAN stack monitoring

If you’ve followed along with this tutorial, you should now have deeper visibility into the key components of your MEAN stack application. And with Datadog’s 250 + integrations, you can monitor your stack alongside all of the other services running in your infrastructure. If you don’t have a Datadog account, you can sign up for a free trial to start monitoring your applications today.

Datadog
Datadog is a monitoring service for IT, Operations and Development teams who write and run applications at scale, and want to turn the massive amounts of data produced by their apps, tools and services into actionable insight.
Tools mentioned in article
Open jobs at Datadog
Software Engineer - Data Science
Paris

About Datadog:

We're on a mission to build the best platform in the world for engineers to understand and scale their systems, applications, and teams.  We operate at high scale—trillions of data points per day—providing always-on alerting, metrics visualization, logs, and application tracing for tens of thousands of companies. Our engineering culture values pragmatism, honesty, and simplicity to solve hard problems the right way.

 

The team:

The data science team focuses on improving the observability of dynamic distributed systems by creating practical user-facing features in the Datadog app. We research, develop, and maintain features such as automated alerting, forecasting, anomaly detection, directed root cause analysis, and much more.

 

The opportunity:

This role sits at the intersection of software engineering, data engineering, and data science.  You will be working on large-scale distributed systems, horizontally-scalable datastores and a variety of data-processing frameworks to build our next-generation platform and capabilities for data science at Datadog. You will be contributing to our core products in a way that significantly and directly impacts our customers’ experiences. 

 

You will:

  • Build common services and components encapsulating data science capabilities for other products and teams to leverage
  • Enable new capabilities for our core products by integrating and preprocessing data from tens of sources and preparing it for incorporation into data science models
  • Ensure that our data science-driven features continue to scale well as Datadog adds more customers and ingests more data per customer
  • Leverage modern tools and frameworks to improve the performance and cost-efficiency of data science services
  • Join an amazing team of data scientists and software engineers building a product that our customers love to use

 

Requirements:

  • You can write performant Python code
  • You have experience building data processing systems
  • You can design architecture to solve problems at high scale
  • You want to work on a data science team that ships products
  • You value simplicity and getting the right things done
  • You want to work in a fast-moving, high-growth environment that respects engineers and customers

 

Bonus points:

  • You are an expert in Go or Scala
  • You have built systems that process terabytes of data every day
  • You have more than a passing interest in statistics or data science
Software Engineer - Mobile App
Paris, Remote - EU

About Datadog:

At Datadog, we're on a mission to build the best platform in the world for engineers to understand and scale their systems, applications, and teams. We operate at high scale—trillions of data points per day—providing always-on alerting, metrics visualization, logs, and application tracing for tens of thousands of companies. Our engineering culture values pragmatism, honesty, and simplicity to solve hard problems the right way.

 

The team:

Our customers are engineers; they want a seamless mobile experience to understand their applications and systems status, wherever they are in their day, wherever they are in the world. Our Mobile App team will build the Android and iOS apps that help on-call engineers at 10s of thousands of companies rely on to monitor the health of their applications and systems, and respond quickly to alerts.

 

The opportunity:

As a Software Engineer in the Mobile App team, you will join a fast-growing team of engineers to launch Datadog’s first native Android & iOS applications. 

 

You will:

  • Have a significant impact on a new product and the user experience of fellow engineers who use the product
  • Develop systems that will be used by tens of thousands of engineers around the world
  • Work closely with the product teams to deliver the best-in-class mobile application

 

Requirements:

  • You have significant experience in Java/Kotlin or Objective-C/Swift
  • You value code simplicity and performance
  • You have a BS/MS/PhD in a scientific field or equivalent experience
  • You want to work in a fast, high-growth startup environment that respects its engineers and customers
Engineering Manager - Logs Platform
Paris

About Datadog:

We're on a mission to build the best platform in the world for engineers to understand and scale their systems, applications, and teams. We operate at high scale—trillions of data points per day—providing always-on alerting, metrics visualization, logs, and application tracing for tens of thousands of companies. Our engineering culture values pragmatism, honesty, and simplicity to solve hard problems the right way.

 

The opportunity:

We’re looking for an Engineering Manager to lead and grow the teams building our log management platform. Working very closely with product management, they build the log application but also the log collection agent and integration endpoints. They own all the services from ingestion to processing as well as near-real time systems like live-tail or alerting. 

The platform runs in the cloud, it's always on, low latency and high throughput. It's built on top of high-scale open source frameworks and languages: Kafka, Go, Java, MongoDB, Akka, Kubernetes.

You’ll join at an ideal time to make a big impact, the product is seeing very high growth, with many new features to build as well as a need for scaling up dramatically. You’ll be involved in every step of the process—defining product, designing architecture, managing the team, recruiting excellent engineers.

 

You will:

  • Manage and grow teams of talented engineers, ensuring they deliver high quality, timely work and that they’re happy, motivated, and growing
  • Work with the product team to flesh out the future of the product
  • Work with other managers in Europe and in the US to deliver cross-cutting projects (e.g. running in multiple regions and clouds)
  • Scale the systems to accommodate hyper growth
  • Contribute to shaping the future of the engineering organisation

 

Requirements:

  • You have been building applications for 6+ years and know the systems you’ve worked on from top to bottom
  • You have significant backend programming experience
  • You have managed a team of software engineers
  • You have architected, built, and operated distributed systems to solve problems at high scale
  • You have a BS/MS/PhD in a scientific field or equivalent experience
  • You want to work in a fast-paced, high-growth startup environment to build a brand new product

 

Bonus points:

  • You’ve built a non-trivial application from scratch that sees significant user traffic
  • You’ve recruited a team of engineers before
  • You have deep experience with observability products
  • You have frontend experience
Solutions Engineer - Tokyo (Japanese ...
Tokyo

The team:

We are Datadog's in-house product experts. The Technical Solutions team enables Datadog's worldwide growth by educating potential clients and ensuring that existing customers are happy and successful. We share our technical and product expertise with customers through demos, presentations, technical evaluations, and ongoing support. Technical Solutions is a growing global team that collaborates constantly to share knowledge and continuously advance our technical skill set.

 

The opportunity:

Join a diverse team of traditional and non-traditional backgrounds, working together to solve hard problems the right way. You’ll be joining a startup where you will be challenged and get to immediately witness your contributions to Datadog.

 

You will:

  • Respond to client requests on our fast paced team while continuing to educate our clients on the use of the platform
  • Reproduce issues and dive into the 250+ integrations that Datadog works with
  • Build out documentation and knowledge based articles for a variety of technology
  • Drive product conversations based on needs and problems learned during client interactions

 

Requirements:

  • You’re self-motivated, detail-attentive, and wish to keep learning more
  • You’re a master at one thing: tell us all about it
  • You’re a critical thinker, who defaults to a client-gratification manner
  • You’re a tinkerer with some programming experience and a basic knowledge of Linux
  • Speaks Japanese

 

Bonus points:

  • CS or Engineering majors
  • Active contribution to open-source projects (code, bug reports, etc.) and the Engineering Community (Meetups, etc.)
  • You want to start your own company someday

 


Datadog is a monitoring service for hybrid cloud applications, assisting organizations in improving agility, increasing efficiency, and providing end-to-end visibility across the application and organization. These capabilities are provided on a SaaS-based data analytics platform that enables DevOps and other teams to accelerate go-to-market efforts, ensure application uptime, and successfully complete digital transformation initiatives.

Verified by
Evangelist
Partner Marketing Manager
Marketing Manager | Datadog, inc.
You may also like