How to debug memory leaks in a Node.js application on Heroku

146
Sqreen
Sqreen is a developer-friendly security platform for web applications, API and micro-services

This is post by Vladimir who is a Node.js Software Engineer at Sqreen.

Debugging memory leaks is rarely a piece of cake, especially when they only happen in production. The best way I’ve found to debug memory leaks in a Node.js application on Heroku is to analyze heap dumps.

Obtaining such heap dumps in production can be challenging, as it might be hard to connect remotely to a production instance with the debugger.

In this article, we will go through the steps needed to obtain and analyze heap dumps from a running Heroku dyno. This method will also work on other platforms as long as it is possible to perform similar operations.

To obtain the heap dump we need to:

  • Ensure the Node.js process has a debugger listening
  • Connect Chrome dev tools to the Node.js process
  • Collect the heap dump and download it locally

Enabling the Node.js inspector

Before we can analyze anything, we need to ensure that we have a debugger listening. There are two ways to enable the inspector on a Node.js process:

Solution 1: Changing the startup command

By default, Heroku starts a Node.js application by running npm start. Usually, this calls a script defined in the package.json of the application:

{
  "scripts": {
    "start": "node ./bin/www"
  },
}

Changing this script to add the --inspect (as documented here) flag will start the instances of the application with a debugger listening on a port that will be specified in the logs:

{
  "scripts": {
    "start": "node --inspect ./bin/www"
  },
}

In total, this is what it will look like when you implement this solution.

What changing the startup command looks like Heroku logs

Solution 2: Changing the process state through SSH

Solution 1 is the easiest way to enable an inspector in Node.js, but there are situations in which you can’t or won’t want to enable it. For example, you might not have access to the source code of the application and therefore can’t change the startup script. Or maybe you don’t want to change the state of all your production dynos and deploy your application only for debugging.

Fortunately, there is a way to send a signal to the process to enable a debugger session.

In order to do so, you will need the Heroku CLI to connect to the dyno through an SSH connection.

For all following Heroku commands, you might need to add the --app <app_name> flag to tell the CLI which application to connect to. Also, by default, the CLI will connect to the dyno named web.1 and you might want to change that through the command line (see documentation).

First, let’s connect to the dyno (Heroku might need to restart the dyno at this point):

Then, we need to identify the PID of the Node.js process:

In our case, the process started with node bin/www has the PID 69, we will now send a signal to the process to let it know we need it to enable its debugger:

$ kill -usr1 69

As you can see, we have sent the USR1 signal to the process to change its state (as documented on this page).

This is confirmed through the application’s logs on Heroku:

Changing the process state through signals

Attaching debugging tools to a Node.js process

In order to attach the debugging tools to our Node.js process, we need to make the websocket used by the debugger accessible on our local machine.

To do that, we first need to identify the port we need to forward. This can be found in the logs of the application:

In our case, this is the port 9229.

To forward the port locally, let’s use the Heroku CLI:

When port forwarding is established, we just need to open Chrome DevTools (going to chrome://inspect on Chrome) and after a few seconds, a target should be displayed under “Remote targets.”

If the target does not appear, make sure the port used is listed when clicking on “Configure.”

Collecting the heap dump and reading it

Now it’s time to collect and read the heap dump. First, click on the “inspect” link. This will open a new window with different tabs.

Find the “Memory” one — you should be prompted with the following window:

Click on “Take snapshot.” A new file will appear in the left hand side panel. Clicking on it will display the content of the heap:

In this view, objects are sorted by constructor. For the purpose of this walkthrough, I have introduced a memory leak in this application by creating an instance of the Access class for each request. This instance keeps a reference to the current HTTP requests and is never cleaned:

const Access = class {
  constructor(req) {
    this.req = req;
  }
};

const allAccesses = new Set();
app.use((req, res, next) => {
  allAccesses.add(new Access(req));
  next();
});

You can see for yourself that this indeed leaks in the application.

To detect constructors that have the biggest memory impact, let’s sort the items of this view by “Retained size” (You can learn more about these terms on Chrome’s website).

You can see that 24% of the process memory is held by these objects.

Now let’s look at how to identify where the leak is happening.

When expanding the list of the constructor, we can see all instances of this class. By selecting one of these instances, the list of retainers of this object is displayed:

In our case, the allAccesses set is clearly identified as the bad actor! With the location of the memory leak identified, we have everything we need to go off and fix it.

A few tips for debugging memory leaks in Node.js

Use the compare view

When suspecting a memory leak, you might want to take two separate heap dumps with a few minutes between them. Then, using the “comparison view”, you can identify which elements have been created between the snapshots.

Use constructors and classes in the code

As shown in the article, when reading the heap dump, elements are grouped by their constructor.

Using more than just classes in your code will make it more readable (and arguably more performant, but that’s probably a topic for another article). It will save you so much time when hunting for a memory leak. Do it — future you will be grateful.

Trigger a garbage collection before collecting the snapshot

At the top left hand side of this screen, there’s a little bin picture. Clicking on it will trigger a garbage collection in the application. Doing this before collecting a memory snapshot will actually remove elements that are not leaking and therefore could help save you time when browsing the heap content.

Conclusion

In this article, we’ve taken a look at how to debug memory leaks in a Node.js process running on Heroku by connecting and using a debugger. Feel free to contact me on Twitter if you have any questions or if you want to share your own tips with me!

If you’re looking for next steps or a more advanced way to debug memory leaks in Node.js in Heroku, try this: Since the Heroku CLI is written with Node.js, you could write an automated tool to perform the collection and start analyzing heap dumps.

unsplash-logoPhoto by Daan Mooij

Sqreen
Sqreen is a developer-friendly security platform for web applications, API and micro-services
Tools mentioned in article
Open jobs at Sqreen
Vice President of Engineering

Description

About

Sqreen is a developer platform which detects security anomalies in web applications and provides automated responses in real-time. We improve the security standards of the world's most famous applications and help developers around the world doing a better job, every day. Founded by former security experts at Apple, Sqreen protects hundreds of apps and companies like Algolia, Front, Toptal, and Helpling and has offices in San Francisco and Paris.

Being initially featured in TechCrunch’s Disrupt SF Battlefield in 2016, the company recently graduated YCombinator and secured its soon-to-be-announced funding round! We’re proud to be backed by Silicon Valley’s most well-respected Venture Capital funds which allow us to continue building a solid platform, increase the momentum and scale our product and organisation.

For more information, visit https://www.sqreen.io.

The role

You will be responsible for leading an extremely talented multi-cultural engineering team, developing the engineering culture, coordinating strategic technology choices and keeping the business on schedule. You will interact closely with the CTO and product team in tool selection, delivery roadmaps and other areas.

As the business grows and new customers are onboarded the product will need building in additional development languages, in tandem with evolving in complexity and features for existing languages.

Recruitment of talented engineers will be a priority for the successful candidate, both in Paris and San Francisco. You will need to closely follow the evolution of new web technologies, competitor products, and be passionate about the developer ecosystem.

The founding team split their time between Paris and San-Francisco and the VP Engineering will be the most senior leader based in Paris full-time. With an already international team the culture of the business is diverse and inclusive. As a group of approximately 15 people, the technical ability within the team is high and as a product targeted at the developers and engineers, a personal as well as professional interest in the wider software community is strongly desired.


Responsibilities

  • Build a strong engineering culture, facilitating work and communication across your organisation.
  • Scale the engineering team, recruiting talented engineers who fit well with the Sqreen culture and DNA, capable of carrying out necessary engineering initiatives.
  • Actively inspire, encourage talent development and career progression within the engineering team.
  • Coordinate features design with Product team, define and prioritize the engineering roadmap and ensure delivery is meeting expectations.
  • Drive the right technology choices and internal adoption, along with the CTO. Take appropriate risks, and proactively evaluate and manage incoming technical debt.
  • Develop best practices for testing, deploying, and developing features and releases.

Requirements

  • Technical ability
    • Bachelor's or Master's degree in Computer Science, Computer Engineering, Statistics, Mathematics or a similar field
    • Strong technical background, with a minimum of five years’ experience with web development and the ability to contribute to technical planning and design discussions
    • Excellent communication skills with a minimum of four years’ hands-on technical management experience
    • Experience of having managed a team of at least ten engineers
    • Experience having implemented organizational processes to make sure that team delivers on time
    • Experience of designing/developing cloud-based, scalable applications
    • Experience of designing microservices architectures on hosted networks and cloud infrastructures, along with expertise in the architecture and integration of applications into existing web stacks without performance issues
    • Expertise in developing with a keen appreciation of security requirements
  • Industry
    • Technical leadership in high-growth VC / PE or Privately held Software, SaaS or Data businesses
    • Appreciation of highly performant code and the quality challenges inherent in minimising the hit on performance of the underlying application Sqreen is protecting
    • Understanding of modern architecture and design principles and development methodologies
  • Leadership
    • Must be able to provide technical leadership to a highly technical team, earning their respect, whilst ensuring delivery progress against the defined technology roadmap
    • Experience of building, leading, and scaling engineering and development teams. Ideally, with exposure to various structures including in-house, near-shore, and outsourcing
    • Comfortable peering with the CTO to set a technical product vision and make suggestions for technology choices
    • Can articulate technology vision and delivery strategy in a language that enables discussion with a technical and non-technical audiences. Excellent communicator and issue resolver.

Benefits

  • Joining a fast growing tech company, scaling a team of amazing engineers to build one of the most disruptive security technology
  • Become part of a super passionate team that loves their jobs and company :-)
  • Beautiful office in central Paris (Sentier), in the heart of #FrenchTech
  • Premium health insurance and coached SqreenFit sessions with the team (crossfit)
  • Organic snacks & beverages and much more

Interested? Check us out at our twitter account. We’d love to hear from you!

Software engineer - Core Python engineer

Description

About

Sqreen is a developer platform which detects security anomalies in web applications and provides automated responses in real-time. We improve the security standards of the world's most famous applications and help developers around the world doing a better job, every day. Founded by former security experts at Apple, Sqreen protects hundreds of apps and companies like Algolia, Front, Toptal, and Helpling and has offices in San Francisco and Paris.

Being initially featured in TechCrunch’s Disrupt SF Battlefield in 2016, the company recently graduated YCombinator and secured its soon-to-be-announced funding round! We’re proud to be backed by Silicon Valley’s most well-respected Venture Capital funds which allow us to continue building a solid platform, increase the momentum and scale our product and organization.

For more information, visit https://www.sqreen.io.

The role

As part of our product engineering team, you will be involved in building the the core technology that protects our users' applications. New features are released in coordination with other teams at Sqreen. The software that you build has a direct impact on our users’ web applications in production.

You will be in charge of designing, implementing and challenging our security logic, optimizing our agents performance, and leading our Python efforts at Sqreen. You will also be involved with our developers community and will be able to iterate fast with our users feedback.


What you'll be working on

  • Work closely with our security, back-end and agents technologies teams to architect, build, and launch new features.
  • Debug production issues across multiple components.
  • Coordinate with our developers community and travel occasionally to developer conferences and meetups.
  • Contribute and improve our engineering standards, tooling, and processes.


Who we are looking for

  • Solid experience with Python programming and internals.
  • Hold yourself to a high bar when working with production systems.
  • Have a strong interest in security.
  • Are comfortable with Unix environments.
  • Have experience in developing, maintaining and running back end applications
  • Enjoy working with groups of people with different expertise. Every engineer at Sqreen collaborates with Sales, Support and other teams.
  • Have excellent communication skills (English).
  • Have 5+ years experience building applications and know the systems you’ve worked on from top to bottom.


What we can offer

  • Having the opportunity to dig into Python internals and secrets
  • Become part of a super passionate team that loves their jobs and company :-)
  • Beautiful office in central Paris (Sentier), in the heart of #FrenchTech
  • Premium health insurance and coached SqreenFit sessions with the team (crossfit)
  • Organic snacks & beverages and much more
  • English speaking work environment

Interested? Check us out at our twitter account. We’d love to hear from you!

Senior Back End Software Engineer

Description

About

Sqreen is a developer platform which detects security anomalies in web applications and provides automated responses in real-time. We improve the security standards of the world's most famous applications and help developers around the world doing a better job, every day. Founded by former security experts at Apple, Sqreen protects hundreds of apps and companies like Algolia, Front, Toptal, and Helpling and has offices in San Francisco and Paris.

Being initially featured in TechCrunch’s Disrupt SF Battlefield in 2016, the company recently graduated YCombinator and secured its soon-to-be-announced funding round! We’re proud to be backed by Silicon Valley’s most well-respected Venture Capital funds which allow us to continue building a solid platform, increase the momentum and scale our product and organisation.

For more information, visit https://www.sqreen.io.

The role

As part of our product engineering team, you will be involved in building the back end analysing billions of security events and metrics sent minutely by our agents. This analysis is performed in near real time in order to detect threats and to allow our customer's applications to take an action.

You will work closely with security engineers, agents technologies engineers and dashboard engineers to architect, build, and launch new features. You will also be involved with our developers community and will be able to iterate fast with our users feedback.

What you will be working on

  • Work closely with our security, back-end and agents technologies teams to architect, build, and launch new features.
  • Handle and evolve production infrastructure.
  • Coordinate with our developers community and travel occasionally to developer conferences and meetups.
  • Contribute and improve our engineering standards, tooling, and processes.

What we are looking for

  • Solid experience working with production back end.
  • Hold yourself to a high bar when working with production systems.
  • Proficiency in Unix environments.
  • Experience in developing, maintaining and running back end applications.
  • Enjoy working with groups of people with different expertise. Every engineer at Sqreen collaborates with Sales, Support and other teams.
  • Have excellent communication skills (English).
  • Have 5+ years experience building applications and know the systems you’ve worked on from top to bottom.

What we can offer

  • Become part of a super passionate team that loves their jobs and company :-)
  • Beautiful office in central Paris (Sentier), in the heart of #FrenchTech
  • Premium health insurance and coached SqreenFit sessions with the team (crossfit)
  • Organic snacks & beverages and much more
  • English speaking work environment

Interested? Check us out at our twitter account. We’d love to hear from you!

Software Engineer - Frontend

Description

About

Sqreen is a developer platform which detects security anomalies in web applications and provides automated responses in real-time. We improve the security standards of the world's most famous applications and help developers around the world doing a better job, every day. Founded by former security experts at Apple, Sqreen protects hundreds of apps and companies like Algolia, Front, Toptal, and Helpling and has offices in San Francisco and Paris.

Being initially featured in TechCrunch’s Disrupt SF Battlefield in 2016, the company recently graduated YCombinator and secured its soon-to-be-announced funding round! We’re proud to be backed by Silicon Valley’s most well-respected Venture Capital funds which allow us to continue building a solid platform, increase the momentum and scale our product and organization.

For more information, visit https://www.sqreen.io.

The role

You will be responsible for developing the Sqreen dashboard. It is a critical product at Sqreen since it's the tip of the iceberg: all of Sqreen's complexity needs to be tackled in this interface to provide clear and actionable information to our customers.

You love web technologies, you closely follow the evolution of browsers, web technologies, and the React ecosystem.

You’ll take part in the Sqreen engineering team based in Paris. You will have a strong focus on reliability, performance and modern software engineering practices. You’ll become actor in Sqreen product roadmap by suggesting new ideas and features to implement.

Give a try now to the Sqreen dashboard, the main product you'll be contributing to!


What you will be working on

  • Work closely with Product, Backend, Design and Marketing teams to build features Sqreen users love.
  • Build and test robust, well structured, fast and reusable components.
  • Identify and solve performance issues.
  • Participate in the project's design and code reviews (Github).
  • Proactively propose major changes (architecture, frameworks).

Who we are looking for

  • 3 years experience building modern web applications.
  • Proficient with JavaScript, ES6, React, Redux and a variety of Javascript libraries.
  • Comfortable working with HTTP REST APIs.
  • Good knowledge of HTML, CSS, and related web technologies.
  • Awareness of cross-browser compatibility issues and client-side performance. considerations.
  • Bachelor's or Master's degree in computer science, computer engineering, statistics, mathematics or a similar field.
  • Proficiency in written and spoken English.

What we can offer

  • The opportunity to work on a highly technical dashboard that is intended for developers like you
  • Become part of a super passionate team that loves their jobs and company :-)
  • Beautiful office in central Paris (Sentier), in the heart of #FrenchTech
  • Premium health insurance and coached SqreenFit sessions with the team (crossfit)
  • Organic snacks & beverages and much more
  • English speaking work environment

Interested? Check us out at our twitter account. We’d love to hear from you!


Verified by
Co-founder & CEO
You may also like