Observability with the ELK Stack

4,478
Elastic
Creators of ELK / Elastic Stack (Elasticsearch, Logstash, Kibana, Beats & More)

Written By Tanya Bragin, Product Lead, Elastic


In my role as a Product Lead for Observability at Elastic, I get a few different reactions when I use the term 'observability'. The most common reaction by far today still is: "What is 'observability'?" But I also increasingly hear things like: "We just kicked-off an 'observability initiative', but we're still figuring out exactly how to go about it." And finally, some organizations we have been fortunate to work with already consider 'observability' an integral part of how they design and build products and services.

Given that the term is still gaining traction, I thought it would be useful to demystify how we at Elastic view 'observability', what we learned from our thought-leading customers, and how we think about it from the product perspective as we evolve our stack for operational use cases.

What is 'Observability'?

We certainly did not invent the term 'observability'. We started hearing about it from users, primarily those within the Site Reliability Engineering (SRE) community. Several sources trace back beginnings of this term to SRE organizations from Silicon Valley giants like Twitter. And even though the seminal Google SRE Book does not mention the term, it lays out many of the principles associated with 'observability' today.

'Observability' is not something that a vendor delivers in a box -- it is an attribute of a system you build, much like usability, high availability, and stability. The goal of designing and building an 'observable' system is to make sure that when it is run in production, operators responsible for it can detect undesirable behaviors (e.g. service downtime, errors, slow responses) and have actionable information to pin down root cause in an effective manner (e.g. detailed event logs, granular resource usage information, and application traces). Common challenges preventing organizations from achieving this seemingly obvious goals include not collecting enough information, collecting too much information, but not making it actionable, and fragmenting access to this information.

The first aspect — detection of undesirable behaviors — usually starts with setting of Service Level Indicators (SLIs) and Objectives (SLOs). These are internal measures of success by which production systems are judged in observability-minded organizations. If there is a contractual obligation to fulfill these objectives, an SLI/SLO may also translate to a Service Level Agreements (SLAs). The most common example of an SLI is system uptime, for which you may set an SLO of 99.9999%. System uptime is also the most common SLA exposed to external customers. However, your SLI/SLOs internally may be a lot more granular, and monitoring and alerting on these most important factors of production system behavior is the basis of any observability initiative. This aspect of observability is also known by the term "monitoring".

The second aspect — providing operators with granular information to debug production issues quickly and efficiently — is an area where we see a lot of movement and innovation. There is quite a bit of talk about the "three pillars of observability" — metrics, logs, and application traces. There is also recognition that simply collecting all this granular data using a patchwork of tools is not necessarily actionable and often not cost effective.

'Pillars' of Observability

Let's examine these data collection aspects in more detail. The status quo we typically encounter today is to collect metrics into one system (usually a time series database or a SaaS service for resource monitoring), collect logs into a second system (unsurprisingly, often the ELK stack in our conversations), and to use yet a third tool to instrument applications to provide request level tracing. When an alert fires, indicating a breach in a service level, operators madly dart over to their systems and perform the best "swivel chair integration" they can -- looking at metrics in one browser window, manually correlating it to logs in another window, and pulling up traces (if relevant) in yet a third window.

This approach has several drawbacks. First, manual correlation of different data sources all telling the same story wastes valuable time during service degradation or outage. Second, operational costs of maintaining three different operational data stores are onerous — licensing costs, separate headcount for administrators of disparate operational tools, inconsistent machine learning capabilities in each datastore, "headspace" for thinking through different semantics for alerting — every organization I speak with struggles with all of these challenges.

There is an increasing recognition of how important it is to have all this information in a single operational store with the ability to automatically correlate this data in an intuitive user interface. Nirvana for the users we talk to is to expose their operators to every piece of data relevant to the service they are supporting in a unified way, whether it be a log line emitted by the application, trace data resulting from instrumentation, or resource utilization represented by metrics in a time series. Requirements we hear about stress uniform, ad-hoc access to this data regardless of the source, from search and filtering, to aggregations, to visualizations. Starting with metrics and drilling into logs and traces in a few clicks without switching context accelerates investigations. Similarly, extracting numerical values from structured logs looks surprisingly like metrics and visualizing both side-by-side has tremendous value from an operational perspective.

As mentioned before, simply collecting the data may result in too much information on disk and not enough actionable intelligence when an incident occurs. Increasingly, there is an expectation that the system collecting operational data provides automatic detection of "interesting" events, traces, and anomalies in the patterns of time series. This helps operators investigating a problem zero in on the root cause faster. These anomaly detection capabilities are sometimes referred to as the "fourth pillar of observability". Detecting anomalies across uptime data, resource utilization, anomalies in logging patterns, and most relevant traces is an emerging requirement observability teams put forth.

Observability... and the ELK Stack?

So what does observability have to do with the Elastic Stack (or ELK Stack, as it's lovingly referred to in operational circles)?

ELK Stack is widely known as the de facto way to centralize logs from operational systems. The assumption is that Elasticsearch (a "search engine") is a good place to put text-based logs for the purposes of free-text search. And indeed, simply searching text-based logs for the word "error" or filtering logs based on a set of a well-known tags is extremely powerful, and is often where most users start.

However, as most ELK Stack users know, Elasticsearch as a datastore offers a lot more than an inverted index for efficient full-text search and simple filtering abilities. It also contains a columnar store optimized for storing and operating on dense numerical time series. This columnar store is used to store structure data extracted from parsed logs, both string and numerical. In fact, the use case of converting logs to metrics is what initially drove us to optimize Elasticsearch for efficient storage and retrieval of numbers.

Over time, users started putting numerical time series directly into Elasticsearch, replacing legacy time series databases. Driven by this need, Elastic recently introduced Metricbeat for automated collection of metrics, the concept of automatic rollups, and other metrics-specific functionality both in the datastore and the UI. As a result, increasingly more users that have adopted the ELK Stack for logs, have also started putting metric data, such as resource utilization, into the Elastic Stack. In addition to operational savings already mentioned above, one attractive reason for this was lack of restrictions Elasticsearch places on cardinality of fields eligible for numerical aggregations (a common gripe brought up when discussing many existing time series databases).

Similar to metrics, uptime data has been a highly valued type of data alongside logs, representing an important source of SLO/SLI alerts from an active monitor. Uptime data can provide information about degradation of services, APIs, and websites, oftentimes before the users feel the impact. The bonus is that uptime data is tiny in terms of storage requirements, so a lot of value for very little additional cost.

Within the past year Elastic has also introduced Elastic APM, adding application tracing and distributed tracing capabilities to the stack. This was a natural evolution for us, as several open-source projects and prominent APM vendors were already using Elasticsearch to store and search trace data. Status quo in traditional APM tools is to keep APM trace data separate from logs and metrics, perpetuating operational data silos. Elastic APM offers a set of agents for collecting trace data from supported languages and frameworks as well as supporting OpenTracing, and this trace data is automatically correlated with the metrics and logs.

A common thread across all these data inputs is that each of them is just another index in Elasticsearch. There are no restrictions on aggregations you run on all this data data, how you visualize it in Kibana, and how alerting and machine learning applies to each data source. To see this in action, check out this video.

Observable Kubernetes and the Elastic Stack

One community where the concept of observability is a very active topic of conversation is the set of users adopting Kubernetes for container orchestration. These "cloud native" users, a term popularized by the Cloud Native Computing Foundation (or CNCF), face unique challenges. They face a massive centralization of applications and services built on or migrated to a Kubernetes-powered container orchestration platform, coupled with the trend to split up monolithic apps into "microservices". Tools and methods that worked before to provide necessary visibility into applications running on top of this infrastructure no longer work.

Kubernetes observability deserves a separate post all on its own, so for now I will refer you to the Observable Kubernetes webinar and the Distributed Tracing with Elastic APM blog post for more information.

What's next?

In a post like this, it seems appropriate to leave the reader with a few resources to explore.

To learn more about observability best practices, I recommend starting with the above-mentioned Google SRE Book. Blog posts from companies whose livelihood depends on flawless operation of their critical apps in production are also typically very thought-provoking. For example, I find this recent post by Salesforce engineering to be a pragmatic and practical guide to iteratively improving the state of observability.

To try out Elastic Stack capabilities for your observability initiatives, spin up the latest version of our stack on the Elasticsearch Service on Elastic Cloud (great sandbox even if ultimately you deploy self-managed), or download and install Elastic Stack components locally. Make sure to check out the new Logs, Infrastructuremonitoring, APM, and Uptime (coming soon in 6.7) UIs in Kibana, purpose-built for common observability workflows. And feel free to ping us with questions on Discuss forums — we're there to help!

Elastic
Creators of ELK / Elastic Stack (Elasticsearch, Logstash, Kibana, Beats & More)
Tools mentioned in article
Open jobs at Elastic
Cloud Workload - Backend Engineer
United States
<div class="content-intro"><p>Elastic is a free and open search company that powers enterprise search, observability, and security solutions built on one technology stack that can be deployed anywhere. From finding documents to monitoring infrastructure to hunting for threats, Elastic makes data usable in real-time and at scale. Thousands of organizations worldwide, including Barclays, Cisco, eBay, Fairfax, ING, Goldman Sachs, Microsoft, The Mayo Clinic, NASA, The New York Times, Wikipedia, and Verizon, use Elastic to power mission-critical systems. Founded in 2012, <strong><a href="https://www.elastic.co/about/distributed" target="_blank">Elastic is a distributed company</a></strong> with Elasticians around the globe. Learn more at <strong><a href="https://www.elastic.co/" target="_blank">elastic.co</a></strong>.</p></div><p><span style="font-weight: 400;">We are looking for an outstanding Software Engineer to join our Elastic Security Solution team. This candidate will be responsible for designing and developing technical solutions related to cloud runtime security on the Adaptive Workload Protection team (AWP).</span></p> <p><span style="font-weight: 400;">The AWP team provides visibility, instrumentation and protection for dynamic cloud workloads and infrastructure. Our solution aims to observe and protect our customers’ cloud workloads running in Kubernetes, and virtual machines.&nbsp;</span></p> <p><span style="font-weight: 400;">A part of the AWP team is the Infrastructure and Response Team whose mandate is to move Elastic Security Runtime protection to support the major cloud providers. Our aim is to create a new vertical within Elastic Security that will see cloud assets being protected in real-time.</span></p> <p><span style="font-weight: 400;">The right candidate will be empathetic and thoughtful; Matched with their technical abilities in building microservices that scale and their knowledge of container orchestration technologies. They will help design and develop *new* products to support Cloud Security Runtime Protection. They will work alongside Product Managers, our Linux Kernel and external teams driving Cloud support within Elastic Security.</span></p> <h2><strong>What You Will Be Doing</strong></h2> <p><span style="font-weight: 400;">At Elastic, we want you to succeed, and we’ll be here to support your journey. You’ll have the opportunity to</span></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Work alongside a team of experienced Backend and Infrastructure Engineers to develop observability and security features within new services created by the team.</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Innovate and contribute to the Cloud Security experience by developing microservices in Go, and Rust</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Support external teams by creating tooling and consumable infrastructure with your experience in Kubernetes and Terraform</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Help define the future of runtime protection in Cloud Security while embracing continuously-evolving modern backend and infrastructure technologies</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Write comprehensive tests, including unit, service-level, and end-to-end tests, that provides confidence in the stability and function of the service the team creates and releases</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Collaborate with other infrastructure and control plane teams to innovate on different deployment models as Elastic Security evolves to natively support cloud environments.</span></li> </ul> <h2><strong>What You Bring Along</strong></h2> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Strong system-level programming skills [Go, C++, Rust]</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Deep understanding of&nbsp; Kubernetes internals</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Excellent debugging skills in a Linux environment</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Deep understanding of at least one of the major cloud providers and their cloud-native architecture, user management, and APIs interactions.</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Ability to work independently</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience with git</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Years of experience developing scalable, high-performing services and applications</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Excellent verbal and written English communication skills</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Strong analytical, problem solving, debugging and troubleshooting skills&nbsp;</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience writing automated tests</span></li> </ul> <p><strong>Nice to Have</strong></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Kubernetes experience developing Controllers and/or Operators</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Practical experience developing immutable infrastructure</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Record of green-fields rapid development and prototyping</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience using or running the Elastic Stack, Agent and/or Endpoint</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience with data visualizations</span></li> </ul> <p>&nbsp;</p><div class="content-conclusion"><h2><strong>Additional Information - We Take Care of Our People</strong></h2> <p>As a distributed company, diversity drives our identity. Whether you’re looking to launch a new career or grow an existing one, Elastic is the type of company where you can balance great work with great life. Your age is only a number. It doesn’t matter if you’re just out of college or your children are; we need you for what you can do.</p> <p>We strive to have parity of benefits across regions and while regulations differ from place to place, we believe taking care of our people is the right thing to do.</p> <ul> <li>Competitive pay based on the work you do here and not your previous salary</li> <li>Health coverage for you and your family in many locations</li> <li>Ability to craft your calendar with flexible locations and schedules for many roles</li> <li>Generous number of vacation days each year</li> <li>Double your charitable giving - We match up to $1500 (or local currency equivalent)</li> <li>Up to 40 hours each year to use toward volunteer projects you love</li> <li>Embracing parenthood with minimum of 16 weeks of parental leave</li> </ul> <p><span style="font-weight: 400;">Different people approach problems differently. We need that. Elastic is committed to diversity as well as inclusion. We are an equal opportunity employer and committed to the principles of affirmative action. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender perception or identity, national origin, age, marital status, protected veteran status, or disability status or any other basis protected by federal, state or local law, ordinance or regulation. If you require any reasonable accessibility support, please complete our <a href="https://docs.google.com/forms/d/e/1FAIpQLSfiV7br5dGLslC__KPuDNBii1x6cQnlm59L8b4293O2eFxbMA/viewform" target="_blank">Candidate Accessibility Request Form</a>.&nbsp;</span></p> <p><span style="color: #000000;">Applicants have rights under Federal Employment Laws, view posters linked below:</span> <a href="https://www.dol.gov/sites/dolgov/files/WHD/legacy/files/fmlaen.pdf" target="_blank">Family and Medical Leave Act (FMLA)</a> <span style="color: #000000;">Poster; </span><a href="https://www.eeoc.gov/sites/default/files/migrated_files/employers/poster_screen_reader_optimized.pdf" target="_blank">Equal Employment Opportunity (EEO)</a> <span style="color: #000000;">Poster; and</span> <a href="https://www.dol.gov/sites/dolgov/files/WHD/legacy/files/eppabw.pdf" target="_blank">Employee Polygraph Protection Act (EPPA)</a> <span style="color: #000000;">Poster.</span></p> <p>Please see&nbsp;<a href="https://www.elastic.co/legal/applicant-privacy-statement" target="_blank" data-saferedirecturl="https://www.google.com/url?q=https://www.elastic.co/legal/applicant-privacy-statement&amp;source=gmail&amp;ust=1611068262479000&amp;usg=AFQjCNG3orAPk5sJIr9WVm0ZDxYw82c5DA">here</a>&nbsp;for our Privacy Statement.</p></div>
Cloud Workload - Backend Engineer
Canada
<div class="content-intro"><p>Elastic is a free and open search company that powers enterprise search, observability, and security solutions built on one technology stack that can be deployed anywhere. From finding documents to monitoring infrastructure to hunting for threats, Elastic makes data usable in real-time and at scale. Thousands of organizations worldwide, including Barclays, Cisco, eBay, Fairfax, ING, Goldman Sachs, Microsoft, The Mayo Clinic, NASA, The New York Times, Wikipedia, and Verizon, use Elastic to power mission-critical systems. Founded in 2012, <strong><a href="https://www.elastic.co/about/distributed" target="_blank">Elastic is a distributed company</a></strong> with Elasticians around the globe. Learn more at <strong><a href="https://www.elastic.co/" target="_blank">elastic.co</a></strong>.</p></div><p><span style="font-weight: 400;">We are looking for an outstanding Software Engineer to join our Elastic Security Solution team. This candidate will be responsible for designing and developing technical solutions related to cloud runtime security on the Adaptive Workload Protection team (AWP).</span></p> <p><span style="font-weight: 400;">The AWP team provides visibility, instrumentation and protection for dynamic cloud workloads and infrastructure. Our solution aims to observe and protect our customers’ cloud workloads running in Kubernetes, and virtual machines.&nbsp;</span></p> <p><span style="font-weight: 400;">A part of the AWP team is the Infrastructure and Response Team whose mandate is to move Elastic Security Runtime protection to support the major cloud providers. Our aim is to create a new vertical within Elastic Security that will see cloud assets being protected in real-time.</span></p> <p><span style="font-weight: 400;">The right candidate will be empathetic and thoughtful; Matched with their technical abilities in building microservices that scale and their knowledge of container orchestration technologies. They will help design and develop *new* products to support Cloud Security Runtime Protection. They will work alongside Product Managers, our Linux Kernel and external teams driving Cloud support within Elastic Security.</span></p> <h2><strong>What You Will Be Doing</strong></h2> <p><span style="font-weight: 400;">At Elastic, we want you to succeed, and we’ll be here to support your journey. You’ll have the opportunity to</span></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Work alongside a team of experienced Backend and Infrastructure Engineers to develop observability and security features within new services created by the team.</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Innovate and contribute to the Cloud Security experience by developing microservices in Go, and Rust</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Support external teams by creating tooling and consumable infrastructure with your experience in Kubernetes and Terraform</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Help define the future of runtime protection in Cloud Security while embracing continuously-evolving modern backend and infrastructure technologies</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Write comprehensive tests, including unit, service-level, and end-to-end tests, that provides confidence in the stability and function of the service the team creates and releases</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Collaborate with other infrastructure and control plane teams to innovate on different deployment models as Elastic Security evolves to natively support cloud environments.</span></li> </ul> <h2><strong>What You Bring Along</strong></h2> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Strong system-level programming skills [Go, C++, Rust]</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Deep understanding of&nbsp; Kubernetes internals</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Excellent debugging skills in a Linux environment</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Deep understanding of at least one of the major cloud providers and their cloud-native architecture, user management, and APIs interactions.</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Ability to work independently</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience with git</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Years of experience developing scalable, high-performing services and applications</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Excellent verbal and written English communication skills</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Strong analytical, problem solving, debugging and troubleshooting skills&nbsp;</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience writing automated tests</span></li> </ul> <p><strong>Nice to Have</strong></p> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Kubernetes experience developing Controllers and/or Operators</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Practical experience developing immutable infrastructure</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Record of green-fields rapid development and prototyping</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience using or running the Elastic Stack, Agent and/or Endpoint</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience with data visualizations</span></li> </ul> <p>&nbsp;</p><div class="content-conclusion"><h2><strong>Additional Information - We Take Care of Our People</strong></h2> <p>As a distributed company, diversity drives our identity. Whether you’re looking to launch a new career or grow an existing one, Elastic is the type of company where you can balance great work with great life. Your age is only a number. It doesn’t matter if you’re just out of college or your children are; we need you for what you can do.</p> <p>We strive to have parity of benefits across regions and while regulations differ from place to place, we believe taking care of our people is the right thing to do.</p> <ul> <li>Competitive pay based on the work you do here and not your previous salary</li> <li>Health coverage for you and your family in many locations</li> <li>Ability to craft your calendar with flexible locations and schedules for many roles</li> <li>Generous number of vacation days each year</li> <li>Double your charitable giving - We match up to $1500 (or local currency equivalent)</li> <li>Up to 40 hours each year to use toward volunteer projects you love</li> <li>Embracing parenthood with minimum of 16 weeks of parental leave</li> </ul> <p><span style="font-weight: 400;">Different people approach problems differently. We need that. Elastic is committed to diversity as well as inclusion. We are an equal opportunity employer and committed to the principles of affirmative action. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender perception or identity, national origin, age, marital status, protected veteran status, or disability status or any other basis protected by federal, state or local law, ordinance or regulation. If you require any reasonable accessibility support, please complete our <a href="https://docs.google.com/forms/d/e/1FAIpQLSfiV7br5dGLslC__KPuDNBii1x6cQnlm59L8b4293O2eFxbMA/viewform" target="_blank">Candidate Accessibility Request Form</a>.&nbsp;</span></p> <p><span style="color: #000000;">Applicants have rights under Federal Employment Laws, view posters linked below:</span> <a href="https://www.dol.gov/sites/dolgov/files/WHD/legacy/files/fmlaen.pdf" target="_blank">Family and Medical Leave Act (FMLA)</a> <span style="color: #000000;">Poster; </span><a href="https://www.eeoc.gov/sites/default/files/migrated_files/employers/poster_screen_reader_optimized.pdf" target="_blank">Equal Employment Opportunity (EEO)</a> <span style="color: #000000;">Poster; and</span> <a href="https://www.dol.gov/sites/dolgov/files/WHD/legacy/files/eppabw.pdf" target="_blank">Employee Polygraph Protection Act (EPPA)</a> <span style="color: #000000;">Poster.</span></p> <p>Please see&nbsp;<a href="https://www.elastic.co/legal/applicant-privacy-statement" target="_blank" data-saferedirecturl="https://www.google.com/url?q=https://www.elastic.co/legal/applicant-privacy-statement&amp;source=gmail&amp;ust=1611068262479000&amp;usg=AFQjCNG3orAPk5sJIr9WVm0ZDxYw82c5DA">here</a>&nbsp;for our Privacy Statement.</p></div>
Data Science Engineer
United States
<div class="content-intro"><p>Elastic is a free and open search company that powers enterprise search, observability, and security solutions built on one technology stack that can be deployed anywhere. From finding documents to monitoring infrastructure to hunting for threats, Elastic makes data usable in real-time and at scale. Thousands of organizations worldwide, including Barclays, Cisco, eBay, Fairfax, ING, Goldman Sachs, Microsoft, The Mayo Clinic, NASA, The New York Times, Wikipedia, and Verizon, use Elastic to power mission-critical systems. Founded in 2012, <strong><a href="https://www.elastic.co/about/distributed" target="_blank">Elastic is a distributed company</a></strong> with Elasticians around the globe. Learn more at <strong><a href="https://www.elastic.co/" target="_blank">elastic.co</a></strong>.</p></div><p><span style="font-weight: 400;">Elastic is growing quickly and the organization is evolving to support more complexity in offerings and go to market motions. In order to achieve results, we use a wide variety of programs that impact different areas of the Sales and Marketing funnels (from branding to pipeline generation, acceleration and closing).</span></p> <p><span style="font-weight: 400;">Elastic is hiring a Data Science Engineer that can help us analyze trends, identify patterns, and extract valuable insights from our data that will support business decision-making. You will be responsible for maintaining and developing predictive models and machine-learning algorithms that can be used to enhance and improve our ETL pipeline, certified datasets, and business processes. This role is pivotal in helping us to achieve prescriptive and predictive analytics. You will work closely with multiple domains of the business, including our Demand Generation, Sales, Field, and Partnership teams.</span></p> <p><span style="font-weight: 400;">We are looking for someone who is analytically and technically minded, has a deep understanding of mathematics and statistics, is able to successfully work with multiple teams, and able to translate and communicate findings into practical insights for team members. We want someone with a passion for learning new techniques and creative problem solving.</span></p> <h2><strong>Key Responsibilities Include:</strong></h2> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Analyze data to discover significant trends and patterns</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Build and maintain predictive models, machine-learning algorithms</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Propose solutions and strategies to business challenges</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Present information using best-practice data visualization techniques</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Perform data analysis, quality checks, and debugging associated with Python Apps</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Help to establish and communicate analytics and engineering-related standards and best-practices</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Provide Ad hoc analysis, troubleshooting and assistance for both our and extended teams</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Collaborate with Engineering, Analytics, and Marketing/Sales Operations</span></li> </ul> <h2><strong>Qualifications:</strong></h2> <ul> <li> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Proven experience in Data Science, Machine Learning, and/or Analytics Engineering</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Expertise with SQL, Python, and R, BigQuery preferred</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Understanding of machine-learning, Operations Research, and Database Systems</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Strong Understanding of mathematical concepts used in Algebra and Statistics</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience with data warehousing architecture and data modeling</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Strong analytical and interpersonal skills and a willingness to take initiative and chip in beyond basic responsibilities</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience with cloud providers, BigQuery preferred</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience with Google Cloud MLE, Alteryx R/predictive tool suite, Docker Containerization in Kubernetes, or APIs are all a plus</span></li> </ul> </li> </ul> <p><span style="font-weight: 400;">#LI-JM5</span></p><div class="content-conclusion"><h2><strong>Additional Information - We Take Care of Our People</strong></h2> <p>As a distributed company, diversity drives our identity. Whether you’re looking to launch a new career or grow an existing one, Elastic is the type of company where you can balance great work with great life. Your age is only a number. It doesn’t matter if you’re just out of college or your children are; we need you for what you can do.</p> <p>We strive to have parity of benefits across regions and while regulations differ from place to place, we believe taking care of our people is the right thing to do.</p> <ul> <li>Competitive pay based on the work you do here and not your previous salary</li> <li>Health coverage for you and your family in many locations</li> <li>Ability to craft your calendar with flexible locations and schedules for many roles</li> <li>Generous number of vacation days each year</li> <li>Double your charitable giving - We match up to $1500 (or local currency equivalent)</li> <li>Up to 40 hours each year to use toward volunteer projects you love</li> <li>Embracing parenthood with minimum of 16 weeks of parental leave</li> </ul> <p><span style="font-weight: 400;">Different people approach problems differently. We need that. Elastic is committed to diversity as well as inclusion. We are an equal opportunity employer and committed to the principles of affirmative action. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender perception or identity, national origin, age, marital status, protected veteran status, or disability status or any other basis protected by federal, state or local law, ordinance or regulation. If you require any reasonable accessibility support, please complete our <a href="https://docs.google.com/forms/d/e/1FAIpQLSfiV7br5dGLslC__KPuDNBii1x6cQnlm59L8b4293O2eFxbMA/viewform" target="_blank">Candidate Accessibility Request Form</a>.&nbsp;</span></p> <p><span style="color: #000000;">Applicants have rights under Federal Employment Laws, view posters linked below:</span> <a href="https://www.dol.gov/sites/dolgov/files/WHD/legacy/files/fmlaen.pdf" target="_blank">Family and Medical Leave Act (FMLA)</a> <span style="color: #000000;">Poster; </span><a href="https://www.eeoc.gov/sites/default/files/migrated_files/employers/poster_screen_reader_optimized.pdf" target="_blank">Equal Employment Opportunity (EEO)</a> <span style="color: #000000;">Poster; and</span> <a href="https://www.dol.gov/sites/dolgov/files/WHD/legacy/files/eppabw.pdf" target="_blank">Employee Polygraph Protection Act (EPPA)</a> <span style="color: #000000;">Poster.</span></p> <p>Please see&nbsp;<a href="https://www.elastic.co/legal/applicant-privacy-statement" target="_blank" data-saferedirecturl="https://www.google.com/url?q=https://www.elastic.co/legal/applicant-privacy-statement&amp;source=gmail&amp;ust=1611068262479000&amp;usg=AFQjCNG3orAPk5sJIr9WVm0ZDxYw82c5DA">here</a>&nbsp;for our Privacy Statement.</p></div>
Senior Full Stack Engineer (R5109)
Distributed, EMEA
<div class="content-intro"><p>Elastic is a free and open search company that powers enterprise search, observability, and security solutions built on one technology stack that can be deployed anywhere. From finding documents to monitoring infrastructure to hunting for threats, Elastic makes data usable in real-time and at scale. Thousands of organizations worldwide, including Barclays, Cisco, eBay, Fairfax, ING, Goldman Sachs, Microsoft, The Mayo Clinic, NASA, The New York Times, Wikipedia, and Verizon, use Elastic to power mission-critical systems. Founded in 2012, <strong><a href="https://www.elastic.co/about/distributed" target="_blank">Elastic is a distributed company</a></strong> with Elasticians around the globe. Learn more at <strong><a href="https://www.elastic.co/" target="_blank">elastic.co</a></strong>.</p></div><p><span style="font-weight: 400;">The Observability team is in charge of developing solutions that focus on application developers and engineers that run infrastructure and services supporting these applications. Elasticsearch is an efficient datastore for logs, metrics, and application traces, supporting the three pillars of observability. The Observability team builds and maintains solutions that make getting insights from this data turnkey and efficient, such as our </span><a href="https://www.elastic.co/solutions/apm"><span style="font-weight: 400;">APM</span></a><span style="font-weight: 400;">, </span><a href="https://www.elastic.co/solutions/metrics"><span style="font-weight: 400;">Infrastructure Monitoring</span></a><span style="font-weight: 400;">, </span><a href="https://www.elastic.co/solutions/logging"><span style="font-weight: 400;">Logs</span></a><span style="font-weight: 400;">, and </span><a href="https://www.elastic.co/uptime-monitoring"><span style="font-weight: 400;">Uptime</span></a><span style="font-weight: 400;"> solutions. When developing these solutions, we think about the problem end-to-end: how do we automatically collect data from common data sources, how do we store it efficiently in Elasticsearch, how do we present this information to the user, what actions do we take on the insights from the data? All of these aspects are important in bringing a turnkey solution to the market.&nbsp;</span></p> <p><span style="font-weight: 400;">As a Full-Stack Engineer on the Observability team, you'll primarily be working on UIs and APIs in Kibana devoted to curated observability users. You'll be working with the UI team and product management to build high quality, intuitive and innovative UIs and robust API to drive the experience.</span></p> <h3><strong>Engineering philosophy</strong></h3> <p><span style="font-weight: 400;">We believe that engineering complex, pluggable software for the web that is built to last the test of time is both tricky and exciting. Doing so requires a team of diverse individuals, with sharp minds and the ability to empathize with our users, working together with mutual respect and a common mission.</span></p> <p><span style="font-weight: 400;">We care deeply about giving you full ownership of what you're working on. Our company fundamentally believes great minds achieve greatness when they are set free and are surrounded and challenged by their peers, which is clearly visible throughout our organization. At Elastic, hierarchy does not determine how decisions get made. We feel that anyone needs to be in the position to comment on anything, regardless of their role within the company.</span></p> <h3><strong>What you will be doing</strong></h3> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Work on new and exciting Elastic products.</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Develop the APM, Logs, Metrics, Integrations, Agent Management UIs and APIs in Kibana.</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Collaborate with fellow JavaScript engineers.</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Collaborate with product design, server team, agent engineers and the Kibana team.</span></li> </ul> <h3><strong>What you will bring along</strong></h3> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">JavaScript programming skills</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience with one modern MVC style JS Framework</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Ability to work in a fast paced start-up environment</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Excellent verbal and written communication skills, a great teammate with strong analytical, problem solving, debugging and troubleshooting skills</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Strong preference for Open Source experience</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Ability to work independently</span></li> </ul> <h3><strong>You might have</strong></h3> <ul> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience with React and Typescript</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience with Node.js</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Experience writing automated tests</span></li> <li style="font-weight: 400;"><span style="font-weight: 400;">Exposure to Observability, Performance or Availability tools or services</span></li> </ul> <p>&nbsp;</p><div class="content-conclusion"><h2><strong>Additional Information - We Take Care of Our People</strong></h2> <p>As a distributed company, diversity drives our identity. Whether you’re looking to launch a new career or grow an existing one, Elastic is the type of company where you can balance great work with great life. Your age is only a number. It doesn’t matter if you’re just out of college or your children are; we need you for what you can do.</p> <p>We strive to have parity of benefits across regions and while regulations differ from place to place, we believe taking care of our people is the right thing to do.</p> <ul> <li>Competitive pay based on the work you do here and not your previous salary</li> <li>Health coverage for you and your family in many locations</li> <li>Ability to craft your calendar with flexible locations and schedules for many roles</li> <li>Generous number of vacation days each year</li> <li>Double your charitable giving - We match up to $1500 (or local currency equivalent)</li> <li>Up to 40 hours each year to use toward volunteer projects you love</li> <li>Embracing parenthood with minimum of 16 weeks of parental leave</li> </ul> <p><span style="font-weight: 400;">Different people approach problems differently. We need that. Elastic is committed to diversity as well as inclusion. We are an equal opportunity employer and committed to the principles of affirmative action. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender perception or identity, national origin, age, marital status, protected veteran status, or disability status or any other basis protected by federal, state or local law, ordinance or regulation. If you require any reasonable accessibility support, please complete our <a href="https://docs.google.com/forms/d/e/1FAIpQLSfiV7br5dGLslC__KPuDNBii1x6cQnlm59L8b4293O2eFxbMA/viewform" target="_blank">Candidate Accessibility Request Form</a>.&nbsp;</span></p> <p><span style="color: #000000;">Applicants have rights under Federal Employment Laws, view posters linked below:</span> <a href="https://www.dol.gov/sites/dolgov/files/WHD/legacy/files/fmlaen.pdf" target="_blank">Family and Medical Leave Act (FMLA)</a> <span style="color: #000000;">Poster; </span><a href="https://www.eeoc.gov/sites/default/files/migrated_files/employers/poster_screen_reader_optimized.pdf" target="_blank">Equal Employment Opportunity (EEO)</a> <span style="color: #000000;">Poster; and</span> <a href="https://www.dol.gov/sites/dolgov/files/WHD/legacy/files/eppabw.pdf" target="_blank">Employee Polygraph Protection Act (EPPA)</a> <span style="color: #000000;">Poster.</span></p> <p>Please see&nbsp;<a href="https://www.elastic.co/legal/applicant-privacy-statement" target="_blank" data-saferedirecturl="https://www.google.com/url?q=https://www.elastic.co/legal/applicant-privacy-statement&amp;source=gmail&amp;ust=1611068262479000&amp;usg=AFQjCNG3orAPk5sJIr9WVm0ZDxYw82c5DA">here</a>&nbsp;for our Privacy Statement.</p></div>
Verified by
Product Lead, Observability
You may also like