timchunght great question! When we don't have an email address (or company domain) in cache we use our robust worker platform to doing live lookups. We leverage Sidekiq Pro to process millions of Lookup Jobs per day. We can typically finish a Person Lookup in under 3 seconds by leveraging the massive concurrency that Sidekiq affords us.

Most of the the inter-service communication happens from the background jobs.

For each of those jobs we'll ask the following questions:

• Does it make sense to retry the HTTP request?
• How many retries?
• Can the current work complete without a successful response?

We've written some error handling middleware for Sidekiq that allows us to specify what to do in each of the possible error cases.

We have a couple of things written in Node.js, and we've prototyped a few of the services in Golang.

REPOSE looks interesting. Thanks for the link, will need to give that a look too.