TL;DR

We leverage Unleash as a feature flagging platform to control users being assigned to canary deployments of remote microservices. Alongside, we use Mixpanel super-properties to attribute these canary deployments to every single collected behavior metric.

Attributing Remote Service Deployments to In-App User Behavior

Social networks are, by nature, complex systems. No single user action is isolated but is instead part of a large web of social interactions. This complexity means that simply split-testing user behavior by releasing a new system version and observing its effects isn’t straightforward. Any minor change could have unintended ripple effects, potentially contaminating your data.

A viable option is a full-fledged feature-flagged A/B testing setup, ideally regionalized to avoid spillover between users on different flags within the same sub-network. When managing algorithmic backend deployments that frequently affect users, tracking changes can be challenging.

At Doubble, we continuously enhance our mobile app, search, suggestions, and machine-learning-based matching algorithms to improve the retention and engagement of our users. In this post, I’ll show how we’ve built a high-impact feature-flagging system that attributes user behavior to individual microservice deployments, alongside in-app feature flags, with powerful visualizations and breakdowns.

Deploying code, and still being able to sleep at night

When you’ve written code and want to get it to end users, you likely don’t want to publish your changes to production without thorough testing. Most developers, sometimes, will consider maybe testing their changes before they go live. Maybe.

But what do you do when the effects of your changes are only testable in production? For instance, if you run a web shop, and that shiny new elastic-search algorithm you created with ChatGPT needs testing on real users to determine if it actually boosts sales.

In this case, you’re probably looking to deploy your changes to production, but only to a small subset of users.

At Doubble, we call this a canary deployment—a small “experimental” deployment that allows you to ensure your code doesn’t turn your production environment into an invisible dumpster fire minutes after pushing to prod on a Friday.

It all starts with a feature branch

Suppose you’re at Doubble, and you’ve devised a game-changing algorithm that you believe will skyrocket user retention and create more matches. You just need to get this new if-statement running on your microservice infrastructure ASAP.

To run this code, you need a deployment pipeline. At Doubble, we run our pipeline in the cloud, executing static code analysis, end-to-end integration tests, build, and promote a Docker image to our live deployment environments running in Kubernetes. We can get code running with a simple git tag.

Integrating with the Unleash feature flagging system

With your code running on a canary deployment, the next step is where things get interesting.

Unleash is a powerful feature-flagging platform that allows us to control which users are assigned to which feature flags with ease. It offers features like property-based targeting and user-specific flag assignments, along with an edge proxy service with caching capabilities, helping us stay within the default pricing tier (money🤑).

When the deployment pipeline runs, it informs our managed Unleash instance of a new feature flag via Unleash’s designated API. This flag includes the feature branch name and its corresponding internal cluster URL (resolved by the cluster DNS), making it easy to trace any flag back to its feature branch.

Letting end users consume the canary deployment

Now that the new feature is running on its own canary deployment, and our feature flagging service has updated the new feature flag with connection info, we need to direct traffic to the canary deployment.

When an end user wants to consume the service, the Gateway Service API will ask the Unleash Edge Proxy if the user should be assigned to the canary deployment. The edge proxy uses our external Redis cluster to store the feature flag state, resolving the flag based on the user’s unique ID. If the user is assigned to the canary, their traffic routes to the canary deployment; otherwise, it routes to the stable deployment.

The user’s client is not entirely oblivious to the canary deployment. The clients consume the Unleash edge-proxy as well, gathering all registered feature flags, including those affecting their canary deployment.

Sharing feature flag information with Mixpanel analytics platform

Event properties are crucial for data-driven decision-making, as they allow you to break down data based on these properties. Here, the clients query all feature flags from Unleash and maintain this information as Mixpanel super-properties. Super-properties are event properties sent with every forthcoming generated event.

For instance, if we want to see if people swipe more or less when assigned to a given canary deployment, we can break down our swipe events by the canary-deployment super-property.

User behavior attribution using Mixpanel Super properties

With feature flag information, events are sent to the Mixpanel API, storing events alongside super-properties. Whether tracking users sending messages, rating the app 5 stars, or leaving the app sooner than average, we can attribute these events to the exact canary deployment assigned to the user.

Conclusion

This setup enables multiple simultaneous canary deployments, attributing user behavior to the deployment they’re assigned to. It also mitigates the risk of unintended side effects from changes by breaking down any event by any feature flag. Combining this with in-app feature flags, Unleash’s regionalized flag-targeting, and Mixpanel’s analytics capabilities, you now have comprehensive insights. You’re only 23 custom board funnels and 30 custom reports away from being a data-driven product manager.

Have fun!