Brice Luu

This role allowed me to fine tune my expertise and diversify my skill set. Revamped the data warehouse to CRM (Hubspot) synchronization: switching to using Hubspot's batch APIs in order to bypass the rate limiting that was preventing us from further decreasing of the execution duration Whole handedly implemented the data warehouse to ERP (Netsuite) synchronization in coordination with the integrator that was chosen by our finance team. This allowed the company to automate bill generation and notification when payments come in for all our freelancers Deployed a full fledged Neo4j cluster on k8s (via the fine tuned Helm chart) with an Elasticache Redis instance in front for improved performance Deployed our in house recommendation and NLP API (based on the above graph cluster) on our Kubernetes clusters Deployed a Snowplow distibution on our k8s clusters which allowed us to totally rethink and systemize our platform's behavioral tracking Helped define and implement our community growth hacking solution in order to increase our supply quality and quantity Drove the redesign of our data warehouse construction (DBT jobs scheduled from Airflow and running on Big Query) in order for it to be more easily maintainable, quicker to grasp and faster to build upon Initiated the refactoring of our Looker lookml modeling in order to make it more accessible to end users and to streamline the work needed to have newly computed features exposed to our analysts

Internal tools I developed on the side in my escalation and release management roles allowed me to transition to this role... I developed and maintained automatic monitoring solutions to supervise our applications' behaviour in prod and to validate the release process. I also built a D3.js visualisation of the hundreds of commits progressively piling up in each release across all our major applications. I later refactored it completely and added a search functionality showing in which release the corresponding commits went live.

Release management became part of the escalation team's responsibilities in early '14. This allowed us to grasp better control over the release flow and instill best practices in the way code was rolled out world wide... From the first day I drove the “release war”, I proposed a synthetic view of the latest code releases by application and datacenter. I then automated its building with some AngularJS. I later implemented a new way of regulating the launch of globally impacting A/B tests via a custom JIRA workflow enforcing configuration peer review and automatic blocking of any launch concurrent to issues on the platform with a bit of Groovy.

No one ever gave me a score card nor asked me to play this role, but as it turns out, it was really needed and allowed our team to be recognised for its expertise both in detecting and handling critical incidents and understanding longer term "inexplicable" observations... I used different tools and data sources (JIRA API, Git and Gerrit logs, Graphite metrics exports) to produce aggregated data. This lead me to create custom data structures for more efficient manipulation. I selected and filtered the noise out of the most relevant metrics to monitor each application’s behaviour during new code releases and system updates. Analysing issues reported by BI teams lead me to dig into our HDFS clusters using HiveQL with some custom Java User Defined Functions or often obtain the latest data by directly tailing the ongoing server logs before they were exported through Kafka. I also regularly had to find which clients were impacted by a particular issue from their configurations and connections with hand crafted SQL queries. When possible, I produced Grafana dashboards allowing to concisely monitor the general health of the platform.

This was my first tech role per-say, it allowed me to learn a lot of things: from basics like what's a web app or what insights on system stability can be obtained from user browsing behaviour, to the intricacies of how an R&D/Tech department of 100+ people operates or what happens when we "loose" a DC (and how can we bring it back up in a timely manner)... I quickly specialised in handling issues reported by BI and production teams. On the BI side, it regularly lead to deep ad hoc analysis to understand some subtle differences with the expected behaviour. On the production side, it was either urgent critical alerts (e.g. huge drop in clicks on a whole datacenter) or recurring production alerts that needed their root cause to be identified once and for all. Additionally to developing investigation skills, the role allowed me to work as a cross-team coordinator to resolve the largest incidents.