#BoSRE
Slides from my talk on technical and operational debt that I presented at #BoSRE are now available at https://drive.google.com/file/d/1H93EiP7hPZ-yWCegou23iFjHlfB2Ke9_/view?usp=sharing

#DevOps #SRE
Reducing Technical and Operational Debt.pdf
drive.google.com
January 13, 2025 at 3:36 PM
Uh oh, an all white male panel appears... I am going to walk out. Sorry, @WayfairTech (p.s. I was the only woman speaker, I think... except for one of the k8s panelists) #BoSRE
January 13, 2025 at 3:35 PM
overheard at #BoSRE: "there was the button to do my job, and the button to delete the whole world, right next to each other!"
January 13, 2025 at 3:35 PM
Use exercises such as postmortem reading club or wheel of misfortune to share knowledge and learn! [ed: I really really wish we hadn't used the word "victim" in the first SRE book :/].

Learn more at http://blameless.com/blog! [fin] #BoSRE
Blog | Blameless Resources
Insights from thought leaders on incident management best practices, tools for site reliabiliity engineers
blameless.com
January 13, 2025 at 3:35 PM
We want teams that trust each other and toil less.

@blamelesshq's 5 Weird Tricks to Improve Teams:
(1) Bootstrap with standard templates/checklists
(2) Automate data collection
(3) Aggregate and analyze patterns
(4) Track/prioritize action items
(5) Share/collaborate #BoSRE
January 13, 2025 at 3:35 PM
Psychological safety and blamelessness are critical to having well functioning SRE teams, but you also need accountability to ensure that reliability tasks get appropriately prioritized and done rather than passed like a hot potato. #BoSRE
January 13, 2025 at 3:35 PM
Lots of people use Google Docs or wikis, but it's ridden with toil to cut-paste metrics and events from production systems and ticketing systems into the document.

[ed: we solve this at Google with many, many, many Apps Scripts. Cloud IRM makes this better though.] #BoSRE
January 13, 2025 at 3:34 PM
Next speakers at #BoSRE are @asharrizqi and @lyonwong of @blamelesshq on achieving sanity and better reliability through applying postmortem culture. [ed: I may have some inclusion feedback for them afterwards]
January 13, 2025 at 3:34 PM
Final question: is Wayfair a tech company or retail?
A: Wayfair invests profits into making technological innovations in retail. e.g. being able to do last mile logistics better, help suppliers deliver more easily, etc. [fin] #BoSRE
January 13, 2025 at 3:34 PM
Audience question from your editor: how did you come to doing SRE at Wayfair?

A: because of the desire to (1) specialize roles, (2) have infra/ops be equal peers of product dev, and (3) improve efficiency; they already were pursuing these goals before DevOps & SRE labels. #BoSRE
January 13, 2025 at 3:34 PM
Make sure that teams are minimizing blast radius in order to give them autonomy and enabling them to move fast. #BoSRE
January 13, 2025 at 3:34 PM
Audience question about deprecations: "How did you wean your engineers off of MongoDB?" [with a chorus of people clamoring to want it answered.]

answer (paraphrased): you break it, you keep both pieces. people were begging to move to a supported platform quickly. #BoSRE
January 13, 2025 at 3:34 PM
The importance when deciding on projects is instead on ability to fail-fast if things aren't working. #BoSRE
January 13, 2025 at 3:34 PM
Notable: @WayfairTech's migration to the cloud was not motivated by desire to reduce capex; because retailers are paid before they ship goods, access to capital hasn't been a problem to them. #BoSRE
January 13, 2025 at 3:34 PM
the evolution of a company over 16+ years results in many technical changes, and adapting what was modern at the time. and eventually sunsetting the obsolete tech (@StevenConine is recounting the history of all the technologies used...) #BoSRE
January 13, 2025 at 3:34 PM
There always will be the need to rewrite things, and sometimes you'll wind up right back where you started (e.g. inline sql to stored procedures back to inline sql). #BoSRE
January 13, 2025 at 3:34 PM
Next at #BoSRE: @StevenConine on @WayfairTech's journey from $12 a month hosting and getting emailed payment details in plaintext when orders came in and inputting them into MS Access... to where they are today. #BoSRE
January 13, 2025 at 3:34 PM
and they solicited pain points from the audience to start building community around hybrid cloud in Boston! [fin] #BoSRE
January 13, 2025 at 3:34 PM
So how did they perform this changeover? doing the proof of concept work and starting to automate, then moving fully onto productionizing the automation and deploying to @GCPcloud. and then starting to experiment with k8s. #BoSRE
January 13, 2025 at 3:34 PM
Kudos to the team @WayfairTech for making sure that microphones are being used. Now if only we could get captioning... #BoSRE
January 13, 2025 at 3:34 PM
why performance engineering? because customer happiness, the customer visible latency, and progress through the customer funnel matters. #BoSRE
January 13, 2025 at 3:34 PM
Parts of the organization @WayfairTech created: Platform, Embedded SRE put into individual SWE teams, ARGO (onprem version of spinnaker), and traditional ops (focused on alerting/toil). plus performance engineering. #BoSRE
January 13, 2025 at 3:34 PM
it turns out that pushing the boundaries of on-prem automation is... a thing you hit pretty fast.

e.g. writing EMC, Cisco, etc. providers for Terraform... #BoSRE
January 13, 2025 at 3:34 PM
Wayfair stuck with their monitoring tooling that they'd already developed onprem across cloud and onprem, but use Stackdriver to a limited extent for autoscaling. They also use Datadog.

However, they didn't go fully cloud agnostic, because it limited benefits of cloud #BoSRE
January 13, 2025 at 3:34 PM