Aaaand @mipsytipsy is here to lead off #ServerlessDaysBOS! She'll be talking about observability and teams owning their software all the way into production!
January 13, 2025 at 5:39 PM
Aaaand @mipsytipsy is here to lead off #ServerlessDaysBOS! She'll be talking about observability and teams owning their software all the way into production!
p.s. come to this if you want to hear more after @mipsytipsy's #ServerlessDaysBOS talk on putting your engineers oncall!
It's not as simple as "just hand them the pager" and we'll talk through how to make it not suck/trigger tissue rejection.
https://t.co/adl6rRVSpY
It's not as simple as "just hand them the pager" and we'll talk through how to make it not suck/trigger tissue rejection.
https://t.co/adl6rRVSpY
January 13, 2025 at 5:46 PM
p.s. come to this if you want to hear more after @mipsytipsy's #ServerlessDaysBOS talk on putting your engineers oncall!
It's not as simple as "just hand them the pager" and we'll talk through how to make it not suck/trigger tissue rejection.
https://t.co/adl6rRVSpY
It's not as simple as "just hand them the pager" and we'll talk through how to make it not suck/trigger tissue rejection.
https://t.co/adl6rRVSpY
Datadog also offers tracing/metrics with a slightly different view if you want to do something other than the default xray. [fin] #ServerlessDaysBOS
January 13, 2025 at 5:45 PM
Datadog also offers tracing/metrics with a slightly different view if you want to do something other than the default xray. [fin] #ServerlessDaysBOS
You can also add annotations and metadata to the trace information. annotations are indexed. [ed: ah, but the challenge is... how well does the indexing and aggregation/sampling work?]
You can only have 50 annotations per trace. [ed: grrrrr, noooo.] #ServerlessDaysBOS
You can only have 50 annotations per trace. [ed: grrrrr, noooo.] #ServerlessDaysBOS
January 13, 2025 at 5:45 PM
You can also add annotations and metadata to the trace information. annotations are indexed. [ed: ah, but the challenge is... how well does the indexing and aggregation/sampling work?]
You can only have 50 annotations per trace. [ed: grrrrr, noooo.] #ServerlessDaysBOS
You can only have 50 annotations per trace. [ed: grrrrr, noooo.] #ServerlessDaysBOS
so now they have 1.2 seconds instead of 2.2 seconds of execution time, because we found the critical path [ed: me stealing @el_bhs's words here to describe the concept] and optimized it to be shorter. #ServerlessDaysBOS
January 13, 2025 at 5:45 PM
so now they have 1.2 seconds instead of 2.2 seconds of execution time, because we found the critical path [ed: me stealing @el_bhs's words here to describe the concept] and optimized it to be shorter. #ServerlessDaysBOS
But it doesn't automatically drill down into the third party API calls. you need to decorate with @xray_recorder.capture('description') on each inner function call.
And we realize that we have a series set of query steps that could be parallelized or cached. #ServerlessDaysBOS
And we realize that we have a series set of query steps that could be parallelized or cached. #ServerlessDaysBOS
January 13, 2025 at 5:45 PM
But it doesn't automatically drill down into the third party API calls. you need to decorate with @xray_recorder.capture('description') on each inner function call.
And we realize that we have a series set of query steps that could be parallelized or cached. #ServerlessDaysBOS
And we realize that we have a series set of query steps that could be parallelized or cached. #ServerlessDaysBOS
Traces answer "where and how long?" in request paths.
You can get basic X-ray tracing under "debugging and error handling" for your lambda functions using "enable active tracing".
You may turn up basic flubs like "failed to set response code" or... #ServerlessDaysBOS
You can get basic X-ray tracing under "debugging and error handling" for your lambda functions using "enable active tracing".
You may turn up basic flubs like "failed to set response code" or... #ServerlessDaysBOS
January 13, 2025 at 5:45 PM
Traces answer "where and how long?" in request paths.
You can get basic X-ray tracing under "debugging and error handling" for your lambda functions using "enable active tracing".
You may turn up basic flubs like "failed to set response code" or... #ServerlessDaysBOS
You can get basic X-ray tracing under "debugging and error handling" for your lambda functions using "enable active tracing".
You may turn up basic flubs like "failed to set response code" or... #ServerlessDaysBOS
Each lambda execution only by default shows start, end, and report but nothing else in the logs. So we can just write a line to the console to get cloudwatch logs. It's still annoying to find all the data we need in one place even if we add log lines #ServerlessDaysBOS
January 13, 2025 at 5:45 PM
Each lambda execution only by default shows start, end, and report but nothing else in the logs. So we can just write a line to the console to get cloudwatch logs. It's still annoying to find all the data we need in one place even if we add log lines #ServerlessDaysBOS
So we can get performance timings by function in aggregate, but it doesn't help us identify individual slowness.
The logs are hopefully structured and greppable as well as parseable. [ed: have a bone of contention with idea they help with unknown unknowns.] #ServerlessDaysBOS
The logs are hopefully structured and greppable as well as parseable. [ed: have a bone of contention with idea they help with unknown unknowns.] #ServerlessDaysBOS
January 13, 2025 at 5:45 PM
So we can get performance timings by function in aggregate, but it doesn't help us identify individual slowness.
The logs are hopefully structured and greppable as well as parseable. [ed: have a bone of contention with idea they help with unknown unknowns.] #ServerlessDaysBOS
The logs are hopefully structured and greppable as well as parseable. [ed: have a bone of contention with idea they help with unknown unknowns.] #ServerlessDaysBOS
Metrics can catch the known unknowns. Lambda's builtin CloudWatch has some basic stuff on function health, but we can get a UI for free by putting a metric containing our payload data! Shiny. #ServerlessDaysBOS
January 13, 2025 at 5:45 PM
Metrics can catch the known unknowns. Lambda's builtin CloudWatch has some basic stuff on function health, but we can get a UI for free by putting a metric containing our payload data! Shiny. #ServerlessDaysBOS
And it mostly works, but it's kind of slow and unresponsive.
So, how to debug? Esp when no sirens flashing obvious problem to look at? No http 500s, etc.
"Three pillars of observability" etc. - metrics, logs, traces. [ed: wishing we had the why vs how] #ServerlessDaysBOS
So, how to debug? Esp when no sirens flashing obvious problem to look at? No http 500s, etc.
"Three pillars of observability" etc. - metrics, logs, traces. [ed: wishing we had the why vs how] #ServerlessDaysBOS
January 13, 2025 at 5:45 PM
And it mostly works, but it's kind of slow and unresponsive.
So, how to debug? Esp when no sirens flashing obvious problem to look at? No http 500s, etc.
"Three pillars of observability" etc. - metrics, logs, traces. [ed: wishing we had the why vs how] #ServerlessDaysBOS
So, how to debug? Esp when no sirens flashing obvious problem to look at? No http 500s, etc.
"Three pillars of observability" etc. - metrics, logs, traces. [ed: wishing we had the why vs how] #ServerlessDaysBOS
So you have to provide token and the various document/table/column IDs corresponding to the human-readable names, and POST the data and IDs to Coda's API.
Each ID fetch is a separate API call. #ServerlessDaysBOS
Each ID fetch is a separate API call. #ServerlessDaysBOS
January 13, 2025 at 5:44 PM
So you have to provide token and the various document/table/column IDs corresponding to the human-readable names, and POST the data and IDs to Coda's API.
Each ID fetch is a separate API call. #ServerlessDaysBOS
Each ID fetch is a separate API call. #ServerlessDaysBOS
Next up is @Technovangelist on tracing serverless. Interestingly, if the abstract is to be believed, he's going to be talking about Amazon X-Ray rather than about the company he works at! Kudos for that. #ServerlessDaysBOS
January 13, 2025 at 5:44 PM
Next up is @Technovangelist on tracing serverless. Interestingly, if the abstract is to be believed, he's going to be talking about Amazon X-Ray rather than about the company he works at! Kudos for that. #ServerlessDaysBOS
I continue boycotting Cloudflare talks given their leadership has vehemently defended proxying/shielding the site stalking and harassing me. :( #ServerlessDaysBOS
January 13, 2025 at 5:44 PM
I continue boycotting Cloudflare talks given their leadership has vehemently defended proxying/shielding the site stalking and harassing me. :( #ServerlessDaysBOS
Going to be around the Honeycomb booth for the next 40 minutes! #ServerlessDaysBOS
and remember, if you have questions for @mipsytipsy or me, drop by the @honeycombio booth to leave a question in the bowl, or fill out https://goo.gl/forms/rhcvEQUQUPRLNGzO2! #ServerlessDaysBOS
Honeycomb AMA: submit a question
Please fill out this form to submit a question on observability, serverless, culture, architecture, etc. for Charity or Liz to answer. The best and/or most popular questions will be answered during the event!
By submitting your email address, you also agree to receive follow-up emails from Honeycomb marketing. Thanks for playing!
goo.gl
January 13, 2025 at 5:44 PM
Going to be around the Honeycomb booth for the next 40 minutes! #ServerlessDaysBOS
Next up at #ServerlessDaysBOS is my former coworker @BretMcG but sadly I have to duck out for a booth shift :(
January 13, 2025 at 5:44 PM
Next up at #ServerlessDaysBOS is my former coworker @BretMcG but sadly I have to duck out for a booth shift :(
Make sure you know about customer success metrics, workload on customer support team, user personas, buyer personas, etc. -- make sure your work has meaning.
Focus least on tech. Focus on what matters: making people happy. [fin] #ServerlessDaysBOS
Focus least on tech. Focus on what matters: making people happy. [fin] #ServerlessDaysBOS
January 13, 2025 at 5:44 PM
Make sure you know about customer success metrics, workload on customer support team, user personas, buyer personas, etc. -- make sure your work has meaning.
Focus least on tech. Focus on what matters: making people happy. [fin] #ServerlessDaysBOS
Focus least on tech. Focus on what matters: making people happy. [fin] #ServerlessDaysBOS
Serverless is just as good of a mechanism for delivering garbage faster and more efficiently to users as it is for delivering value. Pay attention to what you're building. [ed: and also on your impacts on the world! ethics matter!] #ServerlessDaysBOS
January 13, 2025 at 5:44 PM
Serverless is just as good of a mechanism for delivering garbage faster and more efficiently to users as it is for delivering value. Pay attention to what you're building. [ed: and also on your impacts on the world! ethics matter!] #ServerlessDaysBOS
Making it faster and more reliable has diminishing returns. It doesn't matter if your architecture is designed to support 1M users if it isn't actually capable of attracting/retaining 1M users. #ServerlessDaysBOS
January 13, 2025 at 5:44 PM
Making it faster and more reliable has diminishing returns. It doesn't matter if your architecture is designed to support 1M users if it isn't actually capable of attracting/retaining 1M users. #ServerlessDaysBOS
and do you know how application performance corresponds to user happiness? Do you know what good enough means for your application? [ed: e.g. having SLOs that are truly customer-centric]
Are you chasing service objectives that aren't necessary? #ServerlessDaysBOS
Are you chasing service objectives that aren't necessary? #ServerlessDaysBOS
January 13, 2025 at 5:44 PM
and do you know how application performance corresponds to user happiness? Do you know what good enough means for your application? [ed: e.g. having SLOs that are truly customer-centric]
Are you chasing service objectives that aren't necessary? #ServerlessDaysBOS
Are you chasing service objectives that aren't necessary? #ServerlessDaysBOS
Talk about growth metrics with your product teams. Do you know what scale you're designing for, or are you winging it? More scale is not always better. #ServerlessDaysBOS
January 13, 2025 at 5:44 PM
Talk about growth metrics with your product teams. Do you know what scale you're designing for, or are you winging it? More scale is not always better. #ServerlessDaysBOS
[ed: @GremlinInc is making it easier to do without being an expert now! they have a community edition product now!]
Shift left to product; add value by reducing toil and spend newfound time on what we're building/why instead of adding shiny tech. #ServerlessDaysBOS
Shift left to product; add value by reducing toil and spend newfound time on what we're building/why instead of adding shiny tech. #ServerlessDaysBOS
January 13, 2025 at 5:44 PM
[ed: @GremlinInc is making it easier to do without being an expert now! they have a community edition product now!]
Shift left to product; add value by reducing toil and spend newfound time on what we're building/why instead of adding shiny tech. #ServerlessDaysBOS
Shift left to product; add value by reducing toil and spend newfound time on what we're building/why instead of adding shiny tech. #ServerlessDaysBOS
Bugs do not necessarily manifest in one place; data generation vs. validation vs. usage can be scattered across service boundaries.
Run game days. Be prepared at 3am. [ed: called out, was only one who raised hand].
Who actually does chaos engineering? #ServerlessDaysBOS
Run game days. Be prepared at 3am. [ed: called out, was only one who raised hand].
Who actually does chaos engineering? #ServerlessDaysBOS
January 13, 2025 at 5:44 PM
Bugs do not necessarily manifest in one place; data generation vs. validation vs. usage can be scattered across service boundaries.
Run game days. Be prepared at 3am. [ed: called out, was only one who raised hand].
Who actually does chaos engineering? #ServerlessDaysBOS
Run game days. Be prepared at 3am. [ed: called out, was only one who raised hand].
Who actually does chaos engineering? #ServerlessDaysBOS
You have to be able to code. [ed: and product engineers need to be able to operate too, kinda sick and tired of seeing exhortations for ops to code and not dev to ops]
We need to write load generators, retry frameworks, etc. as operators. #ServerlessDaysBOS
We need to write load generators, retry frameworks, etc. as operators. #ServerlessDaysBOS
January 13, 2025 at 5:44 PM
You have to be able to code. [ed: and product engineers need to be able to operate too, kinda sick and tired of seeing exhortations for ops to code and not dev to ops]
We need to write load generators, retry frameworks, etc. as operators. #ServerlessDaysBOS
We need to write load generators, retry frameworks, etc. as operators. #ServerlessDaysBOS
Making purchasing decisions often winds up being an ops decision.
Operability winds up being useful to design into systems from the start e.g. adding dead letter queues, alerting, etc.
But we also need to code. #ServerlessDaysBOS
Operability winds up being useful to design into systems from the start e.g. adding dead letter queues, alerting, etc.
But we also need to code. #ServerlessDaysBOS
January 13, 2025 at 5:44 PM
Making purchasing decisions often winds up being an ops decision.
Operability winds up being useful to design into systems from the start e.g. adding dead letter queues, alerting, etc.
But we also need to code. #ServerlessDaysBOS
Operability winds up being useful to design into systems from the start e.g. adding dead letter queues, alerting, etc.
But we also need to code. #ServerlessDaysBOS