Transcript delivered to you by IEEE Software program journal. This transcript was mechanically generated. To recommend enhancements within the textual content, please contact content material@pc.org and embrace the episode quantity and URL.

Robert Blumen 00:00:16 For Software program Engineering Radio. That is Robert Bluman. Right now I’ve with me Alex Boten. Alex is a senior employees software program engineer at LightStep. Previous to that, he was at Cisco. He’s contributed to open-source tasks within the telemetry space, together with the OpenTelemetry undertaking. He’s the creator of the ebook, Cloud Native Observability with OpenTelemetry, and that would be the topic of our dialog right now. Alex, welcome to Software program Engineering Radio.

Alex Boten 00:00:50 Good day. Thanks for having me. It’s nice to be right here.

Robert Blumen 00:00:52 Would you want so as to add something about your background that I didn’t point out?

Alex Boten 00:00:57 I feel you captured most of it. I’ve been contributing to OpenTelemetry for somewhat bit over three years. I’ve labored on numerous parts of the undertaking in addition to the specification, and I’m presently a maintainer on the OpenTelemetry Collector.

Robert Blumen 00:01:11 Nice. Now on Software program Engineering Radio, we now have lined numerous telemetry-related points, together with Logging in episode 220, Excessive Cardinality Monitoring, which was 429, Prometheus Distributed Tracing and episode 455, which was referred to as Software program Telemetry. So, listeners can undoubtedly hearken to a few of these in our again catalog to get extra normal info. We’ll be focusing extra on this dialog about what OpenTelemetry brings to the desk that we now have not already lined. Let’s begin out with, within the telemetry area, the place may you situate OpenTelemetry? What’s it just like? What’s it completely different? What downside does it resolve?

Alex Boten 00:02:02 That’s an ideal query. So, I feel the issue that OpenTelemetry goals to resolve — and we’ve already seen it occur within the business right now — is it adjustments how utility builders instrument their utility, how telemetry is generated, and the way it’s collected, after which transmitted throughout programs. And if I had been to consider what it’s just like the very first thing that involves thoughts are the tasks that actually brought on it to emerge, that are OpenCensus and OpenTracing, that are two different open-source tasks that had been shaped somewhat bit earlier. I feel it began in possibly 2017, 2016, to offer a normal round producing distributed tracing. After which additionally OpenCensus additionally addressed somewhat bit round metrics and log assortment.

Robert Blumen 00:02:50 What was occurring within the telemetry space previous to these tasks that created the necessity for them, and what did they do?

Alex Boten 00:02:57 Yeah, so I feel, in case you consider telemetry because the area in software program, it’s been round for a very very long time, proper? Like, folks as early because the earliest of pc scientists wished to know what their computer systems had been doing. And earlier within the days of getting a single machine, it was pretty simple to print some log statements and take a look at what your machine was doing. However because the business grew, because the Web of Issues picked up, as programs grew to become bigger and bigger to handle the rising demand, I feel programs grew to become inherently extra advanced. And we’ve seen an evolution of what software program telemetry actually grew to become. So, in case you consider earlier we had been in a position to log knowledge on a single system. As folks needed to deploy a number of programs, a necessity for centralized logging got here alongside as a way to mixture and do mixture searches on logs.

Alex Boten 00:03:54 And that grew to become actually expensive. After which we noticed a rise in people eager to seize extra significant metrics from their programs the place they may create dashboards and do queries, whereas it was cheaper than going by and analyzing log knowledge. And I feel the factor that I’ve seen occur within the final 20 years is each time there was a brand new possibly paradigm round the kind of telemetry that programs ought to emit, there was an opportunity for innovation to happen, which is nice to see, however in case you’re an finish consumer who’s simply making an attempt to get telemetry out of a system, out of an utility, it’s a very irritating course of to should go and reinstrument your code each few months or each few years, relying on what the flavour of the day is. And I feel what OpenCensus and OpenTracing and OpenTelemetry tried to seize is addressing the ache that customers have in the case of instrumenting their code.

Robert Blumen 00:04:49 What’s the relationship of OpenTelemetry to different programs on the market, equivalent to Zipkin, Jaeger, Graylog, Prometheus?

Alex Boten 00:05:00 So the connection that OpenTelemetry has with the Zipkin, the Jaegers and the Prometheus of the world is admittedly round offering interoperability between these programs. So, an utility developer would instrument their code utilizing OpenTelemetry, after which they’ll emit that telemetry knowledge to no matter backend programs they need. So, in case you wished to proceed utilizing Jaeger, you might undoubtedly try this with an utility that’s instrumented with OpenTelemetry. The opposite factor that OpenTelemetry tries to do is it tries to offer a translation layer so that people which might be possibly right now emitting knowledge to Zipkin or to Jaeger or to Prometheus can deploy a collector inside their environments after which translate the information from a selected format of these different programs into the OpenTelemetry format, in order that they’ll then emit the information to no matter backend they select by merely updating the configuration on their Collector with out having to return to their functions who could also be legacy programs that no one needs to change anymore and nonetheless be capable of ship their knowledge to completely different locations.

Robert Blumen 00:06:06 Is OpenTelemetry then an interoperability commonplace, a system, or each?

Alex Boten 00:06:13 It’s actually the usual to instrument your functions and to offer the interoperability between the completely different programs. OpenTelemetry doesn’t provide a backend; there’s no log database or metrics database that OpenTelemetry offers. Possibly sooner or later sooner or later that that may occur. We’re actually seeing folks which might be supporting the OpenTelemetry format beginning to present these backend choices for folk which might be emitting solely OpenTelemetry knowledge. However that’s not one thing the undertaking is excited about fixing at this level. It’s actually in regards to the instrumentation piece and the gathering and transmission of the information.

Robert Blumen 00:06:52 In studying about this, I got here throughout dialogue of a protocol referred to as OTLP. Are you able to clarify what that’s?

Alex Boten 00:07:00 So the OpenTelemetry protocol is a protocol that’s generated from protobuf definitions. Each implementation of OpenTelemetry helps its purpose is to offer excessive efficiency knowledge transmission in a format that’s standardized throughout all of the implementations. It’s additionally supported by the OpenTelemetry Collector. And what it actually means is, so this format helps all of the completely different alerts that OpenTelemetry helps. So, log traces, metrics, and possibly down the street, occasions and profiling, which is presently being developed within the undertaking. And the thought is in case you assist the OpenTelemetry protocol, that is the protocol that you’d use to both transmit the information, or in case you’re a vendor or in case you’re a backend supplier, you’ll use that protocol to obtain the information. And it’s truly been actually good to see even tasks like Prometheus beginning to assist the OTLP protocol for transmitting knowledge.

Robert Blumen 00:07:56 So, let me summarize what we now have thus far, and you may inform me if I’ve understood. I’m constructing an utility, I may instrument it in a method that’s suitable with this commonplace. I may not even know the place my logs or metrics are going to finish up. After which whoever makes use of my system, which can be folks in the identical group or possibly I’m transport an open-source undertaking, which has many customers — they’ll then plug of their backend of selection, and they aren’t essentially tied to any choices I made about how I feel the telemetry can be collected. It creates the power of customers to plug and play between the functions and the backends. Is that roughly appropriate?

Alex Boten 00:08:42 Yeah, that’s precisely proper. I feel it actually decouples the instrumentation piece, which traditionally has been the most costly facet of organizations gaining observability inside of their programs, from the choice of the place am I going to ship that knowledge. And the great factor about that is that it actually frees the tip customers from the thought of vendor lock-in, which I feel lots of us who’ve labored in in programs for a very long time all the time discovered it to be tough. The dialog of making an attempt to possibly check out a brand new vendor in case you wished to check some new function that you simply wished to have or no matter, normally would imply that you would need to return and re-instrument your code. Whereas now with OpenTelemetry, when you’ve got instrumented your utility, hopefully that is the final time it’s a must to fear about instrumenting your utility as a result of you possibly can simply level that knowledge to completely different backends.

Robert Blumen 00:09:34 A short time in the past you probably did point out the Collector, and we can be spending a while on that, however I need to perceive what are the attainable configurations of the system. What I feel we’re speaking about now’s if the code is instrumented with the OpenTelemetry commonplace, that it may discuss on to backends. The opposite possibility being you may have a Collector in between them. Are these the 2 important configurations?

Alex Boten 00:10:02 Yeah, that’s proper. It’s additionally attainable to configure your instrumented utility to ship knowledge to backends immediately: in case you wished to decide on to ship the information to Jaeger, I feel most implementations that assist OpenTelemetry formally have a Jaeger exporter, for instance. So there are alternatives in case you wished to ship knowledge out of your utility to your backend, however ideally you’ll ship that knowledge in a protocol which you could then configure utilizing an OpenTelemetry Collector later down the road.

Robert Blumen 00:10:31 Let’s come again to Collector in a bit, however I need to speak about instrumentation. Typically if I need to discuss to a sure backend, I would like to make use of their library to emit the telemetry. How does that change with OpenTelemetry?

Alex Boten 00:10:49 Yeah, so with the OpenTelemetry commonplace, you may have two features of the instrumentation. So, there’s the OpenTelemetry API, which is admittedly what most builders would work together with. There’s a really restricted quantity of floor space that the API covers. For instance, for tracing the APIs, basically you can begin a span and you may end a span and get a tracer. That’s roughly the floor space that’s making an attempt to be lined there. And the thought we wished to push ahead with, with our restricted API, is to only scale back the cognitive load that customers must tackle to undertake OpenTelemetry. The opposite piece of the instrumentation that people should work together with is the SDK, which actually permits finish customers to configure how the telemetry is produced and the place it’s despatched to. When you’re excited about this within the context of how is it completely different from explicit backend and its instrumentation, the, the distinction is what OpenTelemetry you’ll solely ever use the OpenTelemetry API and configure the SDK to ship knowledge to the backend of selection.

Alex Boten 00:11:55 However the API that you’d use for instrumenting the code wouldn’t be any completely different relying on which backend you ship it to. And there’s that clear separation between the API and the SDK that means that you can actually solely instrument with that minimal interface and fear in regards to the particulars of how and the place that knowledge is distributed utilizing the SDK configuration, which in my ebook I check with as telemetry pipelines.

Robert Blumen 00:12:17 In that dialogue you talked about tracing, I’ve seen lots of logging programs, you possibly can log no matter you need after which it places the burden on a Collector to choose up the logs and format them. After which metrics, you will have to make use of a library. If I’m adopting OpenTelemetry, how does it deal with logs and metrics?

Alex Boten 00:12:40 Yeah, so for metrics, there’s an API that calls out particular devices. So OpenTelemetry has an inventory of, I consider it’s six devices presently that it helps to roughly have the identical performance as just like the library. And I feel lots of these devices had been developed in collaboration with each the open metrics and the Prometheus communities to make sure that we’re suitable with these people. So, for the logging library, that’s somewhat bit completely different in OpenTelemetry — or a minimum of it was on the time of writing my ebook, which was written in 2021, largely. The thought behind logging and OpenTelemetry was, we already had been conscious there have been so many various APIs for logging in every language. Every language has like a dozen logging APIs and we didn’t essentially need to create a brand new logging API that folks must undertake. And so, the thought was to essentially hook into these current APIs. It’s been an fascinating transition although. I feel prior to now, possibly prior to now six or eight months or so, there’s been nearly an ask for an API and an SDK within the logging sign as properly. That’s nonetheless presently in improvement. So, keep tuned for what’s going to occur there.

Robert Blumen 00:13:51 In what languages are the OpenTelemetry SDKs accessible?

Alex Boten 00:13:57 Yeah, so there’s presently 11 formally supported languages. I’m most likely going to overlook a few of them, however there’s undoubtedly one in C++, in Go, in Rust, in Python, Ruby, PHP, Java, JavaScript, all these languages are lined formally by OpenTelemetry. And what this implies is that the implementations had been reviewed by somebody on the technical committee, and the implementations themselves reside inside the OpenTelemetry group in GitHub and has the identical course of. We now have maintainers and approvers for every a kind of languages. There’s a few further implementations that aren’t formally supported but, however that’s actually simply because there hasn’t been sufficient contributors to it but. So, I feel there’s one in Lua and possibly Julia is the opposite one?

Robert Blumen 00:14:46 I’ve discovered when instrumenting code up and spend lots of time doing issues like writing a message {that a} sure technique has been referred to as, and listed below are the parameters — very boilerplate steps. I perceive that OpenTelemetry can to some extent automate that? How does that work?

Alex Boten 00:15:08 Yeah, so there’s — one of many very first OTEPs (the OpenTelemetry Enhancement Proposals) that was created within the early levels of the undertaking was to assist to assist auto instrumentation out of the field. So, the trouble of auto instrumentation in numerous languages is at completely different levels. So, I do know the Java and the Python auto instrumentation efforts are somewhat bit additional alongside. I feel .NET is coming alongside properly, and I feel JavaScript is, as properly. However the concept behind auto instrumentation with OpenTelemetry particularly is similar to what we’ve seen in different efforts earlier than the place it actually ties instrumentation to current third get together open-source library or third get together libraries. Proper? And the thought being, for instance, in case you’re utilizing the Python SDK — I’m utilizing that for instance as a result of I spent an honest period of time writing some code there.

Alex Boten 00:16:02 When you’re utilizing the Python SDK and also you wished to make use of, for instance, the Python Redis library, properly you might use the instrumentation library that’s supplied by OpenTelemetry, which lets you name to this library, which monkey patches the Redis library that it then makes a name to. However, in that intermediate step, it acts as a center layer that devices the calls to the library that you’d be making. So, in case you had been calling Konnect, for instance, it might name Konnect on the instrumentation library begin span, possibly document some type of metric in regards to the operation, make the decision to the Redis library, after which on the return it might finish the span and produce some telemetry there with some semantic conference attributes.

Robert Blumen 00:16:49 Clarify the time period monkey patching.

Alex Boten 00:16:52 So monkey patching is when a library intercepts a name and replaces a name with itself as a substitute of the unique name. So, within the case of the Redis instance I used to be utilizing, the Redis instrumentation library intercepts the decision to connect with Redis, after which it replaces it with its personal join name, which does the instrumentation, as properly.

Robert Blumen 00:17:17 This I may see being very helpful in that in case you’ve acquired a library and one thing’s going mistaken within the library, I don’t know the place, then the earlier possibility has been that I have to get the supply code of the library, and if I would like logging, I must go and insert log statements or insert metrics or no matter kind of telemetry I’m making an attempt to seize into another person’s supply code and rebuild it. So, does this allow you to get visibility of what’s occurring inside third-party libraries that you simply’ve downloaded together with your package deal supervisor and also you’re not excited about modifying the code?

Alex Boten 00:17:57 Proper. I feel that’s a key good thing about it’s that you simply’re lastly in a position to see what these libraries are doing, or possibly you’re not accustomed to the code otherwise you’re probably not certain of the trail by the code and also you’re in a position to see the entire library calls which might be instrumented on beneath the unique name of your utility, which lots of the time you’ll discover issues there, but it surely’s actually arduous to determine them since you don’t essentially know what’s occurring with out studying the supply code beneath in any respect.

Robert Blumen 00:18:24 I’ve used a few of these languages within the 11. I’m conscious that each language is completely different so far as what entry it provides you to intercept issues at runtime or possibly generate byte code and inject it into the library. I’d assume that the power to do that goes to vary significantly primarily based on the language, and possibly C++ being slightly unfriendly to that. Do you anticipate to realize a parity with all of the languages within the extent which you could provide this function? Or will it all the time work higher on some than others?

Alex Boten 00:19:02 That’s an ideal query. I feel, ideally, I think about that instrumentation libraries are a brief repair. I actually consider that what everyone’s hoping for inside the group, and we’ve seen some Open Supply tasks already attain out and begin instrumenting their functions. We’re actually hoping that these libraries will in use the OpenTelemetry API to instrument themselves and take away the necessity for these instrumentation libraries altogether. For instance, if an HTTP server framework had been to instrument its calls to its endpoints utilizing OpenTelemetry, the tip consumer wouldn’t even want this instrumentation library. And we may obtain parity throughout all of the languages as a result of every a kind of libraries would simply use the usual slightly than counting on both byte code manipulation or monkey patching, which it really works for what it’s, but it surely’s not all the time the best possibility.

Alex Boten 00:20:01 With monkey patching, possibly the underlying libraries name adjustments parameters, and it’s a must to preserve observe of these adjustments inside these instrumentation libraries. And in order that, that all the time poses a problem. However ideally, like I mentioned, these libraries would, will go away because the undertaking continues to achieve traction throughout the business. And we’ve already seen, I feel there was a number of Python open-source tasks that reached out. I do know the Spring people in Java had a undertaking to instrument utilizing OpenTelemetry. Envoy and some different proxies have additionally began utilizing OpenTelemetry. So it’s undoubtedly, I feel in some magician lab we’re nice for the quick time period, however in the long run it might be perfect if issues had been instrumented themselves.

Robert Blumen 00:20:45 That might be nice. However there are all the time going to be some older libraries that possibly not underneath as energetic improvement the place there’s probably not anybody round to change them. Then you definately all the time have this to fall again on in these circumstances. I wouldn’t see it’s going away.

Alex Boten 00:21:02 Proper. Ideally it might, the norm would grow to be instrument your libraries with OpenTelemetry, and for these libraries that aren’t being modified and completely proceed to make use of the mechanisms that we now have in place right now.

Robert Blumen 00:21:16 Now I feel it’s the time to begin speaking in regards to the Collector. We’ve talked in regards to the supply and the way this knowledge will get printed. A short time in the past we talked about you possibly can ship immediately knowledge from a writer to a backend or you possibly can have a Collector in between. What’s the Collector, what does it do, why may I would like one?

Alex Boten 00:21:36 Yeah, so the Collector is a separate course of that may be operating inside your setting. It’s a binary that’s printed as a separate binary, or docker picture in case you’re excited about that. There’s additionally packages for, I feel, Debian and RedHat. And the Collector can be a vacation spot on your telemetry that may then act as a router. So, it has a number of, I consider it’s over 100 receivers, which assist completely different codecs and in addition can scrape metric knowledge from completely different programs. And it has exporters, and once more, I lose observe of it, however I feel it’s over 100 codecs of exporters that the OpenTelemetry Collector helps. So you possibly can ship knowledge to it in a single format and export it utilizing a unique format in case you’re so eager on. You may as well use processors inside the Collector, which let you manipulate the information, whether or not it’s for issues like redacting, possibly PII that you simply may need, or in case you wished to counterpoint the information with some further attributes — possibly about your environments that solely the Collector would find out about.

Alex Boten 00:22:44 And that’s the Collector in a nutshell. It’s accessible to deploy, as I mentioned, as a picture or as a package deal. There’s additionally, you possibly can deploy utilizing Helm charts. You’ll be able to deploy utilizing the OpenTelemetry operator in case you’re utilizing a Kubernetes setting.

Robert Blumen 00:22:59 I’m going to delve into a few of these inner parts. I need to discuss first somewhat bit in regards to the networking. It may be less complicated if I’ve N sources and variety of Ok backends, as a substitute of an N cross Ok topology, an N cross 1 and 1 cross Ok. Do you may have any ideas on, is {that a} motivator to simplify your networking and every little thing that goes together with that? Is {that a} motivator for adopting a Collector?

Alex Boten 00:23:30 Yeah, I feel so. I feel the Collector makes it very interesting for a wide range of causes. One being that your egress out of your community might solely be coming from one level. So, from a safety auditing type of perspective, you possibly can see the place all the information is admittedly going out slightly than having a bunch of various endpoints that should be linked to some exterior programs. I feel from that time alone, it’s undoubtedly price deploying a Collector inside a community. I feel there’s additionally the power to throttle the information that’s going out is vital. If in case you have N endpoints which might be sending knowledge, it’s actually tough to throttle how a lot knowledge is definitely leaving your community, which may find yourself being expensive. So, in case you wished to do issues like sampling, you’ll most likely need to have a Collector in place, in order that you might actually regulate it as wanted.

Robert Blumen 00:24:22 How a lot telemetry can one occasion of Collector deal with?

Alex Boten 00:24:30 Yeah, I imply I feel that all the time will depend on the dimensions of the occasion that you simply’re operating. They’re on the OpenTelemetry Collector repository. There’s a fairly complete benchmarks which have been run towards the Collector for each traces and logs and metrics. And I consider the occasion sizes that had been used, if reminiscence serves proper, they had been utilizing ECE2 for the testing for the benchmarks. And I consider that’s all listed on the web site there. For people which might be excited about discovering out.

Robert Blumen 00:25:01 If I wished to both run extra workload than what I may put by one occasion or for high-availability causes, have a clustered implementation with a a number of Collectors, is it attainable say to place a load balancer in entrance of it and distribute it? Or what are the choices for a extra clustered implementation?

Alex Boten 00:25:24 Yeah, so the way in which you’ll need to most likely deploy that is: you’ll need to use some type of load balancer relying on the, the telemetry you’re sending out, you might need to use like a routing processor that means that you can be extra particular as to which knowledge every one of many Collectors can be receiving. So for instance, in case you had, possibly a bunch of Collectors which might be deployed which might be nearer to your functions, that may then be routed by possibly a Collector as a gateway and also you wished to ship solely a sure variety of traces to the Collector as a gateway, you might fork it utilizing the routing processor primarily based on the hint IDs or one thing like that, in case you wished to.

Robert Blumen 00:26:06 So, with stateless servers you possibly can arrange a reasonably dumb load balancer and each request would get routed basically to a random occasion. Is there any causes I’ve a bit extra of a sharding or pinning of sure workloads in a clustered implementation?

Alex Boten 00:26:27 I feel a few of this will depend on what you’re doing with the Collectors. So for instance, in case you’re doing sampling on traces, you wouldn’t need your sampling choice being made throughout, like there’s, there’s no solution to share that sampling choice throughout Collectors. And so, you’ll need to have the ability to make that call on the identical occasion of the Collector, for instance. And so you’ll really need the entire knowledge for a selected hint to go to the identical Collector to have the ability to make the choice on the pattern.

Robert Blumen 00:26:56 You employ the phrase gateway, which is a typical phrase, however I perceive it means one thing particular in OpenTelemetry the place you may have a gateway mannequin and an agent mannequin. Clarify these two fashions, the distinction between them.

Alex Boten 00:27:11 Yeah, so within the agent deployment for the OpenTelemetry Collector, you’ll be operating your OpenTelemetry Collector on the identical host or the identical node, possibly as a part of a demon set in Kubernetes. So, you’ll have a separate occasion of the Collector for every one of many nodes which might be operating inside your setting. And you’ll have your utility sending knowledge to the native agent earlier than it might then ship it as much as wherever your vacation spot is. Within the gateway deployment mannequin, you’ll have the Collector act as a standalone utility, and it might have its personal deployment. Possibly you’ll have one per knowledge heart or possibly one per area. And that may act as possibly the egress out of your community. And that’s type of the gateway deployment.

Robert Blumen 00:28:02 What you described as an agent mannequin that sounds similar to me of what I’ve seen referred to as sidecar with another providers. Is agent the identical as a sidecar?

Alex Boten 00:28:14 Sure and no. It may be like a sidecar, I feel once I consider a sidecar as, I’d assume that it might be hooked up to each utility that’s operating with a sidecar alongside it, which might imply that you simply may find yourself with a number of situations of the Collector operating on the identical node, for instance, which can be needed in particular circumstances, or it might not be, it actually will depend on your use case, whether or not or not there’s accessibility out of your utility to the host in any respect. That will depend on what your insurance policies are, how your insurance policies are confined or outlined. So, it could possibly be the identical because the sidecar, but it surely doesn’t essentially should be.

Robert Blumen 00:28:52 Delving extra into the internals of the Collector and what you are able to do, you talked about processors and exporters — and also you’ve lined a few of this earlier than, however why don’t you begin with what are a few of the main sorts of processors that you simply may need to use?

Alex Boten 00:29:11 Yeah, so I feel that the 2 advisable processors by the group are the, the batch processor, which tries to take your knowledge and batch it slightly than sending it each time there’s telemetry coming in. That is making an attempt to optimize a few of the compression and scale back the quantity of information that will get despatched out. In order that’s one of many advisable processor. The opposite one is the reminiscence restrict processor, which limits type of the higher certain of reminiscence that you’d permit a Collector to make use of. So you’ll most likely need to use that within the case of you may have a selected occasion of some type with some type of reminiscence outlined, you’ll need to configure your reminiscence restrict processor to be under that threshold in order that when the Collector hits that reminiscence restrict, it could begin returning error messages to all of its receivers in order that possibly the senders of the information can go forward and again off on the quantity of information that’s being despatched or one thing like that.

Alex Boten 00:30:02 One of many different processors that’s actually fascinating to many people is the rework processor, which let you use the OpenTelemetry Transformation Language to change knowledge. So, possibly you need to strip some explicit attributes, or possibly you need to change some values inside your telemetry knowledge and you are able to do that with the rework processor, which continues to be presently underneath improvement. However I feel it early days within the processor there was lots of pleasure round what could possibly be achieved with processors. And so, folks began creating filtering processors and attribute processor for metrics and all these different type of processors that made it somewhat bit difficult to know which processors people must be utilizing as a result of there’s so lots of them. And generally, one might assist one sign however not the opposite, whereas the rework processor actually tries to possibly unify this and to a single processor like that can be utilized to do all of that.

Robert Blumen 00:30:55 You mentioned there’s lots of pleasure round this function. What was it that folks discovered so thrilling about it?

Alex Boten 00:31:01 Yeah, I feel from the maintainer and contributor standpoint, I feel we had been trying ahead to deprecating a few of the different processors that could possibly be mixed inside a single one. It reduces the, once more, I feel it reduces the cognitive load that folks should cope with when ramping up on OpenTelemetry. I feel understanding that if you wish to modify your telemetry, all it’s a must to do is use this one processor and, be taught the language that you’d want to remodel the information versus going by and looking the repository for 5 or 6 completely different processors. I feel that’s typically nice to consolidate that somewhat bit.

Robert Blumen 00:31:39 Inform me extra in regards to the language that’s used to do these transforms.

Alex Boten 00:31:43 Yeah, so the OpenTelemetry language for folk which might be excited about discovering the complete definition is it’s all accessible contained in the OpenTelemetry Collector: can journey repository, but it surely actually permits people to outline in a language that sign agnostic what they wish to do with their knowledge. So it means that you can get explicit attributes, set explicit attributes, and modify knowledge inside your Collector.

Robert Blumen 00:32:09 The opposite inner element of Collectors I need to spend a while on is exporters. What do these do?

Alex Boten 00:32:17 Yeah, so the exporter take the information that’s been ingested by the OpenTelemetry Collector. So, the OpenTelemetry Collector use receivers to obtain the information in a format that’s particular to whichever receiver is configured. It then transforms the information to inner knowledge format inside the Collector after which it exports it utilizing whichever exporter is configured. So, the exporter’s job is to take the information, the inner knowledge format, and format it to the specification of the vacation spot of the exporter.

Robert Blumen 00:32:50 Okay. So, what are some examples of various exporters which might be accessible?

Alex Boten 00:32:54 Yeah, so there’s a bunch of exporters which might be vendor-specific exporters that reside within the repository right now. There’s additionally lots of the open-source tasks have their very own exporters. So, Jaeger has its personal, Prometheus has its personal exporter. There’s a number of completely different logging choices as properly. Yeah.

Robert Blumen 00:33:12 So knowledge is available in, it goes by some variety of processors after which goes out by an exporter. Is there an idea of a pipeline that maps the trail that knowledge takes by the Collector?

Alex Boten 00:33:26 Yeah, so the most effective place to search out that is actually contained in the Collector configuration. So, the Collector is configured utilizing YAML and on the very essence of it, you’ll configure your exporters, your receivers, and your processors, and you then would outline the trail by these parts within the pipeline part of the configuration, which lets you specify what pipelines you need to configure for tracing, and for logs, and for metrics to undergo to the Collector. So, you’ll configure your receivers there, after which your processors, after which your exporters inside every a kind of definitions. And you may configure a number of pipelines for every sign, giving them particular person names.

Robert Blumen 00:34:07 And the way does incoming knowledge choose or get mapped onto a selected pipeline?

Alex Boten 00:34:14 Yeah, so the way in which that the information could be mapped on every pipeline is by way of the particular receiver that’s used to obtain the information. So for instance, in case you’ve configured a Jaeger receiver on one pipeline and a Zipkin exporter on a unique pipeline and also you’re sending knowledge by Zipkin, then the pipeline that has the Zipkin endpoint could be the vacation spot of that knowledge, after which that’s the pipeline that the information would undergo.

Robert Blumen 00:34:40 So, does every endpoint hear on a unique port or does it have a path or what’s the mapping?

Alex Boten 00:34:47 Yeah, in order that will depend on the particular receiver. So, some receivers have the power to configure completely different paths; some solely configure completely different ports. It additionally will depend on the protocol that you simply’re utilizing for the receiver and whether or not it helps it or not. And as I discussed, there’s additionally these items often known as scrapers, that are receivers that may exit and scrape completely different endpoints for metrics, for instance. And people can be configured as receivers, which might then take their very own path to the Collector.

Robert Blumen 00:35:17 I feel we’ve been largely speaking about underneath the belief of a push mannequin, however this scraper sounds prefer it additionally helps pull. Did I perceive that accurately?

Alex Boten 00:35:28 Yeah, that’s appropriate. And, in case you consider the Prometheus receiver, for instance, the Prometheus receiver makes use of the pull mannequin as properly. So, you’ll outline the targets that you simply wish to scrape, after which the information can be pulled into the Collector versus pushed to the Collector.

Robert Blumen 00:35:43 So to wrap this all up, then I’d instrument or configure my sources to level them towards the OTel Collector or Collectors. My community, they might have a website title or an IP handle and a port and possibly a path that comes after that. They’re instrumented, they push knowledge out, it goes to the Collector, the Collector will course of it after which export it again into backend of selection. Is {that a} good description of the entire course of?

Alex Boten 00:36:17 Yeah, that’s precisely proper.

Robert Blumen 00:36:18 How do the sources authenticate themselves to the Collector?

Alex Boten 00:36:23 Yeah, so for authenticating to the OpenTelemetry Collector, there’s a number of extensions which might be accessible for authentication. So, there’s OIDC authentication extension, there’s the bear token authentication extension. You may as well use the fundamental Auth extension in case you’d like. So, there’s a number of completely different accessible extensions for that.

Robert Blumen 00:36:43 Yeah, okay. Properly, let’s speak about extensions. So, what are the extension factors which might be provided?

Alex Boten 00:36:49 Yeah, so extensions are basically parts within the Collector that don’t essentially have something to do with the pipeline of the telemetry going by the Collector. And so, a few of the extensions which might be accessible are the pprof extension, which lets you get profiling knowledge out of the Collector. There’s the well being examine extension, which lets you run well being checks towards the Collector, and there’s a number of different ones which might be all accessible within the Collector repositories.

Robert Blumen 00:37:20 Okay. So, we’ve just about lined most of what I had deliberate about what it does, the way it works. Suppose you may have a undertaking that has not been constructed with this in thoughts and is excited about migrating. What’s a attainable migration path to OTel from a undertaking that may have been constructed a number of years in the past earlier than this was accessible?

Alex Boten 00:37:45 I’d say the primary path that I’d advocate to people is admittedly to consider is there a method that I can drop in a Collector and obtain knowledge within the format that’s already possibly being emitted by an utility. That’s actually the very first step that I’d recommend taking. I do know that there’s a number of completely different mechanisms for accumulating telemetry that predate the Collector. So, telegraph is an instance of a kind of. If in case you have telegraph operating in your setting and also you’re excited about seeing in case you can join it to the Collector, possibly that’s a very good place to begin is, to have a look at connecting the 2. And I do know Telegraph, for instance, emits OTLP, in order that’s already one thing that’s considerably supported. In order that’s actually step one I’d take is can I simply get away with dropping in a Collector and emitting a format that’s possibly already supported?

Alex Boten 00:38:30 One factor to notice is when you’ve got a format on the market that’s not presently supported within the Collector, you possibly can all the time go to the group and ask, ‘hey, is that this a element that people are excited about in adopting?’ And that’s all the time a very good avenue to type of tackle. When you’ve acquired dedication out of your group to possibly change the instrumentation libraries that you simply’re utilizing inside your code, then nice. I’d begin assets. I do know there’s a number of completely different use circumstances which have been documented, I feel on OpenTelemetry.io round migrating away from both OpenTracing or OpenCensus. So, I’d undoubtedly begin in search of these assets.

Robert Blumen 00:39:07 So we’ve talked in regards to the historical past and what it does, what’s on the roadmap?

Alex Boten 00:39:12 Yeah, so on the roadmap for OpenTelemetry, which we truly very just lately printed. So, up till earlier this yr there wasn’t an official roadmap printed by the group. However we’re lastly beginning to change the method somewhat bit to try to actually focus the efforts of the group. So, presently on the roadmap we now have 5 tasks which might be occurring. So, a few of the work is being achieved round each client-side instrumentation, so both, internet browser-based or cellular purchasers, and round profiling. So, that is profiling knowledge being emitted both utilizing an current format, however there’s some dialogue round whether or not or not there’s going to be a further sign referred to as profiles to OpenTelemetry. There’s additionally lots of effort being put into making an attempt to stabilize semantic conventions. So, in case you’ve seen the semantic conventions contained in the OpenTelemetry specification, you’ll most likely know that lots of them are marked as experimental.

Alex Boten 00:40:10 And that’s simply because we haven’t had the prospect of actually focus the group on making an attempt to come back to settlement on what steady Semantic conventions ought to seem like. So, there’s lots of effort to herald specialists in every one of many domains to make sure that they make sense. The opposite efforts that I’m enthusiastic about, as a result of I’m a part of the work, is to place collectively a configuration layer for OpenTelemetry as a complete in order that customers can configure utilizing some type of configuration file, take that configuration file throughout any implementation, and know that the identical outcomes will happen. So, for instance, in case you’re configuring your Jaeger exporter in Python, utilizing this configuration format you’d be capable of take that very same configuration to your .NET implementation or Java and never have to jot down code manually to translate that configuration. After which, there’s some effort round operate as a service assist from OpenTelemetry. So, the group is presently centered round lambdas as a result of that’s the primary serverless or operate as a service mannequin that’s come to us. However there’s additionally effort to herald people from Azure and GCP as properly. To type of spherical that out.

Robert Blumen 00:41:19 We’re at time, we’ve lined every little thing. The place can listeners discover your ebook?

Alex Boten 00:41:25 Yeah, so yow will discover a ebook on Amazon. You may as well purchase immediately from Packet Publishing. And yeah, it’s additionally accessible at your native bookstores.

Robert Blumen 00:41:35 If customers wish to discover your presence anyplace on the web, the place ought to they give the impression of being?

Alex Boten 00:41:40 Yeah, to allow them to, they’ll discover me on LinkedIn somewhat bit on Mastadon or on Twitter — although not as a lot anymore. They usually can discover me on the Slack channels for the CNCF Slack occasion. I’m fairly energetic there.

Robert Blumen 00:41:55 Alex Boten, thanks very a lot for chatting with Software program Engineering Radio.

Alex Boten 00:41:59 Yeah, thanks very a lot. It’s been nice.

Robert Blumen 00:42:01 This has been Robert Blumen for Software program Engineering Radio. Thanks for listening. [End of Audio]