Join us
@obedrina ă» Nov 11,2021 ă» 18 min read ă» 869 views ă» Originally posted on semaphoreci.com
In this episode, we welcome Adam Gordon Bell, Developer Advocate at Earthly, host of CoRecursive podcast. We talk about what Earthly is and how teams can adopt it. We discuss whether being a late adopter is a good or bad thing. Adam also shares his experience with starting his own podcast.
Whatâs in the episode:
Darko Fabijan: Welcome to Semaphore Uncut, a podcast for developers about building great products. Today, Iâm excited to welcome Adam Gordon Bell. Adam, thank you so much for joining us. Please go ahead and introduce yourself.
Adam Gordon Bell: Thanks for inviting me! Iâm excited. I work at Earthly, and Iâm a developer. Iâm working in developer relations now. I also have a podcast about software development, and I live in Peterborough, Canada.
Darko Fabijan: For those of us who donât know, can you please give us an intro to Earthly?
Adam Gordon Bell: Iâll start with the story. Vlad, the founder of Earthly, worked at Google. Internally, they have this build tool called Blaze, which has been open sourced as Bazel. If you work at Google or have a really large monorepo, itâs a great tool chain for building software.
But it only works in a specific context where you have a team that support it, youâre using languages that support it, you have a monorepo with things vendored. It doesnât necessarily work great outside of this context.Â
Read also: Beyond Docker with Earthly
Vlad wanted to come up with a way to build a tool like that, that supported normal development. The way that Bazel works is it uses a file system sandbox to ensure that you donât have any dependencies that are undeclared between each build step, to make the builds reproducible. So, Vlad saw that containers exist and are also a way of sandboxing things.
So he built Earthly. Earthly is a build tool. Itâs like a combination between a Makefile and Dockerfiles. You put targets and describe the build steps within them, and you use it to build your software in a way that is reproducible and parallel.
Earthly is a build tool. Itâs like a combination between a Makefile and Dockerfiles.
-Adam Gordon Bell, Developer Advocate at Earthly
Darko Fabijan: For people who are just getting started with Earthly, what would be some trigger points that they might experience in their workflow and their needs?
Adam Gordon Bell: One would be, you want to be able to make changes to your build but itâs a slow process because you have to change something in the build script and then send it off, have it run and see whatâs wrong. That can be slow. Earthly allows you to abstract all those tasks into something you can run locally because itâs containerized.Â
Another common point is, you have a team that uses more than one language. Once you need to work across more than one language tool, one way to deal with that is writing a bash script or a Makefile.
Earthly is another way. The way we see a lot of people enter is they want to have this way to do things across more than one tool. Or they want to be able to do something locally that previously they could only do in CI or some staging environment.
Another way people join Earthly is coming from multi-stage Dockerfiles. They can get really gnarly. A lot of people come to Earthly because itâs a really nice way to make a multi-stage Docker build.
Darko Fabijan: One of the use cases that Iâve seen for Docker containers that the developers seem happy about is Docker Compose, when itâs going to boot a couple of containers for you and connect them over DNS. Does Earthly work in conjunction with Docker Compose or it has its own way of solving that problem?
Adam Gordon Bell: Earthly supports Docker Compose files. You can use Earthly to build a Docker container but you can also use Docker containers within Earthly and Docker Compose files as well.Â
Hereâs an example. I was helping the Phoenix project, which is an Elixir web framework, work on their CI process. In the CI process for Phoenix they want to make calls that access a database. Itâs an integration test, and they want to test across several databases.Â
So, they have a Docker Compose that starts up like Postgres, SQL Server and MySQL, and then they build their code and then they run tests again. Within Earthly, you just specify those as dependencies and then you say docker run, you give the Compose file. And then within that context you run your tests, and then when you exit that scope then the Docker Compose shuts down. So, thatâs how you could use it with Docker Compose.
Darko Fabijan: You mentioned Bazel. Can you compare Earthly to Bazel?
Adam Gordon Bell: Our origin story is based around Bazel. Bazel builds monorepos in a distributed way with lots of caching, and it does it very well. To do that, it makes certain trade-offs. For instance, the support for the language that you want to build needs to be baked into the tool. Right now, it supports the languages that were commonly used at Google, like Python and Java. If youâre using another tool, then⊠if itâs not built-in support for it, then youâre going to have a hard time.
Also, you need to declare inputs and outputs to these tools, and that can be kind of verbose. Itâs designed to be an amazing build tool for a place where you can have dedicated people to work on it and understand the tool.Â
Earthly is a tool designed to have some of the same principles but to approach it very differently. With Earthly, you can use whatever tool youâre currently using. We tried to give people the benefits of a big tool but that would be more approachable for everyday people.
With Earthly, you can use whatever tools youâre currently using. We tried to give people the benefits of a big tool but that would be more approachable for everyday people.
-Adam Gordon Bell, Developer Advocate at Earthly
Another thing is Earthly supports polyrepos. You pull down your dependencies from this Git repository, build that, and then pull down from here, build that. And it doesnât all have to be in one, big monorepo.
Read also: What Is Monorepo (And Should You Use It)?
Darko Fabijan: If someone wants to try Earthly today, what do you suggest as some first steps to getting to know Earthly?
Adam Gordon Bell: Go to Earthly.dev. Thereâs a landing page that will explain some of the steps. You can download Earthly, and if youâre on Mac, just do brew install earthly
. Then you can take a look at some of the examples.Â
Darko Fabijan: Yeah, when you watch that video on the landing page, it becomes pretty clear how it works, and itâs very very approachable. Earthly is a CLI tool that you can use. You mentioned that you approach some of the tools, like the very classic tools from the Unix ecosystem, or you reapproach them with a fresh eye. Can you talk more about that?
Adam Gordon Bell: In this role, Iâm helping people build software. Earthly is very much built around a developer-first approach to building things thatâs based in the command line and not a big YAML file but something more declarative.
Earthly is very much built around a developer-first approach to building things thatâs based in the command line and not a big YAML file.
-Adam Gordon Bell, Developer Advocate at Earthly
It made me start to think about other command line tools that are classically used and sometimes get used in builds. And thereâs tools that are really useful but I never really knew how to use.Â
So, the first one was JQ, a command line tool for dealing with JavaScript. When I didnât know how to use it, I would go to StackOverflow and end up copying and pasting something from there.
I spent time trying to understand how JQ works and writing it down in a way that would explain it to other people. If I spent half a day learning it, think of the dividends that would pay. I wrote an article about it, called An Introduction to JQ.
The takeaway: learn how to use your tools.
Darko Fabijan: Exactly! I have many regrets that back in the day when I had much more time, I would learn something to a shallow extent. Iâd read only the first two or three chapters from a book, Iâd get the idea. Then Iâd end up limping for the next 10 years of my career because I hadnât spent the time learning.Â
I had a similar experience with other tools which are not my primary tools, like Bash. I know bits here and there but there are a couple of syntax-y things that I just donât know. As my career progressed, I started helping other developers develop and move on. I started noticing the same in other people, that I wasnât the only one leaving the last two or three chapters.Â
So I ended up suggesting to people to pick those essential tools that will not age or are going to age well, and invest. As you have put nicely, one will get dividends over time.
Adam Gordon Bell: Thatâs interesting to think about. Similarly to the experience you had with Bash, I had the same experience with Ack. When I wrote the JQ article, somebody was like, âThis is really great. I feel like JQ is like Ack. I never quite knew how to use it.â And I was like, âYeah, I have no idea how to use it. Iâve copied some things once in a while.âÂ
I think you hit the nail on the head by saying itâs something to do with tools that are a bit older, because thereâs less risk. If somethingâs still around 10 years later, thereâs less risk itâs going to go away.Â
Speaking of tools, if somethingâs still around 10 years later, thereâs less risk itâs going to go way.
-Adam Gordon Bell, Developer Advocate at Earthly
Darko Fabijan: Another thing that you do that I find super amazing is your podcast. You put a lot of care into it, and itâs very interesting to listen to. The guests are also amazing. Can you please tell our listeners about the origins?Â
Adam Gordon Bell: My podcast is called CoRecursive. I started doing podcasts as a guest host for a couple of other podcasts, one of which is Software Engineering Radio. They have a training process for how you do the podcasts and thereâs a lot of preparation ahead of time. You pick a topic and you make sure that you get all the facts and answers out of people. So I did that and started my own podcast in parallel to it.
I was talking to people a lot about functional programming as I was a Scala programmer at the time. I would just ask people, what is this? What is that?Â
But over time, I felt that the parts of the podcast I really liked were more about peopleâs experiences. Instead of explaining what X is, tell me when you first encountered X. What was it like? What happened? What were you wearing? Just really getting somebodyâs personal story. So Iâve just kind of doubled down on that.
I really like it. I recall being a software developer and being at lunch, and something had blown up yesterday and somebody would get called in, and then they kind of tell you the story. âOh, the web server was down,â and then somebody is like, âOh, did you look, was it this?â And, âOh, I looked. It wasnât that.â The drama, I guess, of somebody recounting these stories instead of explaining to you, âOh, what is a web server?â So, Iâve just been doubling down on this concept. Now my podcast is very story focused. The most recent episode was about this guy, Paul Lutus, and he was known as the Oregon Hermit. He built software for the Apple II from a cabin he built himself in the woods. Itâs a very strange story. Itâs super interesting.
Adam Gordon Bell: Thereâs things to learn from peopleâs stories. His story about how he worked remotely in the woods has some things to teach us about how to work remotely. He also thinks that developers have lost touch with product and the business side. He wants developers to be more fully rounded people. But itâs hard to understand that until you hear his story about how he built the software. I like to focus on the experience, the experiences of software development.
Darko Fabijan: Another thing that fascinated me was the episode about the SQLite and what are the origins of it and how itâs actually connected to the warship, or itâs a battleship story. It might be very inspiring for folks who are just getting started in the area.
Also, there is that generational difference. There are people who started coding in the â80s, and their view on all the current tools and trends and everything is quite different from the people who just started a decade ago. And thatâs why I also found your guests interesting, because a couple of folks that I checked out are really, letâs say, seniors.
Adam Gordon Bell: Yeah, like a seniorsâ⊠both in terms of title and in terms of seniorsâ discounts at a restaurant.
Darko Fabijan: I find that inspiring. A vast majority of people that I worked with were maybe a decade older than me. Thatâs one of the reasons that itâs more interesting to me to talk with folks who have seen industry progress over a couple of decades.
Adam Gordon Bell: Yeah. Itâs interesting how things change really fast, but some things donât change that much. I mean, you were talking about learning how bash works. I mean, thatâs not new, right?
Darko Fabijan: Yeah, absolutely. So I would say, someone learned the best all the way back in the early â90s and covered Postgres in the early â90s or just SQL in depth. Now, the guy or girl is just fine even today.
Adam Gordon Bell: Itâs a tricky thing. At some point I remember learning Angular 1, and I donât think it was ever useful for me besides the couple of months when I used Angular 1, because it turned over very quickly. So you got to get beneath that surface level and beneath the hot new thing and look to the things that have stood the test of time.
Darko Fabijan: Yeah. There is that piece of mesh in the library, itâs called Envoy. Itâs made by Matt Klein. He was one of the first people that I talked to who very strongly expressed that he is a late adopter. He wants to be a late adopter. He explained the number of benefits of that. I mean, as I finished university, I started working and all that. Youâre always looking at the news and whatâs the hot new thing, hot new thing, hot new thing. And whilst the majority of people are kind of like that, âOkay, whatâs next?âÂ
I think there are also huge benefits in looking the other way. Maybe you need to wait for the first 10 years of your career, that you run through a couple of things. And then at that point, find what are those stable components of your career that you are going to invest, and you will be safe building software which is super stable and is going to stand the test of time.
Adam Gordon Bell: Matt is one of the investors into Earthly. Heâs a very talented developer, for certain. I heard somebody else say, who was also a slow adopter, that sometimes they have to learn less than people who are fast adopters. I forgot who this was but they were saying, they missed Ruby. Everybody was excited about Ruby, and they were like, âOh, weâll wait. Iâll wait on this. Iâll wait on this,â and then Ruby fell out of popularity and heâs like, âOh, I didnât have to learn that.â
People love Ruby so I donât want to diss it, but sometimes these things fade. If you move slow enough, you donât even have to learn them.
Darko Fabijan: Also, in the first days thereâs less documentation. The stability, the quality of everything is lower. There are less people. And theyâre also exciting times but they also tend to be quite rough.
Adam Gordon Bell: There is another side where you could make a name for yourself as a developer by going to the newest language and thereâs probably packages missing, and you can develop these packages, right? You could⊠whatever. Youâre the best at making a hash map or a dictionary. And each new language needs some sort of concurrent hash map and you could bring that to them. So thereâs downsides, I guess, to being slow at adopting things.
Darko Fabijan: So, we talked through these couple of approaches, how to pick the right tools and how in-depth to go with them. Can you share your experiences when you were building Earthly? What are some of the challenges that you faced?
Adam Gordon Bell: Earthly is being built in the open, itâs on GitHub. One of the challenges we have is what features we add. So, Earthly is like a syntax for describing a build. Itâs just kind of like a small programming language. And people ask for features, but if you just add every feature youâre not thinking through what the end state will be and you can get to a weird place.Â
So, we spent a lot of time thinking about this. For instance, somebody asked for this feature, using host mounts. That would make builds more performant, at least in their case, but it could hurt the reproducibility. It would be possible to have something not build the same on everybodyâs environment, so should we add that?Â
So, we came up with this list of things that we value in the product, and we try to basically consider whether a feature helps these values or kind of hinders them. So, if our number one value was performance, then that feature that hurts reproducibility but does help performance, we would certainly add it. But actually, we value reproducibility above speed. So, we did not put that feature in.
Adam Gordon Bell: And so, we came up with this list. Weâre actually going to be writing a blog post about it. And this idea came from Bryan Cantrill, who said that every software platform has things they value more than others. We value both speed and reproducibility, but the thing is you need to decide whatâs more important to you.Â
And that order actually ends up being a lot about what your programming language, what your product, becomes. One thing we value a lot is versatility, so we want people to be able to bring whatever programming language, whatever build things theyâre using, and use it within Earthly.Â
So, we put that as our very top value. Bazel, that we were discussing earlier, puts reproducibility at the very top; the build should always be byte for byte perfect. So, that means they canât have this versatility. They canât just allow people to use any given build tool. They need to kind of gate that, right? And so, the way you stack these software values really informs what your product becomes.
Adam Gordon Bell: Weâre writing this up, but we pick versatility as our top goal, so we want people to bring whatever they want and use it to build things. And then we picked approachability as our next. So, builds are often a thing where you donât spend all day looking at it. Itâs just once in a while you need to go and change the build.Â
So for us, that means itâs important that the build should be straightforward to understand. If thereâs a feature thatâs super powerful but it takes a bit to follow what it does, we would rather not include that, because we know that people donât spend their whole days in builds. Itâs a once in a while thing, so it needs to be easy to understand at first blush.Â
Thatâs a very long way of saying, the biggest challenge we face is how to rank these trade-offs between the values that features kind of embody in your product. And I think we figured it out. I mean, thanks to Bryan Cantrill with this idea that you need to put the values of your product first.
Darko Fabijan: Thatâs very interesting. We have been facing backlogs, which would take a century to execute all the time, and itâs a real struggle. There are a lot of great ideas, but there is a limited amount of time, but you can end up somewhere you really donât want to go. And you just look around and you see that maybe three years ago you took the wrong street.
Adam Gordon Bell: Yeah. And if I look at the Semaphore website, it talks a lot about speed, right? Speedâs important in CI, so I would guess that, that oneâs really high for you guys, right? If thereâs something that would slow things down, but whatever other benefits, forget about it. Speed is where weâre focused.
Darko Fabijan: Okay. Thank you so much for your time. Good luck with Earthly and your podcast. Itâs a really amazing podcast, so go and check it out.
Adam Gordon Bell: Thank you. Yeah. You can find me on Twitter, I guess: @adamgordonbell. Or yeah, in your podcast player: Adam Gordon Bell and the CoRecursive podcast. And Earthly is on GitHub or at earthly.dev.
Join other developers and claim your FAUN account now!
Influence
Total Hits
Posts
Only registered users can post comments. Please, login or signup.