Today I found something surprising - a piece of code in C# which awaited a new Task(...)
and which achieved its goal, but did the opposite of what I expected. Indeed, instead
of the task running to completion, it instead never completed. In this post I'll share
some interesting empirical observations about how this works and talk about how this
differs from asynchronous execution systems in other languages.
The Prusa XL is a high-end Core-XY 3D printer from Prusa Research, the technical specifications of which you can find on their product page. I was recently looking add another printer to my home lab, alongside my trusty Voron 2.4R2, and the Prusa XL was one of several printers I considered - it is also ultimately the one I chose to purchase based on its unique set of trade-offs.
The biggest challenge I faced making this decision was the lack of any deep technical reviews of the Prusa XL, so I'm going to try and close that gap here by sharing my thoughts and experiences through the lens of someone who has spent the last couple of years diving into the guts of what makes for a reliable 3D printer.
The tech world is filled with people who have strong views and are more than happy to share them. An unreasonable number of them are priviledged white males. Many of them see the world through the lens of their own personal experience and don't bother much with external perspectives. Far too many are eager to start a fight over their religion of choice (be that a language, a technology, a "thought leader" whose message they subscribe to, or a topic they have precious little familiarity with).
I'm many of those things, I'd really like to avoid being all of them. Let's talk perspective and trying to see the big picture. Let's talk about how to be wrong, and be okay with that.
The world today strikes a particularly jarring contrast with the one I grew up in. Buzz-words and gimicks have taken center stage in an environment where a "cheap" alternative can be found for thruppence on the quart (or however that monetary system worked).
To compete in a sea of identical products, companies pitch differentiating features, many of which seem to have been crafted through a haphazard "Cards against Humanity" approach where every second card is the word "AI". Before that it was "Blockchain", and before that "The Cloud".
Thing is, for all that these products will try to convince you that they have the singular ability to magically solve whatever problem you face; the simple reality is that the laws of physics cannot be negotiated with.
There is no magic, you cannot beat the Shannon compression limit, you can't move information faster than the speed of light, and you must choose between consistency or availability.
If you've found yourself in a situation where your PostgreSQL database sequence numbers are out of sync with the data in your tables, you'll likely want to get to fixing that.
This script is intended to automatically fix the sequence numbers for all tables in the current database.
Every now and then you'll find automated code coverage tooling getting in the way of you landing changes when you're pretty certain you've written tests for the code in question.
Let's take some time to dig into how code coverage works and then use that new knowledge to find ways to unblock ourselves.
In the software and computer industry, we have a tendency to assume that the world, its inhabitants, and most of all ourselves are deterministically rational beings. At the risk of inciting the ire of the orange site, this is patently inaccurate and allows us to justify a diverse array of logical fallacies.
Advent of Code 2023 has just kicked off, and I'm going to try something a bit different this year, I'm going to try and share useful concepts and patterns that play a role in solving each day's puzzle.
Today, I'm going to talk about some how I go about optimizing algorithms for practical performance on data-intensive problems in Rust. I find that knowing how to optimize algorithms for practical performance is one of those skills that many overlook because it's rarely talked about, and yet it can have significant implications on performance for your applications.
Advent of Code 2023 has just kicked off, and I'm going to try something a bit different this year, I'm going to try and share useful concepts and patterns that play a role in solving each day's puzzle.
Today, I want to talk about some cool tricks for visualizing shapes in your
application's debug output by taking advantage of Unicode's Box-drawing
characters. Most humans are visual creatures and have an incredible ability to
spot patterns and interpret visual representations of data, so taking advantage
of some of the tricks I've previously shared in
Advent of Code 2023 has just kicked off, and I'm going to try something a bit different this year, I'm going to try and share useful concepts and patterns that play a role in solving each day's puzzle.
Today, I wanted to talk about making your objects sortable and how you can use this to take advantage of the built-in sort functionality in your language of choice. This is an incredibly useful pattern to know about, and one which I found useful for Day 7 of Advent of Code 2023.
Advent of Code 2023 has just kicked off, and I'm going to try something a bit different this year, I'm going to try and share useful concepts and patterns that play a role in solving each day's puzzle.
Today, I wanted to share a little bit of how I approach solving hard problems, and specifically how I set things up to make it easier to remain engaged in the face of ongoing challenges.
I think this is particularly relevant to Advent of Code, because the puzzles continue to get harder each day and at some point all of us will struggle with a problem that we just can't seem to get a solution for. I find that having a process for remaining engaged in the face of a challenge is a superpower that will help you grow beyond what you thought you were capable of.
Advent of Code 2023 has just kicked off, and I'm going to try something a bit different this year, I'm going to try and share useful concepts and patterns that play a role in solving each day's puzzle.
Today, we're looking at how you can use dynamic programming to save yourself a lot of computation, and how I spot and reason my way towards solutions in this space.
Advent of Code 2023 has just kicked off, and I'm going to try something a bit different this year, I'm going to try and share useful concepts and patterns that play a role in solving each day's puzzle.
Today, we're looking at how you can render objects to human readable representations in Rust, and how you can use that to make debugging your code easier.
Advent of Code 2023 has just kicked off, and I'm going to try something a bit different this year, I'm going to try and share useful concepts and patterns that play a role in solving each day's puzzle.
Today, we're looking at how type converters work in Rust, and how you can use them to create intuitive interfaces for your types.
Advent of Code 2023 has just kicked off, and I'm going to try something a bit different this year, I'm going to try and share useful concepts and patterns that play a role in solving each day's puzzle.
Today, we're looking at iterators (in Rust) and how they can be used to simplify your code by abstracting away the complex details of incremental computation.
Over the years I have tried several different budgeting tools, from spreadsheets, to desktop applications, to one-person-startups, and finally to YNAB. They've all had their quirks and limitations, but in the end YNAB is the one I've hated the least, and with some creative automation, I've been able to close the deal-breaking gaps in its functionality and happily use it for several years now.
In this post, I'll share how I've set built and hosted my own YNAB automation system, and how you can easily (literally a few minutes) set up your own.
Burnout is one of those terms that most of us have heard used in our professional lives, many of us have been subject to, and yet few of us really have a means of grasping the severity of the problem or when it goes too far.
Hopefully this post will give you some ideas no how to reason about burnout in both your professional and personal environments.
If you're running a (large) API on ASP.NET (.NET Framework) there's a good chance that you're significantly throughput and concurrency limited. Part of this is no doubt due to the relative performance increases we've seen in ASP.NET Core over the years, but I'm here to show you a one line change you can make which (depending on your use case) might unlock a significant amount of additional performance headroom.
As engineers, managers, friends, or family members we are often called upon to review the work of others. Reviews are a critical part of our professional social contract and give us the opportunity to build cohesion, socialize knowledge, improve clarity, and support the production of high quality artifacts.
How we comment in a review determines whether we are able to foster those positive outcomes, or end up in a confrontation over the validity of our distinct perspectives. In this blog post I'll talk about how you can approach reviews in a manner which is more likely to succeed in more situations.
Site Reliability Engineering is an incredibly interesting field, one which has straddled the interesting line of both being relatively clearly defined and being hugely open to interpretation. For many familiar with it, the idea of SRE brings thoughts of SLI/SLO/SLAs, incident response, on-call, and an obsession with system architecture and failure modes. For others, the line between concepts like DevOps and SRE is hard to make out, and to add yet another perspective to the pile - it can simply mean "we want to hire an ops team, but nobody wants to work for us unless we call it something else".
Recently, Niall Murphy (one of the original authors of the (in)famous Google SRE Book) wrote a thought provoking piece about the future of SRE and the assumptions that underpin it. It does a great job of articulating some of the problems that have consistently worn a blister in my experience of SRE - especially when it comes to articulating what it is we do to leadership.
I'd like to present what Site Reliability Engineering means to me and a hypothesis for what SRE may be when we take a step back from some of the implementation details.
So, you're using GitHub Actions to deploy your project and have tossed some service principal credentials into your GitHub Actions Secrets to let you do so. The birds are signing, the sun is shining, an hackers are hacking your code coverage service...
How confident are you that your service principal credentials aren't compromised? If you're like me, that number goes to zero very, very quickly. Rotating them for hundreds of repositories and service principals is far from a simple task, and I hate having to do complex work - so let's look at a better solution.
Enter Hashicorp Vault, a comprehensive secrets management platform which (amongst other things) lets you issue short lived credentials with limited permissions. If configured correctly, this can help greatly reduce the risk surface area for compromised credentials and minimize the operator overhead associated with managing them.
This blog post is a top-to-bottom run-through of setting up Hashicorp Vault and GitHub Actions so that you can easily consume secrets from your GitHub Actions workflows.
I've long been a fan of Hashicorp's Vault for its ability to go far beyond the traditional "key/value" secrets management solution. Even "advanced" offerings like Azure's KeyVault tend to barely scratch the surface of what is possible with Hashicorp Vault.
Unfortunately, for all that Vault is phenomenally powerful, it is also a fair bit more difficult to setup and run than its cloud-native counterparts. Where deploying an Azure KeyVault only takes a few clicks and runs almost for free, Hashicorp Vault's suggested configuration relies on you building and maintaining a cluster of virtual machines, which is a lot of additional work and cost.
While I'm not averse to putting in effort and paying a bit of money for a good service, I'm also not going to do so if there is an easier solution at hand. This blob post is a (not so quick) run through of how to run Vault without the overhead of managing your own cluster, and without needing to spend a small fortune on virtual machines.
When it comes to planning and strategy; Site Reliability Engineering teams walk a fine line. On one side of this line is a team who is closely aligned with the product group's priorities, to the point where they cannot be distinguished from one another (staff augmentation). On the other side is a separate operations team who only peripherally works on the same product as the product group.
Neither of these extremes is healthy and a well functioning SRE team will attempt to strike a balance between the two. In my mind, our goal is to balance the broomstick in the palm of our hand. This kind of balance trick is one that every control systems engineer is familiar with and the technique for solving it is a well trodden path. Let's discuss it and use it to formulate a method for planning how your team works on a daily basis.
Modern languages are increasingly moving towards async as the default means by which
concurrency and I/O are managed, reducing our visibility into the threads and processes
which underpin our code. The promise is that, by tossing a few async and awaits around
your codebase, you'll achieve scaling nirvana - but as with all magic, there is a dark side.
Over the course of my career, I have spent a lot of time leveraging and debugging asynchronous
code across a range of different languages. One of the things that has helped me do so is
a solid mental model of how a feature like async is implemented and the impact that has.
This blog post is my rough attempt at explaining that.
Dashboards are one of those things which sound easy to get right on the surface, but often fall short of expectations. In the majority of cases, my experience has been that they, at best, do not support an investigation and at worst can actively hinder it.
That is a huge pity, because when they are done well, dashboards can be one of your most powerful tools for showcasing the current state of your system and enabling engineers to rapidly track down failures.
At one point in my career, I spent over two years building a monitoring stack. It started out the way many do; with people staring at dashboards, hoping to divine the secrets of production from ripples in gravitational waves before an outage occurred. The most experienced engineers could sometimes spot the high energy particles as they were about to flip bits, but I never got that good.
Over these two years we was able to transform not just the technology used, but the entire way the organization viewed monitoring, eventually removing the need for a NOC altogether. To achieve this, we spent a lot of time focusing on what is important and, more importantly, what is irrelevant in a monitoring system.
What I'm going to describe here isn't that stack specifically, but rather the key design decisions and crucial pieces which allowed us to build something successful and which have left me wishing for something equivalent in everywhere I've worked since. If you're planning to build a monitoring stack from the ground up, this might be a good place to start.
Around the world, Israeli intelligence agencies have a reputation for punching well above their weight. For a country of less than 10 million people, they compete with nations orders of magnitude larger and have routinely demonstrated an ability to achieve far more than their larger cousins.
There are many factors which play a role in this, but today I'd like to focus on a small keynote in that story which has, on numerous occasions, been the primary factor in advancing the reliability of the services I support. That factor is the role of the "Devil's Advocate".
Recently I was having a conversation with a colleague who asserted that we (SREs) are broadly the types of engineer who, if given the choice, try to focus on perfecting the fundamentals. This surprised me, because if you were to ask me about my views on engineering, I'd probably lean in a slightly different direction.
My personal view on SRE is that its a game of balance. We're not Software Engineers, we're not Operations Engineers and we're also not Security Engineers. We tread a fine line in the middle, pushing on aspects of the broader (humans included) system to help it find a stable equilibrium in which it delivers maximum value for all stakeholders. That kind of balancing requires a very pragmatic, flexible approach and often depends more on the subtleties of the system at hand than a rigidly theoretical approach can offer.
With that in mind, I think that as engineers, we need to focus on building systems that support that healthy equilibrium. Doing so means balancing a wide range of requirements from different, often competing, stakeholders while attempting to divine what the future may bring. In my experience, however, all of this becomes much easier to deal with if you can solve two key problems: velocity and observability.
Before I dive into that, let's quickly talk about that experience.
Today I work as an SRE, surrounded by dozens of complex systems designed to make the process of taking code we write and exposing it to customers. It's easy to forget that software deployment itself is a problem that many developers have not yet solved.
Today I'd like to run you through a straightforward process I recently implemented for Git Tool to enable automated updates with minimal fuss. It's straightforward, easy to implement and works without any fancy tooling.
So you're sitting in front of your computer, wondering why your unit tests won't show up in Visual Studio's Live Test Window. They appear fine in the normal Tests Window and they run without problems, you haven't done anything weird and all you want is to be able to see whether your code works.
You're not alone and there is a solution!
Have you ever run into a situation where Git just refused to obey your commands? No, I'm
not talking about that time you "typo-ed" git commit and ended up git reset --hard-ing
your repository back to the dawn of the universe, I'm talking about it really, truly,
ignoring you.
I have, so let me tell you a story about what happened and how I fixed it so that you can avoid future hair-loss and avoid questioning the nature of your reality.
Tips
(For those who want to skip right to the punchline)
Make sure your .gitignore file is saved in UTF-8 format, or you'll have a bad time.
As an engineer, I like to think that I help fix problems. That's what I've tried to do most of my life and career and I love doing so to this day. It struck me, though, that there was one problem which has followed me around for years without due attention: the state of my development directories.
That's not to say that they are disorganized, I've spent hours deliberating over the best way
to arrange them such that I can always find what I need, yet I often end up having to resort to
some dark incantation involving find to locate the project I was certain sat under my Work
folder.
No more, I've drawn the line and decided that if I can't fix the problem, automation damn well better be able to!
I'd like to introduce you to my new, standardized (and automated), development directory structure and the tooling I use to maintain it. With any luck, you'll find it useful and it will enable you to save time, avoid code duplication and more easily transition between machines.
If you've built a production API before, you'll know that they tend to evolve over time. This evolution is not only unavoidable, it is a natural state that any active system will exist in until it is deprecated.
Realizing and designing to support this kind of evolution in a proactive way is one of the aspects that differentiates a mature API from the thousands that litter the Wall of Shame.
At the same time, it is important that your API remains easy to use and intuitive, maximizing the productivity of developers who will make use of it.
It's a poorly hidden fact that I love Kubernetes. After spending months running everything from Marathon DCOS and CoreOS to Rancher and Docker Swarm in production, Kubernetes is the only container orchestration platform that has truly struck me as truly "production ready" and I have been running it for the past year as a result.
While functionality when I first started using it (v1.4) was somewhat patchy and uninteresting, some of the more recent updates have been making sizeable strides towards addressing the operations challenges we face on a daily basis.
With v1.8, Kubernetes has introduced the CronJob controller to batch/v1beta1, making it
generally available for people to play with. Sounds like the perfect time to show you how we
use CronJobs to manage automated, scheduled, backups within our environments.
One of the most interesting discussions to have with people, notably those with traditional database experience, is that of the relationship between an off the shelf RDBMS and some modern NoSQL document stores.
What makes this discussion so interesting is that there's invariably a lot of opinion driven from, often very valid, experience one way or another. The truth is that there simply isn't a silver-bullet database solution and that by better understanding the benefits and limitations of each, one can make vastly better decisions on their adoption.
Docker is become an incredibly prevalent tool in the development and operations realms in recent months. Its combination of developer friendly configuration and simple operational management make it a very attractive prospect for companies and teams looking to adopt CI and CD practices.
In most cases, you'll see Docker used to deploy applications in much the same way as a zip file or virtual machine image. This is certainly the most common use case for Docker, but by no means the extent of its functionality.
In this post I'm going to discuss some of the more interesting problems we've used Docker to solve and why it serves as a great solution to them.
Aurelia is a modern web application framework in the spirit of Angular, with an exceptionally concise and accessible developer experience and standards compliant implementation. It is hands down my favorite web framework right now and one I'd strongly recommend for most projects.
One of Aurelia's greatest claims to fame is the incredible productivity you can achieve, enabling you to build a full web application in just days, if not hours.
When building the application becomes that fast, spending a day putting together your deployment pipelines to roll out your application becomes incredibly wasteful, so how can we avoid that?
Well, Docker offers us a great way to deploy and manage the life-cycle of production applications. It enables us to deploy almost anywhere, with minimal additional effort and in a highly reproducible fashion.
In this post I'll go over the process of Dockerizing an existing Aurelia web application built with WebPack, however the same process applies to those built using SystemJS.
With the increasing popularity of Git as a tool for open source collaboration, not to mention distribution of code for tools like Go, being able to verify that the author of a piece of code is indeed who they claim to be has become absolutely critical.
This requirement extends beyond simply ensuring that malicious actors cannot modify the code we've published, something GitHub and its kin (usually) do a very good job of preventing. The simple fact is that by adopting code someone else has written, you are entrusting your clients' security to them - you best be certain that trust is wisely placed.
Using Git's built in support for PGP signing and pairing it with Keybase provides you with a great framework on which to build and verify that trust. In this post I'll go over how one sets up their development environment to support this workflow.
Anybody who has worked in the development world for a significant portion of time will have built up a vast repertoire of abbreviations to describe how they solve problems. Everything from TDD to DDD and, my favourites, FDD and HDD. There are so many in fact that you'll find a website dedicated to naming and shaming them.
I'm not one to add another standard to the mix... Oh who am I kidding, let me introduce you to Chance Driven Development.
Inki is a small proof of concept project I've been working on which is designed to manage transient, single-use, SSH keys for an automated remediation tool our team is in the process of building.
In this blog post I'll go over some of the design decisions motivating a tool like Inki, some of its interesting implementation details and the questions we're hoping it will allow us to answer.
Tips
If you haven't yet read the article on Bash CLI then go read it now.
Bash's ability to automatically provide suggested completions to a command by pressing the Tab key is one of its most useful features. It makes navigating complex command lines trivially simple, however it's generally not something we see that often.
Bash CLI was designed with the intention of making it as easy as possible to build a command line tool with a great user experience. Giving our users the ability to use autocompletion would be great, but we don't want to make it any more difficult for developers to build their command lines.
Thankfully, Bash CLI's architecture makes adding basic autocomplete possible without changing our developer-facing API (always a good thing).
Tips
If you're just looking to hop straight to the final project, you'll want to check out SierraSoftworks/bash-cli on GitHub.
Anybody who has worked in the ops space as probably built up a veritable library of scripts which they use to manage everything from deployments to brewing you coffee.
Unfortunately, this tends to make finding the script you're after
and its usage information a pain, you'll either end up grep-ing
a README file, or praying that the script has a help feature built
in.
Neither approach is conducive to a productive workflow for you or those who will (inevitably) replace you. Even if you do end up adding help functionality to all your scripts, it's probably a rather significant chunk of your script code that is dedicated to docs...
After a project I was working on started reaching that point, I decided to put together a tool which should help minimize both the development workload around building well documented scripts, as well as the usage complexity related to them.
Traefik is an application load balancer written in Go and designed to simplify the task of serving HTTP(S) services whose configuration changes on the fly. Traefik v1.1.0 was recently released with support for Docker Swarm and it works excellently.
In this post, we'll go through how one sets up their Swarm cluster to automatically expose its services through Traefik.
Sierra Softworks has a brand new website, rebuilt from the ground up using the brilliant Hexo project. A lot of emphasis was placed on making it as easy as possible for us to publish new content here while minimizing the rate at which content becomes outdated (something our previous website suffered from rather badly).
As a result, we've tried to move all the project pages to their GitHub repositories
and provide a dynamically generated list of them
Tips
If you can't find one of our older projects here, please send us an email.
Docker Swarm is one of those interesting new technologies which has succeeded in shaking up people's preconceptions around what it means to run a scaleable cluster. In an environment where everyone seems to be building a cluster orchestrator, including some big names like Google's Kubernetes, HashiCorp's Nomad and Mesosphere's Marathon; Swarm has managed to burst through as one of the most attractive orchestration frameworks out there.
As a result of all this hype, it can be difficult to make a decision around whether Swarm is the right tool to use. As someone who has had extensive experience with running Swarm, Kubernetes, DC/OS (Marathon) and Rancher in production environments, I'll try to give you an unbiased view on the reasons you'd choose Swarm and some of the gotchas to be aware of.
Until now, all of my work on websites has been done in HTML. Write HTML for this page, write HTML for that project and so on. HTML is one of those languages which anyone who considers themselves good with computers should know, but it also leaves a lot to be desired. In the latest version of our website, I decided to move to Markdown as our primary markup language for documents. Markdown is one of those languages which continues to grow more popular, especially on very tech-centric sites like StackOverflow and GitHub and yet if you talk to most people who are merely "good" with computers, they have never heard of it. Somewhat strange given that Markdown is designed to be an easier to use, easier to read, shorthand version of HTML for writing documents; but I guess that's just the way of things.
Code Highlighting is one of those things which doesn't seem like a big deal, until you see what a difference it can make. The issue is that source code is inherently difficult to read due to the vast number of keywords and punctuation used by compilers to understand what we are trying to tell them to do. In an effort to combat this difficulty, we rely on two different tools.
The first, formatting, is probably the most important; it is the process of making code easier to read through added whitespace, often this whitespace makes no difference for a compiler but by adding newlines and tabs, humans are able to read it considerably more easily.
The second, highlighting, is the automated (or manual, if you're a masochist) process of colouring different parts of the source code to make it easier for humans to read. This involves colouring specific keywords in certain colours, maybe colouring variable names another etc.
Static websites are synonomous with the dawn of the internet, before database servers became mainstream, before the advent of the CMS and long before the dawn of the web application. Over the years we've seen the advent of web development frameworks like Ruby on Rails, Express.js and MVC to name but a few. These frameworks include support for advanced templating engines, database backed page generation and custom routing, but is it really necessary to use such a framework when a static website might address all the same problems at a fraction of the cost.
First off, I'm not a graphic designer by profession and haven't received any kind of training in the field - so don't take this as a tutorial on how to create your company's logo because in all likelihood I haven't got the faintest clue what I'm talking about. I am, however, a huge fan of learning to do new things; and in my case that generally involves mashing together a bunch of Google searches until I find some information that gets me on the way.
Benjamin Pannell
Site Reliability Engineer, Microsoft
Dublin, Ireland