Nick Bruun

→ Asynchronous task execution at scale

Nov 12, 2020 21:06:00

Where asynchronous task execution fits into a larger system is something I’ve been pondering for a while. It is an interesting type of problem, which exists in an unusually strong intersection of operability, guarantees, and fundamental system design principles. Much like system design itself, two extremes bookend the spectrum of common solutions.

On one end, we treat asynchronous task execution as a purely local concern. In this model, every framework, subdomain, or even service implements its own asynchronous task execution that is just a part of how the subsystem runs. Operations equally become a local concern.

On the other end, asynchronous execution is a fundamental property of the entire system. We tightly integrate frameworks and use them as first-tier hosts of the system. If that description is too abstract, Celery is the Python world’s canonical example. This design is familiar to most as the de facto approach in most monoliths¹. While tightly coupled, it does offer some benefits as compared with the localized design. Most prominently, it makes centralizing operations and tooling natural².

In between these two extremes lies a myriad of possible designs. In reality, though, trade-offs become very nuanced. This work is therefore rarely released, and little has been published about this topic. That only makes it altogether more exciting that Dropbox yesterday published a design paper on their solution to this problem at scale. More exciting still, they went with my current personal local optima: a loosely coupled service that relies on well-defined boundaries rather than implementations to execute tasks as if they were any other type of invocation.

This design beautifully centralizes operations of scheduling and invocation while leaving decisions about and operations of execution localized — in other words, localizing operational expertise and system impact. Specific implementation details, such as the polling-heavy design, clearly result from a particular set of trade-offs. But, at a high level, I genuinely believe this design offers a clear improvement over most approaches I’ve seen in the wild, namely by separating concerns, both technically and organizationally. While nuances remain nuances, it would be truly exciting to see this shape of solution generalized well and open-sourced as a component in our shared toolbox.

Regardless, it’s incredibly exciting to see companies publish their solutions to very big design problems.

At times, this is the implementation approach in a system where asynchronous task execution is solved locally. The big difference is obviously that the implementation is not centralized. When blowing out a monolith, this is a piece that is often kept stable and just localized instead. Dozens of Celery clusters are a special kind of scary to me. ↩
Centralized along with everything else, of course. ↩

→ Stay in the game

Aug 05, 2019 18:39:41

I have referenced this story to a few people over the last month or so. There isn’t any one passage that stands out. It’s just a simultaneously incredibly frightening and heartwarming story about the nature of humans.

→ Better by default

Jun 12, 2019 23:31:20

Going through some old notes on technology choices, I came across Jason Moiron’s excellent commentary on iron.io’s performance gains from their switch to Go from a few years ago:

People have commented that these savings could have been gained from writing critical sections in C or by going over the original code with an eye for performance. Putting the obvious parallelization benefits that Go has over most other languages aside, the point they’re missing here is that you more or less achieve these results by writing normal Go code.

While I still think Go lacks a lot in terms of language features to make productivity a priority, it’s hard to dispute the fact that for a number of cases, Go is better by default. While my notes took Moiron’s point to heart, I still chose Python for the bulk of the backend work of Audacious for now, for that exact reason. There are presently a few services written in Go for critical paths, but for the most part, Go is incredibly unproductive, especially in the context of simple Web service related things, which are, by and large, CRUD with one or two bells or whistles.

→ Women in the room

Jul 03, 2017 20:40:56

Zach Holman makes a very solid point in the wake of the most recent wave of realisations that the tech industry is filled with misogynistic shitheads. The “throw women at the problem” approach to fixing the tech industry simply isn’t the right way about it, and it obviously isn’t working. It has to start somewhere else.

Fundamentally, the assholes who ruin it for everyone else need to either go, or change their behavior in major ways and own up to their past mistakes in a genuine manner. I must say, I’m pretty pleased to see a few of the most gnarly examples I’ve had the chance to interact with get theirs in recent time – including a few who have been smelling their own farts hard in public.

However, while this is all well and good, it shouldn’t be the women’s job to make this happen. There are enough “good guys” in this industry, that we should be able to call these people out and solve it within our own “ranks¹.” Or at least, that’s how it should be – and we should kind of all be embarrassed that this hasn’t been the case. Sure, that gun-wielding founder is mercurial and scary and all, but whatever happened to doing the right thing? The industry as a whole spends so much time talking about “changing the world for the better,” that surely, this must be part of the agenda too.

I mean, the whole industry can’t possibly be this deranged… right?

No, I don’t mean that literally. The point here is: it shouldn’t be the job of women alone to call out misbehaving men. ↩

→ A much needed grain of salt

Jul 17, 2016 11:05:47

I remember reading Malcolm Gladwell’s “Outliers” a few years back and being unable to shake the eerie feeling that the whole premise was a bit too much “pseudo” and not enough “science.” The book is by all accounts an interesting and somewhat thought provoking read on some of the factors that make up a select few “successful people.” But, most conclusions were drawn from Fox News-esque summaries of research, and it left one wondering how heavily influenced the entire book was by selection bias.

Now, I know the book was never made out to be a scientific study of any sorts, but it seems like Gladwell’s persuasiveness got the better of a lot of readers, as I’ve kept on hearing the fabled “10,000 hour rule” – Gladwell’s rule, that if you put in 10,000 hours of practice at something, you’ll be come an expert at it – stated as fact ever since the book came out. As it turns out, it’s nowhere near as simple as that. In fact, earlier this year, a book excerpt written by Anders Ericsson – the main author of the study, “The Role of Deliberate Practice in the Acquisition of Expert Performance,” Gladwell based his rule on – saw the author explain the extent to which Gladwell is incorrect (spoiler alert: Gladwell took a single average and made it a rule.) Especially the deconstruction of Gladwell’s The Beatles example is eye-opening, if you didn’t already balk at the example in the book. But, for a wider audience, I think the distinction of influential practice is probably the most important part of Ericsson’s rebuttal:

This distinction between deliberate practice aimed at a particular goal and generic practice is crucial because not every type of practice leads to the improved ability that we saw in the music students or the ballet dancers.

In other words, even though someone has been doing something for 10 hours a day for 20 years, it does not make them an expert – it’s their approach to it, that does. This is not at all to say that I think Gladwell’s work should be dismissed altogether. It’s merely to say, that I wish people would stop and think about what they read and are told, rather than just accepting it all at face value. For now though, the “10,000 hour rule” serves as a good indicator of when to walk away from a conversation, or counter with such equally fascinating and scientific facts as the 8 spiders a year consumed during sleep average.

The new EU VAT rules are bad for business

Dec 12, 2014 15:33:30

On paper, I mostly think the EU is a great idea. The Euro currency too. It fits well with one of the most loudly broadcasted purposes of EU; to strengthen and easen trade between member countries. Well, it’s not officially phrased that way, but you’d be hard pressed not to interpret the Treaty on European Union that way:

DETERMINED to promote economic and social progress for their peoples, taking into account the principle of sustainable development and within the context of the accomplishment of the internal market and of reinforced cohesion and environmental protection, and to implement policies ensuring that advances in economic integration are accompanied by parallel progress in other fields,

Anyone who runs anything but a local — as in brick and mortar style — business should think this is a great idea. Essentially, it should widen the adressable market for a business with little to no overhead, give or take your sales channels. For Internet companies, this is awesome. Gone are the odd trade agreements between neighbouring European countries, replaced with a simple, consistent model, which everyone agrees on.

For Iconfinder, this has been especially great; we only need to deal with the distinction between EU and non-EU customers, and in the case of EU customers, whether they’re valid legal entities or simply consumers. Based on this information, it’s pretty simple to figure out what to do about VAT. That part about “valid legal entity” is actually not that simple, though, as you are actually required to verify the validity of a legal entity. For this, the EU has a mostly unstable Web service (SOAP, of course), VIES, which is woefully underdocumented, but if you do a bit of work, like we have with our pyvat Python module, you can pretty easily automate this process. It sure as hell still beats dealing with whatever trade agreements existed before the EU.

It’s all down hill from here

But, all these good things are going to come to a screeching halt on January 1st, 2015, at least if you’re in the broadcasting, telecommunications, or, in our case, “electronic services” category of businesses. If you’re so lucky as to be one of these businesses, you now no longer have to pay your local VAT rate to the country in which you operate, when you sell to a consumer in the EU. Instead, you now have to pay VAT in the consumer’s country of residence at the rate applicable to the good, you’re selling, in that country.

In other words, until January 1st, when Iconfinder sells an icon to a consumer in Malta, we have to pay 25 % VAT to the Danish government on the sale. From January 1st, we have to pay 18 % VAT to the Maltesian government instead. This alone, in my opinion, is unnecessarily complex. Think about it; we now have to keep track of VAT rates in all individual EU countries. Sure, the EU published a set of instructions containing a table of these rates, on their 1995 style VAT insanity page, but there’s no obvious way of being subscribed to notifications on changes. Pray that your accountant will keep an eye on that for you, or hope that you remember to visit that site every so often.

But, wait, it gets better. Some countries have varying VAT rates depending on where in the country the consumer lives. Some Greek islands have a discounted VAT rate of 16 % instead of the normal 23 %, so you’re now stuck with having to figure out if that’s the case. As anyone who’s ever tried to parse or validate adresses knows, doing this consistently from user input is going to be borderline impossible. And, then, of course, there’s the varying VAT rates depending on the good you’re selling. In France, e-books are sold at a 5.5 % VAT rate as opposed to the standard 20 % rate, and e-newspapers (whatever they are, exactly) are sold at an even lower rate of 2.1 %. Then, lastly, there are little details like the “use and enjoyment” VAT levying rule, which is of course only applicable in certain countries.

None of this is exposed in any useful manner. It’s 2014 and the EU did not think to make it easy for businesses to comply with these rules by, say, exposing this information through a Web service. Instead, your get a PDF summary and a bunch of documents to read, understand and implement. Have fun.

As for the question of actually paying the VAT owed to ther countries, the EU did actually decide to use some new fangled technology in offering a consolidated platform for payment, the “MOSS,” or “Mini One Stop Shop” (who names these things!?) Knowing how well these things usually work, it’ll probably be utterly useless and a huge headache for the next couple of years, but at least they tried.

Now, as if all of this wasn’t ridiculous enough, the EU laments its self-image of being an economic super power by maintaing the ridiculous rule of countries outside the EU selling to consumers in the EU technically being required to pay VAT in the country where the consumer lives. This is of course in spite of the fact that the rules state, that businesses in the EU doing selling to consumers outside the EU are no longer required to deal with any kind of VAT payments. Fat chance that American companies will start charging me Danish VAT from January 1st, especially when this is now a one-way street.

It’s a treaty breach

The only real upside I can see, is that we’ll improve our margins slightly as only few other countries in the EU have as ridiculous a VAT rate as Denmark. I’m just not sure it’s worth it in time and complexity, not to mention development, bookkeeping and accounting costs.

I honestly think that, if you squint your eyes a little, these new VAT rules are counter to the stated purpose of the EU, and in essense, a breach of the stated goal in the aformentioned treaty. Sure, getting VAT to the country, where a good is consumed makes theoretical sense, as VAT is essentially a consumption based income tax. As so-called electronic services are becoming an ever increasing economic force, this of course skews the balance. However, it’s at the expense of the businesses providing these services. The overhead is substantial, and in that regard raises the barrier to entry for any business wanting to do business with consumers in EU countries — albeit most notably for EU businesses, as the rules can actually be legally enforced here. Sustainable? Maybe — those with skin in the game will manage and adapt. Does it promote economic progress? Definitely not.

The next weeks of my life will be spent implementing a good way to handle these new rules and testing to make sure everything works according to plan. What a royal waste of time, energy and money.

→ Source code optimization

Oct 19, 2014 18:25:25

For some reason, I wound up reading about compilers’ use of vectorization in optimizing code yesterday, and I came across Felix von Leitner‘s pretty interesting presentation from 2009, “Source Code Optimization”.

While I knew that compilers are generally good at optimizing these days, I had no clue just how good they’ve become. This basically also means that a lot of the “general wisdom” I came up with when writing C and C++ no longer really applies, and even more so, that more readable code often seems to produce equally good or even more optimized code than most attempts at being clever. For C(++) code then, this means that we not only can but almost have to mentally shift from being “clever” to being concise when writing code – a development that’s been helped greatly by C11 and C++11.

If you write any C or C++ on a regular basis, I strongly suggest you add this presentation to your list of annual reminders.

→ Smart working

Jul 30, 2014 17:17:10

Startup culture is awash with the heroisation of burning the mythical midnight oil with the 60 or 70 hour work week being a badge of honor — the road to success. Slowly, however, that trend is changing — at least among people who’ve actually done those stupid hours — and while I’m embarrassingly still working way too many hours than is any kind of good, evidence keeps indicating that I really need to stop. Not only evidence in terms of feeling unproductive at times, but actual sort of scientific evidence.

I came across one such piece of evidence today in CamMi Pham‘s post “7 Things You Need To Stop Doing To Be More Productive, Backed By Science”. For the post, she has done a couple of very tangible charts (which seem to correlate well with other data I’ve found) showing the effective productivity of weeks of working 50 and 60 hours respectively. Truth be told, the numbers are scary — if the charts are to be taken at face value, the productivity gain from a 50 or 60 hour work week evaporate after only 9 and 8 weeks respectively:

I probably need to be smart and go home “early.”

→ Apollo 11 launch at 500 frames per second

Jul 21, 2013 04:15:29

As a kid, very few things fascinated me more than space flight (I clearly remember yapping my dad’s ears off about the different stages of the Saturn V launch vehicle’s flight cycle.) To this date, that fascination has stayed with me, and possibly only grown greater as I’ve started to grasp the true extent of the space programmes. Maybe young me would have loved it, but videos like this one by Spacecraft Films from the launch umbilical tower of the Apollo 11 flight from 5 seconds before liftoff at 500 frames per seconds leave me in absolute awe:

A pioneer of augmentation

Jul 04, 2013 11:45:46

It’s with great sadness that I have learned, that a personal hero of mine, Douglas Carl Engelbart has passed await, albeit at a respectable age of 88. However, I’m even more saddened by the way general media is presenting him in obituaries. Lacking rudimentary understanding of Engelbart’s approach and goals, they reduce his intellect, achievements and work to that of a simple inventor of the early personal computing era scoping him as merely “the inventor of the mouse.”

But, as I tried to sit down and figure out how to describe who Engelbart really was, I realized, that the general media is probably well excused. Hell, even as a kid who’s been heads down in computers since he was 8, it wasn’t until a few years ago, where I started to look beyond modern computing as a mere tool, that the seminal nature of Engelbart started to dawn on me. Worse still, it wasn’t until I read Bret Victor’s wonderful piece that the concept of Engelbart became clear (despite the somewhat fuzzy conclusion):

The least important question you can ask about Engelbart is, “What did he build?” By asking that question, you put yourself in a position to admire him, to stand in awe of his achievements, to worship him as a hero. But worship isn’t useful to anyone. Not you, not him.

The most important question you can ask about Engelbart is, “What world was he trying to create?” By asking that question, you put yourself in a position to create that world yourself.

Engelbart wasn’t just another gold rush inventor. He doesn’t fit inside the same boxes as even people as great as Steve Jobs. No, he was nothing less than an amazingly intellectual being who, as one of the first, saw the mainstream availability of computing not as the amazing achievement in itself but rather a means to augment the capabilities of humans. There is no denying that computers truly have transformed the lives of humans and our abilities to collectively solve problems — and probably far more than even young Engelbart could ever have imagined. But, the popular, naive deduction that this is a natural result of the technological development that endured, is nothing short of wrong.

As Bret Victor so concisely puts it; “Engelbart devoted his life to a human problem, with technology falling out as part of a solution.” But, it wasn’t just Engelbart’s technology that “fell off.” A pioneering thought leader as a natural effect of the beautiful purity of the intent of his work, he was and remains both aspirational and inspirational to the technology world as a whole.

His legacy isn’t hypertext, the mouse or any other single piece of work — rather, it’s the total permeation of the technology world with the intent that lead to these almost insignificant pieces. He truly was one of the very few pioneers of augmentation, history has ever known.