A much needed grain of salt

I remember reading Malcolm Gladwell’s “Outliers” a few years back and being unable to shake the eerie feeling that the whole premise was a bit too much “pseudo” and not enough “science.” The book is by all accounts an interesting and somewhat thought provoking read on some of the factors that make up a select few “successful people.” But, most conclusions were drawn from Fox News-esque summaries of research, and it left one wondering how heavily influenced the entire book was by selection bias.

Now, I know the book was never made out to be a scientific study of any sorts, but it seems like Gladwell’s persuasiveness got the better of a lot of readers, as I’ve kept on hearing the fabled “10,000 hour rule” – Gladwell’s rule, that if you put in 10,000 hours of practice at something, you’ll be come an expert at it – stated as fact ever since the book came out. As it turns out, it’s nowhere near as simple as that. In fact, earlier this year, a book excerpt written by Anders Ericsson – the main author of the study, “The Role of Deliberate Practice in the Acquisition of Expert Performance,” Gladwell based his rule on – saw the author explain the extent to which Gladwell is incorrect (spoiler alert: Gladwell took a single average and made it a rule.) Especially the deconstruction of Gladwell’s The Beatles example is eye-opening, if you didn’t already balk at the example in the book. But, for a wider audience, I think the distinction of influential practice is probably the most important part of Ericsson’s rebuttal:

This distinction between deliberate practice aimed at a particular goal and generic practice is crucial because not every type of practice leads to the improved ability that we saw in the music students or the ballet dancers.

In other words, even though someone has been doing something for 10 hours a day for 20 years, it does not make them an expert – it’s their approach to it, that does. This is not at all to say that I think Gladwell’s work should be dismissed altogether. It’s merely to say, that I wish people would stop and think about what they read and are told, rather than just accepting it all at face value. For now though, the “10,000 hour rule” serves as a good indicator of when to walk away from a conversation, or counter with such equally fascinating and scientific facts as the 8 spiders a year consumed during sleep average.

July 17, 2016 |

What’s in an IPO?

2014 saw the tech IPO market back in full swing with Alibaba, ZenDesk, GrubHub and GoPro among the most notable. Mike Evans of GrubHub gives an enlightening glimpse into the process of actually IPO’ing a company. As is to be expected, there’s a not insignificant amount of red tape and “optics” involved. The only somewhat surprising part to me was the amount of effort actually put in by the investment banks:

The best banks had company on-brand presentations that showed they understood the drivers and culture of the business. They had done lots of research including calling our customers to ask about why they liked the product. They even had video of customer perspectives on the business. I should specifically call out that Citi was head and shoulders above the rest, and clearly wanted our business. It proved to be a good choice because they worked very hard.

I’ve seen investors making silly $5 million bets with much higher risks put in less effort than that.

March 4, 2015 |

The new EU VAT rules are bad for business

On paper, I mostly think the EU is a great idea. The Euro currency too. It fits well with one of the most loudly broadcasted purposes of EU; to strengthen and easen trade between member countries. Well, it’s not officially phrased that way, but you’d be hard pressed not to interpret the Treaty on European Union that way:

DETERMINED to promote economic and social progress for their peoples, taking into account the principle of sustainable development and within the context of the accomplishment of the internal market and of reinforced cohesion and environmental protection, and to implement policies ensuring that advances in economic integration are accompanied by parallel progress in other fields,

Anyone who runs anything but a local — as in brick and mortar style — business should think this is a great idea. Essentially, it should widen the adressable market for a business with little to no overhead, give or take your sales channels. For Internet companies, this is awesome. Gone are the odd trade agreements between neighbouring European countries, replaced with a simple, consistent model, which everyone agrees on.

For Iconfinder, this has been especially great; we only need to deal with the distinction between EU and non-EU customers, and in the case of EU customers, whether they’re valid legal entities or simply consumers. Based on this information, it’s pretty simple to figure out what to do about VAT. That part about “valid legal entity” is actually not that simple, though, as you are actually required to verify the validity of a legal entity. For this, the EU has a mostly unstable Web service (SOAP, of course), VIES, which is woefully underdocumented, but if you do a bit of work, like we have with our pyvat Python module, you can pretty easily automate this process. It sure as hell still beats dealing with whatever trade agreements existed before the EU.

It’s all down hill from here

But, all these good things are going to come to a screeching halt on January 1st, 2015, at least if you’re in the broadcasting, telecommunications, or, in our case, “electronic services” category of businesses. If you’re so lucky as to be one of these businesses, you now no longer have to pay your local VAT rate to the country in which you operate, when you sell to a consumer in the EU. Instead, you now have to pay VAT in the consumer’s country of residence at the rate applicable to the good, you’re selling, in that country.

In other words, until January 1st, when Iconfinder sells an icon to a consumer in Malta, we have to pay 25 % VAT to the Danish government on the sale. From January 1st, we have to pay 18 % VAT to the Maltesian government instead. This alone, in my opinion, is unnecessarily complex. Think about it; we now have to keep track of VAT rates in all individual EU countries. Sure, the EU published a set of instructions containing a table of these rates, on their 1995 style VAT insanity page, but there’s no obvious way of being subscribed to notifications on changes. Pray that your accountant will keep an eye on that for you, or hope that you remember to visit that site every so often.

But, wait, it gets better. Some countries have varying VAT rates depending on where in the country the consumer lives. Some Greek islands have a discounted VAT rate of 16 % instead of the normal 23 %, so you’re now stuck with having to figure out if that’s the case. As anyone who’s ever tried to parse or validate adresses knows, doing this consistently from user input is going to be borderline impossible. And, then, of course, there’s the varying VAT rates depending on the good you’re selling. In France, e-books are sold at a 5.5 % VAT rate as opposed to the standard 20 % rate, and e-newspapers (whatever they are, exactly) are sold at an even lower rate of 2.1 %. Then, lastly, there are little details like the “use and enjoyment” VAT levying rule, which is of course only applicable in certain countries.

None of this is exposed in any useful manner. It’s 2014 and the EU did not think to make it easy for businesses to comply with these rules by, say, exposing this information through a Web service. Instead, your get a PDF summary and a bunch of documents to read, understand and implement. Have fun.

As for the question of actually paying the VAT owed to ther countries, the EU did actually decide to use some new fangled technology in offering a consolidated platform for payment, the “MOSS,” or “Mini One Stop Shop” (who names these things!?) Knowing how well these things usually work, it’ll probably be utterly useless and a huge headache for the next couple of years, but at least they tried.

Now, as if all of this wasn’t ridiculous enough, the EU laments its self-image of being an economic super power by maintaing the ridiculous rule of countries outside the EU selling to consumers in the EU technically being required to pay VAT in the country where the consumer lives. This is of course in spite of the fact that the rules state, that businesses in the EU doing selling to consumers outside the EU are no longer required to deal with any kind of VAT payments. Fat chance that American companies will start charging me Danish VAT from January 1st, especially when this is now a one-way street.

It’s a treaty breach

The only real upside I can see, is that we’ll improve our margins slightly as only few other countries in the EU have as ridiculous a VAT rate as Denmark. I’m just not sure it’s worth it in time and complexity, not to mention development, bookkeeping and accounting costs.

I honestly think that, if you squint your eyes a little, these new VAT rules are counter to the stated purpose of the EU, and in essense, a breach of the stated goal in the aformentioned treaty. Sure, getting VAT to the country, where a good is consumed makes theoretical sense, as VAT is essentially a consumption based income tax. As so-called electronic services are becoming an ever increasing economic force, this of course skews the balance. However, it’s at the expense of the businesses providing these services. The overhead is substantial, and in that regard raises the barrier to entry for any business wanting to do business with consumers in EU countries — albeit most notably for EU businesses, as the rules can actually be legally enforced here. Sustainable? Maybe — those with skin in the game will manage and adapt. Does it promote economic progress? Definitely not.

The next weeks of my life will be spent implementing a good way to handle these new rules and testing to make sure everything works according to plan. What a royal waste of time, energy and money.

December 12, 2014 |

Tracking down a rogue kernel_task

I’m one of the lucky ones who have the pleasure of working on a 2014 Mac Pro (yes, the trash can) every day. I love it to bits. It’s blazingly fast, never misses a beat and no matter what I feel like doing, resources are never a constraint. However, my particular Mac Pro has been a bit wonky from the start. Ever so often (somewhere between once a week and three times a day), the kernel will start spinning out of control on one of the cores. As a side effect, applications will start hanging, and the system basically becomes useless until I force a reboot (once the kernel finds itself in an endless loop, a graceful shutdown goes out the window — the best you can do is try to issue a sync to make sure the file system will come back up in a reasonable state, but with HFS+, you never really know.)

The dreaded kernel_task at 100 % CPU usage

The usual remedies haven’t helped. Reset this, reset that, full reinstall and so on. For the longest time, then, I’ve just ignored the issue. It wasn’t sufficiently annoying to warrant working from my laptop for however long it would take to fix the issue. However, after a series of all too frequent incidents in the last couple of weeks, I decided to take affair. Given that I haven’t been able to isolate the trigger of the problem, I decided to try and gather as much data as I could, as searches on the matter of kernel_task running at 100 % yielded so many different explanations, none of which seemed directly applicable to my situation. Giving Apple’s support a call and saying “I occasionally get kernel_task stuck on a core” would probably not be of much use to anyone, then. What I needed to do, was to figure out exactly what was going on in the kernel at these times.

Luckily, Mac OS X ships with every developer’s wet dream, dtrace. Even better, Brendan Gregg made a simple Perl script back in 2006 to sample calls in the kernel using dtrace, which almost works today. Given the “almost” part, and the fact that I like more readable output for debugging, I decided to whip up a quick improvement in Go, osxkerneltracer, which I’ve been using for the past couple of weeks to gather evidence whenever this problem occurred.

On my machine in its normal state, the output looks something like this, after sampling for 5 seconds:

Module            | Method                                                 | Calls |      Share
------------------+--------------------------------------------------------+-------+-----------
kernel            | machine_idle+0x1fd                                     | 49756 |  98.4819 %
kernel            | 0xffffff801cd52880+0x313                               |   240 |   0.4750 %
kernel            | processor_idle+0xc5                                    |   101 |   0.1999 %
kernel            | processor_idle+0xb4                                    |    41 |   0.0812 %
kernel            | do_mfence+0x3                                          |    27 |   0.0534 %
kernel            | dcache_incoherent_io_store64+0xab                      |    23 |   0.0455 %
kernel            | ipc_mqueue_post+0x227                                  |    18 |   0.0356 %
kernel            | machine_idle+0x1ff                                     |    18 |   0.0356 %
kernel            | 0xffffff801cd52880+0x3bf                               |    17 |   0.0336 %
kernel            | processor_idle+0xd9                                    |    10 |   0.0198 %
kernel            | processor_idle+0xab                                    |     7 |   0.0139 %
kernel            | user_trap+0x9f                                         |     7 |   0.0139 %
kernel            | wait_queue_wakeup_all+0xb9                             |     7 |   0.0139 %
AMD7000Controller | _OSReadInt32(void const volatile*, unsigned long)+0x1a |     6 |   0.0119 %
kernel            | thread_block_reason+0xbe                               |     6 |   0.0119 %
kernel            | processor_idle+0xbb                                    |     5 |   0.0099 %
kernel            | 0xffffff801cd6acc0+0x2d0                               |     4 |   0.0079 %
kernel            | __bzero+0xb                                            |     4 |   0.0079 %
[...]

This is basically what you’d expect from a system at almost idle; most of the time in the kernel is spent in an idle function — in this case, the kernel’s machine_idle function (as a side note, for a kick, read up on how this actually works.) When things go awry, it’s always the same thing that happens. That innocent 32 bit read function to the AMD 7000 controller you can see in the call trace above suddenly represents roughly 6.25 % of the calls traced — the full share of one of the 16 hyper-threaded cores on this machine. In other words, the kernel seems stuck in an endless loop trying to read data from the graphics controller.

With this data in hand, I’ve now contacted Apple support, who immediately bumped the issue to a guy who understood what not only a kernel is, but also knew that dtrace existed (in other words: data expedites support when dealing with intermittent issues.) I agreed to ship a bunch of logs for them to look at before taking the next step of actually taking the machine in, and I’m now waiting to see what they come back with.

If you’re experiencing similar, intermittent issues, I suggest you do the same thing. It’ll save you and probably also the immediate support staff a lot of time fumbling in the dark and will much more easily get your issue bumped to the right level, if it’s actually Apple’s problem.

November 11, 2014 |

Source code optimization

For some reason, I wound up reading about compilers’ use of vectorization in optimizing code yesterday, and I came across Felix von Leitner‘s pretty interesting presentation from 2009, “Source Code Optimization”.

While I knew that compilers are generally good at optimizing these days, I had no clue just how good they’ve become. This basically also means that a lot of the “general wisdom” I came up with when writing C and C++ no longer really applies, and even more so, that more readable code often seems to produce equally good or even more optimized code than most attempts at being clever. For C(++) code then, this means that we not only can but almost have to mentally shift from being “clever” to being concise when writing code – a development that’s been helped greatly by C11 and C++11.

If you write any C or C++ on a regular basis, I strongly suggest you add this presentation to your list of annual reminders.

October 19, 2014 |