Know your guarantees, Go edition

I was directed to a thread about a poor soul who started a project in Go, eventually had to hand it off to the community, and discovered that his original source no longer compiled, due to third-party dependencies having changed. Key quote:

Not even the original programmer, with the original files on his original dev machine, can compile the source anymore.

I feel for the guy. Unfortunately, the above quote is not correct. He didn’t have all of the original files. He only had his own.

See, he took dependencies on third-party code he doesn’t control. That’s a fundamental choice to make. It is a fundamental characteristic of his program (and his dev process).

The complaint is that Go does not prevent this. It’s true! Versioned dependencies are not a feature of the base platform.

It’s also a deliberate choice, where the Go authors chose not to implement a feature when they felt that the trade-offs were no good.

One low-level reason they made this choice is to avoid slow compilation and bloated binaries (which are two sides of the same coin). Remember, packages depend on other packages. So Foo might depend on Bar 2.1. Foo might also depend on Baz which in turn depends on Bar 1.9, and on down the tree. So that would mean compiling and linking several copies of nearly identical code.

Depending on several versions of the same package also means knowing which version one is calling, whereby the dependency mess seeps into your source code.

Which leads us to the high-level reasoning behind the Go platform punting on this feature: they did not have a logical solution they considered acceptable. It’s not that they don’t understand the problem; it’s that, at the moment, there is not a solution they like. So they choose no feature over over a regressive one.

It’s a controversial stance. After all, npm and bundler and many other systems have dependency versioning built-in, and people work with them every day. But if you’ve used them, you know they are not without flaws.

I’ll speculate further: perhaps dependency versioning is not unlike “high availability” software. It makes promises of reliability, at the expense of increased complexity and bloat. After all, you are running a lot more code with a lot more relationships.

Often, “high availability” solutions don’t net out to being more reliable, due to this underestimated complexity. Perhaps it’s better to make no guarantee, than to make one that you can’t keep.

This is also something of a cultural experiment on Go’s part. By removing these (possibly) false guarantees, you are forced to be much more deliberate about managing your code and its dependencies. Trust but verify.

It sounds like our poor soul from the opening paragraph trusted, but didn’t verify.

As a practical matter, here’s what he should have done. First, recognize that he was depending on code that is out of his control. Second, make a choice: don’t depend on it, or turn it into code that he does control.

He could fork those repos and depend on his own copy. He could “vendor” them into his solution with something like godep. He could depend on the weak (but not non-existent) promise of tagged versions.

Remember: everything about third-party code is a decision about trust. Waving the wand of “versioned dependencies” doesn’t change that. In fact, it might fool us into seeing guarantees that don’t exist.

discuss on hacker news

The two edges of “culture fit”

Here’s a smart and funny bit about culture fit. I think it’s brutal and by and large true. [1]

When I first heard the term “culture fit”, I thought intuitively, yes. Great way to build a company. Companies are cultures and we should aim to build them.

Then, not long after, I thought, wait. “Culture fit” among highly-educated-20-something-males-on-the-spectrum might not actually make for a great culture, if it’s going to be of any size.

That last bit is key: “culture fit” is probably unavoidable in early days, but rapidly transitions to creepy when your company starts growing.

When you start from scratch, you choose partners. To partner up, by definition, means you probably share a vision. Otherwise, why would you be partners?

In all likelihood, sharing a vision means sharing priors. So your early team will not be diverse. It will likely be a tight group of people whose cultures, um, fit.

(It doesn’t have to be this way. But odds are it will.)

Now, if your company’s goal are to be a tight, focused lifestyle business, great. That’s a good way to go. You don’t need much diversity.

However, if you are funded, your goals are probably a big audience and a big exit. Your ability to reach new and diverse audiences requires diversity in your company. I don’t just mean skin color and gender. I mean, people of varying experiences who see opportunities that you don’t.

You need type-A glad-handing salespeople. You need neckbeard sysadmins. You need touchy-feely designers. You need nagging accountants. They will grow the business. And you won’t get them if they need to be like you.

(If you achieve the above, you’ll get the skin-tone-and-gender diversity for free, btw.)

So, start with culture fit. Then know when to lay off.

 

[1] See also

How a savvy landlord would handle Airbnb

Megan McArdle describes the understandable discomfort that some tenants and property owners have about Airbnb’ers coming into their buildings.

Airbnb will naturally drive up rents in places where it’s being used. If a tenant has the capacity to pay (e.g.) $2000 for an apartment, but can get an additional (e.g.) $300 a month, the tenant’s capacity to pay is now $2300.

A savvy landlord or condo association would want to capture this. Because it’s new, it’s seen as a disturbance. A little bit of thinking would reveal that there is now $XXX new money coming in the door (due to better utilization of the property).

It’s in the owners’ ultimate interest to figure out how to either a) accommodate this explicitly and safely, and capture some of the new revenue or b) prohibit it because the tenants prefer it that way, but understand that the price of a “non-shared” building will carry a premium.

One way a condo association might sell the idea to co-tenants is to point out that the new Airbnb money might be used for building upgrades or to reduce condo fees.

They might set up rules that only Airbnb’ers of a specified high rating may be allowed. They might limit the total number of nights per month. They might upgrade their keying systems.

I’d also point out that Airbnb’ers are likely nothing special, behavior-wise. I have recalcitrant noisy neighbors (across the courtyard out back) who are legitimate, paying leaseholders. There will be a non-zero and manageable amount of problems, but not more than the status quo.

Another way this is nothing new: some single-family homeowners raise an eyebrow at condos; some owners raise looks askance at renters, and down the line. They are trade-offs with pros and cons.

It seems clear there is a net economic gain in sharing real estate. The challenge is for parties to agree on the distribution of that gain, seeing beyond the natural status quo bias.

discuss on hacker news

PS: a commenter on HN (on Megan’s original story) mentions that Fannie & Freddie, the ultimate buyers of many mortgages, wouldn’t countenance this, and this would hamper the whole market. True?

De facto immutability

Another in a series of posts wherein I spitball about things that computer scientists already know, but have only just occurred to me.

Browsing the new Swift docs reveals that immutability is given pretty high billing. There are two keywords for variable declaration/assignment: let and var.

The former declares an immutable value; the latter a mutable one. The former is used much more often in the examples, providing a hint about the intentions of the designers.

The first thing I like about this is that the programmer is encouraged to default to immutable. It’s a nudge which suggests, when using var, that perhaps they should think about why they are doing that. It feels exceptional and just a little unsafe. Could this code be rearranged to avoid it?

It also got me thinking about Go, which doesn’t really have a notion of immutability. Well, it does have const, but it’s only for primitives.

Do compilers have a notion of “de facto immutable”? Which is to say, although the programmer has not intended this value to be immutable, the compiler can prove that, in fact, it is not mutated; that there is no path by which the value can change.

This opens up optimization possibilities. One is that values can be inlined in simple situations.

A second, as pointed out by Kevin Montrose, is that ordering of reads and writes can be skipped for de facto immutable values. (Similarly helpful in concurrent situations, of course.)

A third is that anything passed by value, but known not to be mutated, need not result in a copy of the value. Under the hood, a pointer can be passed safely, reducing work as well as memory/GC pressure. This is analogous to interning strings.

Would be interested to know which languages exploit these ideas, or if there are logical problems with them that I’ve missed.

discuss on hacker news

What Netflix bought from Comcast is a CDN

With the hubbub about net neutrality, fast lanes and Netflix, I though I would illustrate what we are dealing with technologically.

Netflix’s deal with Comcast is called “interconnect” or “peering”. It’s part of the wholesale, or B2B, side of the Internet.

Traditionally, Netflix would buy “transit” from a third-party like Cogent. Cogent, in turn, would make a deal with consumer ISPs such as Comcast, Verizon, Time Warner, et al, and thereby deliver Netflix’s bits to consumers.

cdn

What happened in this instance, however, is that Netflix decided to skip Cogent and connect directly to Comcast (and Verizon), and to pay for the connection. The term for this is “paid interconnect”. We call it this because in many instances, interconnect is done without an exchange of money — those circumstances are called “settlement-free”.

Netflix’s arrangement with Comcast been called a fast lane, and it is. But it is technologically indistinguishable from a CDN, which most of us are familiar with, and which have been around for a long time.

As you can see from the diagram above, a CDN and paid interconnect are the same thing. Bits — Netflix’s video content in this case — are allowed ingress to Comcast’s network. By being on a CDN, or using paid interconnect, the buyer is given a privileged place on Comcast’s network, in exchange for money.

I can understand the upset that people have over the Netflix arrangement. I completely agree that Comcast et al are monopolies, and their behavior reflects that. (I am on Time Warner and can point you to the grey hairs I acquired when dealing with them.)

But I also believe being upset about these monopolies is being conflated with being upset about fast lanes. We have fast lanes, have had for a while, and use them happily in exchange for money.

If we wish to prohibit fast lanes, then we wish to prohibit CDNs. It might be the right thing to do, given the state of local ISP monopolies. But let’s articulate what it is we want.

discuss on hacker news

Updates 11 May

I’ve updated the image above to note that Netflix must also replicate its content to the CDN cache. The replication is not bandwidth-intensive the way consumer delivery is; once Netflix delivers a movie to the CDN it can be delivered to consumers hundreds of times without going back to Netflix’s servers.

Also, the diagram above refers to a CDN where a cache server is on the premises of the last-mile ISP. One can also imagine a third-party CDN off-premises:

cdn3

The basic logic does not change: if there is a high volume of bandwidth being delivered to the last mile, the ingress to Comcast’s network still needs to accommodate; it’s a fast lane with more middlemen.

Middle class in NYC

There is a notion I hear from time to time that earning (say) $100,000 while living in NYC makes one “middle class”. The logic being, NYC is expensive and $100K doesn’t go as far as it would in other places. Or, that many nearby people earn similar or higher incomes, putting one “in the middle”.

The facts are correct but the logic is exactly backwards. One lives in NYC because one has the means to do so. It is evidence that one is among a small number of people that can afford such things.

(A median household [50th percentile] in the US earns around $46K; making $100K is 80th percentile.)

I’ve come to think of NYC, economically, as a country club (especially so in Manhattan and parts of Brooklyn). It’s quite exclusive. And people who join do so because they want to be part of it.

Which is great! I am here to be part of something special, too.

But complaining about the expense of NYC is like complaining about country club dues. Yes, it’s expensive. But the expense, should one choose to pay it, does not imply that one middle class.

The best we can say, perhaps, is that we are average among a very privileged group.

discuss on the hacker news

Update 30 May: A NYT story demonstrating exactly the above.

NYC doesn’t need to be Silicon Valley