The Problem with Curation

Recently I received a question from a user asking about "cabal hell" when installing one of my packages. The scenario in question worked fine for us, but for some reason it wasn't working for the user. When users report problems like this they usually do not provide enough information for us to solve it. So then we begin the sometimes arduous back and forth process of gathering the information we need to diagnose the problem and suggest a workaround or implement a fix.

In this particular case luck was on our side and the user's second message just happened to include the key piece of information. The problem in this case was that they were using stackage instead of the normal hackage build that people usually use. Using stackage locks down your dependency bounds to a single version. The user reporting the problem was trying to add additional dependencies to his project and those dependencies required different versions. Stackage was taking away degrees of freedom from the dependency solver (demoting it from the driver seat to the passenger seat). Fortunately in this case the fix was simple: stop freezing down versions with stackage. As soon as the user did that it worked fine.

This highlights the core problem with package curation: it is based on a closed-world assumption. I think that this makes it not a viable answer to the general question of how to solve the package dependency problem. The world that many users will encounter is not closed. People are constantly creating new packages. Curation resources are finite and trying to keep up with the world is a losing battle. Also, even if we had infinite curation resources and zero delay between the creation of a package and its inclusion in the curated repository, that would still not be good enough. There are many people working with code that is not public and therefore cannot be curated. We need a more general solution to the problem that doesn't require a curator.

Comments

kanishka said…
I have been trying to read more on the different sides of the curation vs solver debate (in Haskell and beyond). I don't have a strong opinion yet. In response to your point about closed source, I think most closed source projects would be willing to submit the build-depends section of their projects to a public curator. That leaves the question about finite resources not being able to curate reasonably fast. I am curious about various opinions on that point.
April 1, 2017 at 11:57 PM
Post a Comment

Popular posts from this blog

A Hopefully Fair and Useful Comparison of Haskell Web Frameworks

Recently there has been a lot of discussion and questions about the differences between the big three Haskell web frameworks: Happstack, Yesod, and Snap. Different pieces of the answer have been discussed in a number of places. In this post, I'd like to try to give a more complete comparison. Hopefully it will be relatively unbiased, but without being so watered down that it is useless. I've succeeded if you can't tell which framework I'm a major contributor to based solely on the text of this post. Happstack First, up is Happstack , the oldest of the three with original commit history as early as February, 2005. Happstack used to be called HAppS . It was primarily composed of a web server based on lazy IO, a fairly radical (for the time) in-memory persistence approach for storing and versioning native Haskell data types, and a library called IxSet for making it easier to use this state system with relational data access patterns. I've heard a number of pe...
Read more

Setting Up A Private Nix Cache

I recently went through the process of setting up a private Nix binary cache. It was not obvious to me how to go about it at first, so I thought I would document what I did here. There are a few different ways of going about this that might be appropriate in one situation or another, but I’ll just describe the one I ended up using. I need to serve a cache for proprietary code, so I ended up using a cache served via SSH. Setting up the server For my cache server I’m using an Amazon EC2 instance with NixOS. It’s pretty easy to create these using the public NixOS 18.03 AMI. I ended up using a t2.medium with 1 TB of storage, but the jury is still out on the ideal tradeoff of specs and cost for our purposes. YMMV. The NixOS AMI puts the SSH credentials in the root user, so log in like this: ssh -i /path/to/your/key.pem root@nixcache.example.com To get your new NixOS machine working as an SSH binary cache there are two things you need to do: generate a signing key and turn on cache serv...
Read more

Dependent Types are a Runtime Maybe

Awhile back I was discussing dependent types with someone and we ended up concluding that dependent types can always be replaced by a runtime Maybe.  This seemed to me then (and still does today) as a fairly surprising and provocative conclusion.  So I thought I'd put the idea out there and see what people think. Let's look at a few examples that are commonly used to illustrate dependent types: Vectors of length N Matrices of size m x n Sorted lists Height-balanced trees (trees where the height of subtrees differ by at most one) The argument is roughly as follows.  All of these examples ultimately boil down to enforcing some kind of constraint on some data types.  The more powerful your dependent type system, the more rich and expressive will be the constraints that you can enforce.  If we take this thought experiment to its logical conclusion, we end up with a dependent type system that allows us to enforce any constraint that can be computed. The important rea...
Read more