Nix Overlays: Add attribute to “lib” and avoid “infinite recursion error”

Today, I was about to add an attribute to the lib attribute of nixpkgs using a nixpkgs overlay. I thought that this is as straight forward as adding regular packages directly in an overlay. This wasn’t my first time using overlays, but my first time adding something to lib. There was a tiny but ugly pitfalls, that made me create this short blog post.

Let’s look at the following code foo.nix:

# This script can be invoked like this: $ nix-build ./foo.nix
let
  # Overlay that adds functionality to "pkgs.lib".
  overlay = (self: super:
    let
      pkgs = super.pkgs;
      lib = super.lib;
      toPretty = lib.generators.toPretty;
      # Function that takes a value to trace and an expression to return. The
      # semantics is the same as for builtins.trace, but the traced value is
      # prettified before being printed.
      tracePretty = val: ret: (builtins.trace (toPretty { } val) ret);
      # Wraper around tracePretty that traces the given value and returns it.
      tracePrettyVal = val: tracePretty val val;
    in
    {
      # Must be super.lib and not super.pkgs.lib, otherwise: infinite recursion
      lib = lib // {
      	inherit tracePretty tracePrettyVal;
      };
    }
  );
  # pkgs with the applied overlay.
  pkgs = import <nixpkgs> { overlays = [ overlay ]; };
in
# Usage of the overlayed attribute.
pkgs.lib.tracePrettyVal { arr = [ 1 2 3 ]; }

With

overlay = (self: super:
    let
      pkgs = super.pkgs;
      # Attention: super.lib, not super.pkgs.lib!
      lib = super.lib;
      toPretty = lib.generators.toPretty;
      # Function that takes a value to trace and an expression to return. The
      # semantics is the same as for builtins.trace, but the traced value is
      # prettified before being printed.
      tracePretty = val: ret: (builtins.trace (toPretty { } val) ret);
      # Wraper around tracePretty that traces the given value and returns it.
      tracePrettyVal = val: tracePretty val val;
    in
    {
      # Must be super.lib and not super.pkgs.lib, otherwise: infinite recursion.
      # We must merge that with the original lib module, as overlays do not 
      # perform deep merges.
      lib = lib // {
      	inherit tracePretty tracePrettyVal;
      };
    }
  );

I define a basic overlay that adds the functions tracePretty and tracePrettyVal to a lib attribute that is merged with super.lib. We must be very careful that we merge it with super.lib but not super.pkgs.lib. As I occasionally write Nix files like this:

{ pkgs ? import <nixpkgs> {}
, lib ? pkgs.lib
}: ...

I was tricked to think that I have to use super.pkgs.lib. However, actually, super refers to what pkgs is in the example above and super.pkgs.lib in my overlay code refers to pkgs.pkgs.lib in the example above. So, be careful that you merge your library functions with super.lib!

The rest of the code in the beginning of the blog post is just necessary clutter to use a Nix overlay. So, as a summary: Write an overlay that adds something to lib like this:

(self: super: {
  lib = super.lib // {
    foo = "bar";
  };
})

Otherwise, when you accidentally use super.pkgs.lib, you end up with an ugly “infinite recursion error”.

TIL

Today I learned that pkgs = import <nixpkgs> {} also has a pkgs.pkgs sub-attribute, which is a reference to the top-level pkgs. So you can use pkgs.pkgs.pkgs..... I use Nix for a few months now, but I never encountered this. However, the problem above shows that overlays should always add stuff to the top layer.