Configuration layers development

Emacs Lisp files contain code that can be evaluated. When evaluated, the functions, macros and modes defined in that file become available to the current Emacs session. Henceforth, this will be termed as loading a file.

One major problem is to ensure that all the correct files are loaded, and in the proper order. Another issue is to ensure that not too many files are loaded immediately. This causes startup to take too long. Instead, we want to make sure that files are loaded only as needed, and not all at once.

How is this done in Emacs, and how is it done in Spacemacs?

A better way to load what you need is to use features. A feature is a symbol that typically has the same name as the file it resides in. Let us say you have the following contents in a file called my-feature.el.

;; Your code goes here ...

(provide 'my-feature)

To have Emacs load this file, call require, as such:

(require 'my-feature)

This checks whether the feature my-feature has already been loaded. If not, it looks for a file called my-feature.el, my-feature.elc or some such. If it finds such a file, it will load it. When the call to provide is evaluated, the feature is added to the list of loaded features, so that subsequent calls to require will do nothing.

This will cause an error if no such file can be found.

The file my-feature.el may very well contain other calls to require, and in fact this is quite a common way to ensure that dependencies are loaded before your code runs.

Package authors should use this technique to make sure that dependencies are loaded before their code runs.

Calling require is nothing more than a glorified way of calling load-file. It solves the problem of ensuring that files are loaded in the correct order, and to some degree it solved the problem of where to find the files on disk but a long list of calls to require at startup would still cause Emacs to take forever to load.

Emacs uses auto-loading to solve this problem. When a function is registered as auto-loading, an "empty" definition is provided. When that function is called, the file that provides the function is immediately loaded (along with all its required features). Finally, the "empty" function is substituted with the real one and called normally. The end user will see only a slight delay when first calling the function, while subsequent calls to that function (or any other function loaded as part of the same procedure) will be as quick as normal.

To register a function as auto-loadable, we call autoload:

(autoload 'some-function "some-file")

This instructs Emacs that whenever some-function is called, load some-file.el first, and then proceed.

After evaluating the above code, you can try to inspect some-function by doing SPC h d f some-function. It will say it's an auto-loaded function, and that nothing else is known about it until it is loaded. The call to autoload can optionally include more information, such as a doc-string, whether the function can be called interactively, and so on. This provides more information to the end-user without her having to actually load the file first.

Open your elpa directory, go to helm and look at the file helm-autoloads.el. This provides all the auto-loads for all the files in Helm. However, this file is not written by hand. Instead, it is automatically generated from "magic" comments in the source code of Helm. They look like this:

;;;###autoload
(defun my-function ()
  ;; Source code...
  )

The magic comment ;;;###autoload instructs Emacs that the following definition should be auto-loaded. This automatically generates an appropriate call to autoload.

Things that can be auto-loaded generally involve anything "definable", such as functions, macros, major or minor modes, groups, classes, and so on.

Magic comments also work on other things, such as variable definitions (defvar), but in that case, the definition is just copied verbatim into the auto-loading file. For example, this code will load Helm on startup, long before your file is actually evaluated, probably not what was intended:

;;;###autoload
(require 'helm)

It is the responsibility of the package authors to ensure that their package can be appropriately auto-loaded, and most packages do this quite well.

Spacemacs makes thorough use of auto-loading. Almost everything in Spacemacs is loaded when needed instead of right away.

Often, we will want to configure packages after loading them. We may want to set some variables or call some functions. This is trivial with require, because it loads immediately, but it can be tricky with autoloading, because the configuration code must also be deferred.

Emacs offers with-eval-after-load for this purpose. It can be used like this:

(with-eval-after-load 'helm
     ;; Code
     )

This arranges for the relevant code to be executed after Helm is loaded (using either require or an autoload), or if Helm is already loaded, the code is executed immediately.

Since with-eval-after-load is a macro and not a function, its argument does not have to be quoted.

For end users who are trying to put together an efficient Emacs configuration, there is a very useful package called use-package that provides a macro which is also called use-package which does a very good job of streamlining the whole process of loading packages.

The aspiring layer author is recommended to have a look at the use-package documentation. Some examples follow.

(use-package helm)

This simply loads Helm. It is essentially equivalent to (require 'helm).

(use-package helm
  :defer t)

This defers the loading of Helm using the auto-load facility and the auto-load commands provided by the Helm source code. It is, in fact, a no-op.

(use-package helm
  :defer t
  :init
  ;; Code to execute before Helm is loaded
  :config
  ;; Code to execute after Helm is loaded
  )

This form includes code to execute before and after Helm is loaded. The :init section can be executed immediately, but since Helm is deferred, the :config section is not executed until after loading, if ever. It is essentially equivalent to simply running the :init block, and then adding the :config block in an with-eval-after-load.

(use-package helm
  :commands (helm-find-files helm-M-x))

This creates auto-load references for additional commands, if you find that the package author has been slacking.

(use-package ruby-mode
  :mode "\\.rb\\'")

For packages that provide major modes, you can associate file extensions to that mode by using the :mode keyword. This adds an entry to auto-mode-alist and an auto-load for ruby-mode. Typically this is not required, as ruby-mode should already be auto-loadable, and the package should associate Ruby files with itself already.

Use-package supports heaps of useful keywords. Look at the documentation for more.

This file is the first file to be loaded and this is the place where additional layers can be declared as dependencies.

For instance, if layer A depends on some functionality of layer B, then in the file layers.el of layer A, we can add:

(configuration-layer/declare-layer-dependencies '(B))

The effect is that B is considered a used layer and will be loaded as if it was added to dotspacemacs-configuration-layers variables.

It contains the list of packages of the layer and the actual configuration for the packages included in the layer.

This file is loaded after layers.el.

It must define a variable called <layer>-packages, which should be a list of all the packages that this layer needs. Some valid package specifications are as follows:

(defconst mylayer-packages
  '(
    ;; Get the package from MELPA, ELPA, etc.
    some-package
    (some-package :location elpa)

    ;; A local package
    (some-package :location local)

    ;; A local package to be built with Quelpa
    (some-package :location (recipe :fetcher local))

    ;; A package recipe
    (some-package :location (recipe
                             :fetcher github
                             :repo "some/repo"))

    ;; An excluded package
    (some-package :excluded t)
    ))

The :location attribute specifies where the package may be found. Spacemacs currently supports packages on ELPA compliant repositories, local packages, remote packages hosted on Git repositories (including specific helpers for GitHub, GitLab, and Bitbucket) and MELPA recipes (through the Quelpa package). Local packages should reside at <layer>/local/<package>/.

For information about recipes see the MELPA documentation.

As you may have noticed from examples above, there are two ways to declare a local package: using either :location local or a Quelpa recipe with the Spacemacs-specific pseudo-fetcher local. The former is for the simplest packages that declare no external dependencies, since it just adds the package directory to the load-path. The latter is for packages that do have external dependencies declared and thus have to be built with Quelpa.

Packages may be excluded by setting the :excluded property to true. This will prevent the package from being installed even if it is used by another layer.

For each included package, you may define one or more of the following functions, which are called in order by Spacemacs to initialize the package.

1. <layer>/pre-init-<package>
2. <layer>/init-<package>
3. <layer>/post-init-<package>

It is the responsibility of these functions to load and configure the package in question. Spacemacs will do nothing other than download the package and place it in the load path for you.

Note: A package will not be installed unless at least one layer defines an init function for it. That is to say, in a certain sense, the init function does mandatory setup while the pre-init and post-init functions do optional setup. This can be used for managing cross-layer dependencies, which we will discuss later.

It contains all the defined functions used in the layer.

This file is loaded after packages.el and before config.el.

It is good practice to guard the definition of functions to make sure a package is actually used. For instance:

(when (configuration-layer/package-used-p 'my-package)
  (defun spacemacs/my-package-enable () ...)
  (defun spacemacs/my-package-disable () ...))

By guarding these functions we avoid defining them when the package my-package is not used.

This file configures the layer by declaring layer variables' default values and setting up some other variables related to the layer.

This file is loaded after funcs.el.

It contains general key bindings.

This is the last file loaded.

The word general here means independent of any package. Since the end user can exclude an arbitrary set of packages, you cannot be sure that, just because your layer includes a package, that package will necessarily be loaded. For this reason, code in these files must be generally safe, regardless of which packages are installed.

More on this in the next section.

Spacemacs provides a couple of additional useful functions you can use to check whether other layers or packages are included.

• check if a layer is enabled (configuration-layer/layer-used-p)
• check if a package is or will be installed (configuration-layer/package-used-p)

These are useful in some cases, but usually you can get the desired result just by using post-init functions.

For layers that require another layers to be enabled, use the functions configuration-layer/declare-layer and configuration-layer/declare-layers to ensure that layers are enabled even if the user has not enabled them explicitly. Calls to these functions must go in the layers.el file.

Shadowing is the operation of replacing a used layer by another one. For instance if a used layer A can shadow a used layer B and the layer A is listed after the layer B in the dotfile then the layer A replaces the layer B and it is like only the layer A is being used.

Examples of this mechanism are helm/ivy layers or neotree/treemacs layers.

A layer can shadow other layers by calling in its layers.el file the function configuration-layer/declare-shadow-relation. This function declares a can-shadow relation between all the layers.

can-shadow is a commutative relation, if layer A can shadow layer B then layer B can shadow layer A.

The shadow operator is a binary operator accepting two layer names, it is not commutative and the order of the operands is determined by the order of the layers in the dotfile (like the ownership stealing mechanism).

If :can-shadow property is set explicitly to nil in the dotfile then the layer won't shadow any layer.

For instance to install both ivy and helm layer:

(setq dotspacemacs-configuration-layers
 '(
   ivy
   (helm :can-shadow nil)
   )

note that due to the commutative relation can-shadow the above example can also be written like this (in this case, :can-shadow should be read :can-be-shawdowed):

(setq dotspacemacs-configuration-layers
'(
  (ivy :can-shadow nil)
  helm
  )

We will prefer the first form as it is more intuitive.

In the vast majority of cases, a package init function should do nothing but call to use-package. Again, in the vast majority of cases, all the configuration you need to do should be doable within the :init or :config blocks of such a call.

What goes where? Since :init is executed before load and :config after, these rules of thumb apply.

In :config should be

• Anything that requires the package to be already loaded.
• Anything that takes a long time to run, which would ruin startup performance.

The :init block should contain setup for the entry points to the package. This includes key bindings, if the package should be loaded manually by the user, or hooks, if the package should be loaded upon some event. It is not unusual to have both!

Spacemacs includes a macro for adding more code to the :init or :config blocks of a call to use-package, after the fact. This is useful for pre-init or post-init functions to "inject" code into the use-package call of the init function.

(spacemacs|use-package-add-hook helm
  :pre-init
  ;; Code
  :post-init
  ;; Code
  :pre-config
  ;; Code
  :post-config
  ;; Code
  )

Since a call to use-package may evaluate the :init block immediately, any function that wants to inject code into this block must run before the call to use-package. Further, since this call to use-package typically takes place in the init-<package> function, calls to spacemacs|use-package-add-hook always happen in the pre-init-<package> functions, and not in post-init-<package>.

If you break any of these rules, you should know what you are doing and have a good reason for doing it.