Introduction

chezmoi_modify_manager is an addon for chezmoi that deals with settings files that contain a mix of settings and state. So far handling INI-style files are supported.

A typical example of this is KDE settings files. These contain (apart from settings) state like recently opened files and positions of windows and dialog boxes. Other programs (such as PrusaSlicer) also do the same thing.

chezmoi_modify_manager allows you to ignore certain sections of those INI files when managing the configuration files with chezmoi.

Note! This documentation reflects the latest release and may not match older versions or newer in-development versions.

Features

  • Ignore entire sections or specific keys in an INI style file.
  • Ignore a key in a section based on regular expressions.
  • Force set a value (useful together with templating).
  • Force remove a section, key or entries matching a regex (useful together with templating).
  • Apply a transformation to the value of a specified key. These are special operations that are built in and provide more complicated transformations. Some examples that this can do:
    • Look up a password in the platform keyring
    • Ignore the sorting order of a list style value (key=a,b,c,d)
    • etc.
  • Assisted adding/updating of files in your chezmoi source state.
  • Optional built in self-updater

Installation & upgrades

It is assumed you already have chezmoi set up and understand the basics of how it works.

  1. To your root .chezmoiignore add: **/*.src.ini. These files should not be checked out into your target directory, but acts as the "source of truth" for the modify script.
  2. Do one of these:
    • Recommended: Install chezmoi_modify_manager into your $PATH. This can be done by one of (in descending order of preference):
      • Using a distro package (if available for what you use)
      • Download the binary from the releases on GitHub and install it somewhere into your PATH.
      • Install from [crates.io] using cargo (only do this if you know what you are doing).
    • Not recommended: Install chezmoi_modify_manager from the releases page into <chezmoi-source-directory>/.utils/chezmoi_modify_manager-<os>-<arch> where <os> is typically linux and <arch> is typically x86-64. If you use another path, the template modify script that is added will be wrong.
  3. Run chezmoi_modify_manager --doctor and make sure it reports no major issues with your installation.

Tab completion

Optionally you can install tab completion. The tab completion can be generated using the hidden command line flag --bpaf-complete-style-SHELL_NAME, (e.g. --bpaf-complete-style-zsh, --bpaf-complete-style-bash, ...). As this is handled internally by the command line parsing library we use, please see their documentation for detailed instructions.

For the Arch Linux AUR package, the completions are already installed for you (except for elvish, which doesn't support a global install).

Upgrading

Depending on the installation method:

  • chezmoi_modify_manager --upgrade
  • With your package manager
  • For each OS and architecture, update the file .utils/chezmoi_modify_manager-<os>-<arch>. Note! For executables that you can run (i.e. the native one) you can still use --upgrade to do this.

You are in control of updates. Nothing will happen unless you pass --upgrade. Consider subscribing to be notified of new releases on the GitHub repository. This can be done via Watch -> Custom in the top right corner on that page after logging in to GitHub. Or just remember to check with --upgrade occasionally.

Basic Usage

Theory of operation

For each settings file you want to manage with chezmoi_modify_manager there will be two files in your chezmoi source directory:

  • modify_<config file> or modify_<config file>.tmpl, e.g. modify_private_kdeglobals.tmpl
    This is the modify script/configuration file that calls chezmoi_modify_manager. It contains the directives describing what to ignore.
  • <config file>.src.ini, e.g. private_kdeglobals.src.ini
    This is the source state of the INI file.

The modify_ script is responsible for generating the new state of the file given the current state in your home directory. The modify_ script is set up to use chezmoi_modify_manager as an interpreter to do so. chezmoi_modify_manager will read the modify script to read configuration and the .src.ini file and by default will apply that file exactly (ignoring blank lines and comments).

However, by giving additional directives to chezmoi_modify_manager in the modify_ script you can tell it to ignore certain sections (see chezmoi_modify_manager --help-syntax for details). For example:

ignore "KFileDialog Settings" "Show Inline Previews"
ignore section "DirSelect Dialog"

will tell it to ignore the key Show Inline Previews in the section KFileDialog Settings and the entire section DirSelect Dialog. More on this below.

Adding files

Always refer to chezmoi_modify_manager --help for the most up-to-date details that matches the version you are using.

There are two modes to add files in:

  • -s/--smart-add: Smart re-add mode that re-adds files as managed .src.ini if they are already managed, otherwise adds with plain chezmoi.
  • -a/--add: This adds or converts from plain chezmoi to managed .src.ini.

Here are some examples:

# Add configs to be handled by chezmoi_modify_manager (or convert configs
# managed by chezmoi to be managed by chezmoi_modify_manager).
chezmoi_modify_manager --add ~/.config/kdeglobals ~/.config/kwinrc

# Re-add config after changes in the live system.
chezmoi_modify_manager --add ~/.config/kdeglobals

# Don't remember if chezmoi_modify_manager handles the file or if it is raw chezmoi?
# Use smart mode (-s/--smart-add) to update the file!
chezmoi_modify_manager --smart-add ~/.config/PrusaSlicer/PrusaSlicer.ini

In addition, you can control readding behaviour with some settings in the modify_<config file>, to filter out entries while readding. This is covered in the next chapter.

Configuring filters

The file modify_<config file> (or modify_<config file>.tmpl if you wish to use chezmoi templating) contain the control directives for that config file which controls the behaviour for chezmoi apply of those files (as well as when readding files from the system). The full details on this file are in the next chapter, this section just covers the basics.

A basic such file will have this structure:

#!/usr/bin/env chezmoi_modify_manager

source auto

# This is a comment
# The next line is a directive. Directives are delimited by newlines.
ignore "SomeSection" "SomeKey"

ignore section "An entire section that is ignored"

ignore regex "Some Sections .*" "A key regex .*"

This illustrates some basics:

  • The first line needs to be a #! that tells the OS that chezmoi_modify_manager should be the interpreter for this file. (This still works on Windows because chezmoi handles that internally as far as I understand, though I don't use Windows myself.)
  • The source directive tells chezmoi_modify_manager where to look for the .src.ini file. As of chezmoi 2.46.1 this can be auto-detected. If you use an older version, chezmoi_modify_manager --add will detect that and insert the appropriate template based line instead.
  • The ignore directive is the most important directive. It has two effects:
    • When running chezmoi apply it results in the matching entries from .src.ini being ignored, and the current system state is used instead.
    • When running chezmoi_modify_manager --add (or --smart-add) it results in not copying matching entries to the .src.ini to begin with.

There are several other directives as well, here is a basic rundown of them, they are covered in more detail in the next chapter. Here is a short summary:

  • set: Sets an entry to a specific value. Useful together with chezmoi templating.
  • remove: Remove a specific entry, also useful together with chezmoi templating.
  • transform: Apply a custom transformation to the value. Can be used to handle some hard to deal with corner cases, supported transforms are covered in a later chapter.
  • add:hide & add:remove: Useful together with certain transforms to control re-adding behaviour.
  • no-warn-multiple-key-matches: If there are multiple regex rules that overlap a warning will be issued. You can use this directive to quieten those warnings if this is intentional. See action evaluation order for more information on this.

Syntax of configuration files

chezmoi_modify_manager uses basic configuration files to control how to merge INI files. These are the modify_<config_file_name> files. They can also be templated with chezmoi by naming the file modify_<config_file_name>.tmpl instead. The easiest way to get started is to use -a to add a file and generate a skeleton configuration file.

Syntax

The file consists of directives, one per line. Comments are supported by prefixing a line with #. Comments are only supported at the start of lines.

Note! If a key appears before the first section, use <NO_SECTION> as the section.

Note! The modify script can itself be a chezmoi template (if it ends with .tmpl), which can be useful if you want to do host specific configuration using the set directive for example.

This however will slow things down every so slightly as chezmoi has to run its templating engine on the file. Typically, this will be an overhead of about half a millisecond per templated modify script (measured on an AMD Ryzen 5 5600X).

Directives

source

This directive is required. It specifies where to find the source file (i.e. the file in the dotfile repo). It should have the following format to support Chezmoi versions older than v2.46.1:

source "{{ .chezmoi.sourceDir }}/{{ .chezmoi.sourceFile | trimSuffix ".tmpl" | replace "modify_" "" }}.src.ini"

From Chezmoi v2.46.1 and forward the following also works instead:

source auto

ignore

Ignore a certain line, always taking it from the target file (i.e. file in your home directory), instead of the source state. The following variants are supported:

ignore section "my-section"
ignore "my-section" "my-key"
ignore regex "section.*regex" "key regex.*"
  • The first form ignores a whole section (exact literal match).
  • The second form ignores a specific key (exact literal match).
  • The third form uses a regex to ignore a specific key.

Prefer the exact literal match variants where possible, they will be marginally faster.

An additional effect is that lines that are missing in the source state will not be deleted if they are ignored.

Finally, ignored lines will not be added back when using --add or --smart-add, in order to reduce git diffs.

set

Set an entry to a specific value. This is primarily useful together with chezmoi templates, allowing you to override a specific value for only some of your computers. The following variants are supported:

set "section" "key" "value"
set "section" "key" "value" separator="="

By default, separator is " = ", which might not match what the program that the ini files belongs to uses.

Notes:

  • Only exact literal matches are supported.
  • It works better if the line exists in the source & target state, otherwise it is likely the line will get formatted weirdly (which will often be changed by the program the INI file belongs to).

remove

Unconditionally remove everything matching the directive. This is primarily useful together with chezmoi templates, allowing you to remove a specific key or section for only some of your computers. The following variants are supported:

remove section "my-section"
remove "my-section" "my-key"
remove regex "section.*regex" "key regex.*"

(Matching works identically to ignore, see above for more details.)

transform

Some specific situations need more complicated merging that a simple ignore. For those situations you can use transforms. Supported variants are:

transform "section" "key" transform-name arg1="value" arg2="value" ...
transform regex "section-regex.*" "key-regex.*" transform-name arg1="value" ...

(Matching works identically to ignore except matching entire sections is not supported. See above for more details.)

For example, to treat mykey in mysection as an unsorted comma separated list, you could use:

transform "mysection" "mykey" unsorted-list separator=","

The full list of supported transforms, and how to use them can be listed using --help-transforms.

add:remove & add:hide

These two directives control the behaviour when using --add or --smart-add. In particular, these allow filtering lines that will be added back to the source state.

add:remove will remove the matching lines entirely. The following forms are supported:

add:remove section "section name"
add:remove "section name" "key"
add:remove regex  "section-regex.*" "key-regex.*"

(Matching works identically to ignore, see above for more details.)

add:hide will instead keep the entries but replace the value associated with those keys. This is useful together with the keyring transform in particular, as the key needs to exist in the source or target state for it to trigger the replacement. The following forms are supported:

add:hide section "section name"
add:hide "section name" "key"
add:hide regex  "section-regex.*" "key-regex.*"

(Matching works identically to ignore, see above for more details.)

no-warn-multiple-key-matches

This directive quietens warnings on multiple regular expressions matching the same section+key. While the warning is generally useful, sometimes you might actually "know what you are doing" and want to suppress it.

Transforms

This is a list of supported transforms. These are used to support some special hard-to-handle cases. The general syntax is documented elsewhere, but in short:

transform "section" "key" transform-name arg1="value" arg2="value" ...
transform regex "section-regex.*" "key-regex.*" transform-name arg1="value" ...

For example:

transform "mysection" "mykey" unsorted-list separator=","

Below is a list of supported transforms, but remember to check chezmoi_modify_manager --help-transforms for the most up-to-date list.

unsorted-list

Compare the value as an unsorted list. Useful because Konversation likes to reorder lists.

Arguments:

  • separator=",": Separating character between list elements

kde-shortcut

Specialised transform to handle KDE changing certain global shortcuts back and forth between formats like:

playmedia=none,,Play media playback
playmedia=none,none,Play media playback

No arguments.

keyring

Get the value for a key from the system keyring. Useful for passwords etc that you do not want in your dotfiles repo.

Arguments:

  • service="service-name": Service name to find entry in the keyring.
  • user="user-name": Username to find entry in the keyring.

You can add an entry to the secret store for your platform with:

chezmoi_modify_manager --keyring-set service-name user-name

Examples

This chapter has examples of how to configure chezmoi_modify_manager for several different programs, as well as some general examples on more advanced topics.

Examples: Ignores & transforms

Here are some useful examples of flags for various settings files I have come across.

KDE

dolphinrc

ignore section "MainWindow"
ignore section "KPropertiesDialog"
ignore "General" "ViewPropsTimestamp"
ignore "Open-with settings" "History"

kdeglobals

ignore "General" "ColorSchemeHash"
ignore "KFileDialog Settings" "Show hidden files"
ignore "KFileDialog Settings" "Show Inline Previews"
ignore section "DirSelect Dialog"

kglobalshortcutsrc

There are two issues in this configuration.

First, ActivityManager switch-to-activity entries. There are multiple entries, making it a perfect fit for a regular expression. Note that this is not state per se. It does however seem to vary between computers, having different UUID values.

Second, certain shortcut keys like flipping between two representations. A specialised transform has been added to handle this case. When this is needed you will see diffs like the following:

-playmedia=none,,Play media playback
+playmedia=none,none,Play media playback

In summary, the following seems to work well:

# The two regex below have overlapping matches, this is OK in this case so
# turn off the warning for this file.
no-warn-multiple-key-matches

ignore regex "ActivityManager" "switch-to-activity-.*"
transform regex ".*" ".*" kde-shortcut

konversationrc

Konversation has two relevant quirks:

  1. It saves the password in the settings file (instead of using kwallet)
  2. It resorts it alias list every time.
ignore "ServerListDialog" "Size"
transform "Aliases" "AliasList" unsorted-list separator=","
transform "Identity 0" "Password" keyring service="konversation" user="konversation_id0"
# Make sure the password isn't added back into the config file on re-add
add:hide "Identity 0" "Password"

To store the password for Identity 0 in your keyring of choice you can use:

$ chezmoi_modify_manager --keyring-set konversation konversation_id0
[Enter your password at the prompt]

kwinrc

Similar to kglobalshortcutsrc there are computer specific UUIDs. In addition, the tiling configurations seem to be overwritten by KDE Plasma between computers.

ignore regex "Desktops" "Id_.*"
ignore regex "Tiling\\]\\[.*" ".*"

plasmanotifyrc

ignore section "DoNotDisturb"

Trolltech.conf

This is a Qt config, rather than a KDE config (strictly speaking) but since KDE uses Qt, it is sitll relevant.

ignore "Qt" "filedialog"

PrusaSlicer / SuperSlicer

PrusaSlicer and the fork SuperSlicer also use INI style files:

PrusaSlicer.ini / SuperSlicer.ini

ignore "<NO_SECTION>" "auto_toolbar_size"
ignore "<NO_SECTION>" "downloader_url_registered"
ignore "<NO_SECTION>" "freecad_path"
ignore "<NO_SECTION>" "last_output_path_removable"
ignore "<NO_SECTION>" "last_output_path"
ignore "<NO_SECTION>" "version_online_seen"
ignore "<NO_SECTION>" "version_online"
ignore "<NO_SECTION>" "version_system_info_sent"
ignore "<NO_SECTION>" "version"
ignore "<NO_SECTION>" "window_mainframe"
ignore "font" "active_font"
ignore "presets" "filament"
ignore "presets" "print"
ignore "presets" "printer"
ignore "presets" "sla_material"
ignore "presets" "sla_print"
ignore regex "<NO_SECTION>" "desktop_integration_.*"
ignore regex "<NO_SECTION>" "print_host_queue_dialog_.*"
ignore regex "font:.*" ".*"
ignore regex "presets" "filament_.*"
ignore section "recent_projects"
ignore section "recent"

PrusaSlicerGcodeViewer.ini / SuperSlicerGcodeViewer.ini

ignore "<NO_SECTION>" "version"
ignore "<NO_SECTION>" "window_mainframe"
ignore section "recent_projects"

PrusaSlicer physical printer settings

PrusaSlicer allows you to configure "physical printers" (with connection details to e.g. OctoPrint or PrusaLink). There will be one such config per physical printer you configured, located at .config/PrusaSlicer/physical_printer/<my_printer_name>.ini

As these contain login details you probably want to put that in your keyring instead of in git. This works similarly to konversation.

For example, you might use the following if you have a Prusa Mk3.9:

transform "<NO_SECTION>" "printhost_password" keyring service="chezmoi_modify_manager" user="prusa_mk39_password" separator=" = "
transform "<NO_SECTION>" "printhost_apikey" keyring service="chezmoi_modify_manager" user="prusa_mk39_apikey" separator=" = "
add:hide "<NO_SECTION>" "printhost_password"
add:hide "<NO_SECTION>" "printhost_apikey"

To add your password and API key you would then use:

chezmoi_modify_manager --keyring-set chezmoi_modify_manager prusa_mk39_password
Password: [Enter password]
chezmoi_modify_manager --keyring-set chezmoi_modify_manager prusa_mk39_apikey
Password: [Enter the API key]

KeePassXC

keepassxc.ini

KeePassXC stores private and public keys for KeeShare in the config. You may not want to commit this to the repository.

ignore "KeeShare" "Active"
ignore "KeeShare" "Foreign"
ignore "KeeShare" "Own"

GTK-3.0/GTK-4.0

settings.ini

The file ~/.config/gtk-<version>/settings.ini has a DPI value in it that changes between computers. Thus, each of those setting files need the following:

ignore "Settings" "gtk-xft-dpi"

Advanced examples: set, remove, add:*

set/remove

The set and remove directives are meant to be used together with templating in the modify scripts. For example, there might be a key binding in KDE you only want on computers were a specific program is installed. This could be accomplished by something like the following for kglobalshortcutsrc

{{if lookPath "my-fancy-program"}}
set "my-fancy-program.desktop" _k_friendly_name "My fancy program" separator="="
set "my-fancy-program.desktop" _launch "Ctrl+Shift+Y,none,my-fancy-program" separator="="
{{end}}

# Make sure the lines aren't added back into the config for all systems
# This should be outside the if statement
add:remove "my-fancy-program.desktop" _k_friendly_name
add:remove "my-fancy-program.desktop" _launch

(In this case, note that you might need to manage the .desktop file with chezmoi as well. KDE normally creates these in $HOME/.local/share/applications/.)

Similarly, remove can be used to remove entries, but be careful when readding the source files: If you blindly re-add the file on the computer where the lines are filtered out, they will get lost for all computers.

add:remove/add:hide

The directives add:remove and add:hide can be used to remove entries and hide values respectively when re-adding files from the system to the chezmoi source state.

Some use cases for this are:

  • Use add:hide to prevent a password from being added back to the source state when you re-add a file with other changes. See the konversationrc example for an example of this. By using add:hide, the line will still be present in the source file, but without its value. This ensures that the keyring transform is able to find it in the source state and do its work when checking out the file on a new system.
  • Use add:remove to prevent a line from entering the source state at all. This can be useful together with system specific configuration with the set directive: 
    {{ if (.is_work) }}
set "Default Applications" "x-scheme-handler/jetbrains" "jetbrains-toolbox.desktop" separator="="
{{ end }}
# Completely remove the line when adding back (regardless of which computer this is on).
add:remove "Default Applications" "x-scheme-handler/jetbrains"
    This example for the mimeapps.list file will add a specific line only if is_work is true. The add:remove directive helps prevent that line from being added back to the source state by mistake (where it would be applied to other computers unintentionally).

NOTE: The add:hide and add:remove directives are processed as is without going through chezmoi's template engine when re-adding files. This means it won't matter if they are inside an if block, nor can you use template expressions in their arguments.

NOTE: ignore directives also result in an implicit add:remove. Again, it doesn't matter if it is inside an if block or not currently during adding of files, and any template expressions will not be expanded.

Both of these limitations may change in the future.

Migration

This chapter cover migrations between major versions of chezmoi_modify_manager.

Migration from version 1.x to 2.x

The new Rust code base has a superset of the features of the 1.x version, and it is also about 50x faster (in release builds, about 25x in debug builds).

However, there is some work involved in migrating:

In addition, the following differences are good to know about:

  • The separate shell script to help with adding files is gone, the functionality is now built into the main program (see --help output).
  • For binary installs from GitHub, you can now use a built-in self updater (--upgrade).
  • The regex syntax is different. Previously Python re module was used, now the regex crate for Rust is used. For most simple regular expressions there will be no relevant difference. However, some features (such as back references and look arounds) are not supported.
  • Platform support with precompiled binaries is somewhat limited (compared to everything that Python supports). This is due to what I can build & test and what GitHub CI supports. Pull requests that enable testing and building for more platforms are welcome however (if the tests cannot be executed on GitHub CI, I will not accept it however).

Installation

The methods of installation are different. No longer do you need (or should) add this repo as a submodule in your dotfiles repo. Remove that and instead see the installation section in the README.

Modify scripts: Automatic conversion

There is a script that can help if you have standard shaped files (i.e. as created by the old chezmoi_ini_add).

However, it will not handle 100% of the conversion for transforms. The argument list format has changed, as have some of the argument names. See below for more details.

Also, special consideration needs to be taken for the keyring transform.

Modify scripts: Manual conversion

The first line should now be used to invoke chezmoi_modify_manager. It should be one of:

Use this if chezmoi_modify_manager is installed in PATH (recommended):

#!/usr/bin/env chezmoi_modify_manager

Use this if you keep chezmoi_modify_manager in your chezmoi source directory:

#!{{ .chezmoi.sourceDir }}/.utils/chezmoi_modify_manager-{{ .chezmoi.os }}-{{ .chezmoi.arch }}

In addition, the way to specify the source file has changed. The line to specify the source file would now typically look like:

source "{{ .chezmoi.sourceDir }}/{{ .chezmoi.sourceFile | trimSuffix ".tmpl" | replace "modify_" "" }}.src.ini"

Finally, you need to convert the actual ignores and transforms themselves:


-ik key value -> ignore "key" "value"

-is section-name -> ignore section "section-name"

-ikr key-re value-re -> ignore regex "key-re" "value-re"

# Note change of argument order for transforms, the transform name
# now comes after the match.
-tk transform_name key value '{ "arg1": "value1", "arg2": "value2" }'
   -> transform "key" "value" transform-name arg1="value1" arg2="value2"

-tkr transform_name key value '{ "arg1": "value1", "arg2": "value2" }'
   -> transform regex "key" "value" transform-name arg1="value1" arg2="value2"

Transforms

Transform arguments have changed. Before they were a JSON object, now they are a series of key="value".

Apart from that, transform names have changed:

  • kde_shortcut -> kde-shortcut
  • unsorted_list -> unsorted-list

Finally, the argument name has changed for keyring: username is now just user.

Keyring

As stated in the previous section, the argument names have changed.

In addition, because the backend for talking to the platform secret store is different, there can be other incompatibilities. Known ones include:

  • On Linux, KDE KWallet is no longer supported. Only secret stores over DBus SecretService are supported. This means it will likely end up using GNOME's secret store (Seahorse) instead. See the example for konversationrc for how to add the password, if you need to migrate.

Other platforms are untested (since I don't have any of those), but I welcome any feedback to improve this documentation.

Migration from version 2.x to 3.x

Migrating from hook scripts

In 2023 hook scripts were deprecated and then removed in early 2024 in version 3.0. They are now replaced by the add:remove and add:hide directives.

For example, to make sure that a password isn't added back into the source state you might use something like this:

transform "LoginSection" "Password" keyring service="myprogram" user="myuser"
# Make sure the password isn't added back into the config file on re-add
add:hide "LoginSection" "Password"

This would pull the password from the OS keyring, but erase it when doing a re-add.

The add:remove directive can be used to completely remove the entry instead. This can be useful together with set and system specific configuration:

{{ if (.is_work) }}
set "Default Applications" "x-scheme-handler/jetbrains" "jetbrains-toolbox.desktop" separator="="
{{ end }}
# Completely remove the line when adding back (regardless of which computer this is on).
add:remove "Default Applications" "x-scheme-handler/jetbrains"

This example for mimeapps.list would add an entry only when .is_work is true, but also make sure that the value isn't added back to the config file and thus prevents transferring it to other computers by mistake.

NOTE: The add:hide and add:remove directives are processed as is without going through chezmoi's template engine when re-adding files. This means it won't matter if they are inside an if block, nor can you use template expressions in their arguments.

NOTE: ignore directives also result in an implicit add:remove. Again, it doesn't matter if it is inside an if block or not currently during adding of files.

Troubleshooting

The first step should be to run chezmoi_modify_manager --doctor and correct any issues reported. This will help identify some common issues:

  • chezmoi_modify_manager needs to be in PATH
  • **/*.src.ini needs to be ignored in the root .chezmoiignore file
  • Old chezmoi and/or using CHEZMOI_MODIFY_MANAGER_ASSUME_CHEZMOI_VERSION, see this documentation for more details on when or when not to use this.

Limitations

Alas, no software (apart from perhaps a simple Hello, world!) is perfect. Here are some known limitations of chezmoi_modify_manager:

  • When a key exists in the .src.ini file but not in the target state it will be added to the end of the relevant section. This is not an issue as the program will usually just resort the file next time it writes out its settings.
  • modify_ scripts bypass the check for "Did the file change in the target state" that chezmoi performs. This is essential for proper operation. However, it also means that you will not be asked about overwriting changes. Always look at chezmoi diff first! I do have some ideas on how to mitigate this in the future. See also this chezmoi bug for a more detailed discussion on this.

source: How chezmoi_modify_manager finds the data file

Background

chezmoi_modify_manager needs three inputs to work:

  • The modify script with directives (ignores, transforms, etc)
  • The state of the config file in your home directory
  • The source state of the config file.

The first two are provided by chezmoi, no issues. But as far as chezmoi is concerned, the modify script itself is the source state. As such we need an alternative mechanism.

Problem

The obvious solution would be a path relative to the modify script. However, chezmoi always copies the modify script to a temporary directory before executing it, even if the modify script isn't templated. So this doesn't work. (It is however used internally in the test suite of chezmoi_modify_manager using source auto-path, which might be relevant if you are working on the chezmoi_modify_manager codebase itself.)

Prior to chezmoi 2.46.1, we had to rely on making the modify script a template, as chezmoi didn't expose enough information to us (see this chezmoi issue for more historical details). Basically we can make chezmoi find the source file for us using the following line:

source "{{ .chezmoi.sourceDir }}/{{ .chezmoi.sourceFile | trimSuffix ".tmpl" | replace "modify_" "" }}.src.ini"

Since chezmoi 2.46.1, chezmoi now provides us with two environment variables:

  • CHEZMOI_SOURCE_DIR: Path to the source directory root
  • CHEZMOI_SOURCE_FILE: Path to our modify script (relative the source directory root)

With these two together we no longer need templating, and the following works:

source auto

What the code does

Since chezmoi_modify_manager 3.1, it will auto-detect the version of chezmoi (based on executing chezmoi --version). This is used for:

  • The template that --add creates to either use the templated source string or the simpler source auto.
  • Interpreting the meaning of --style=auto (default value for style) to either create a templated modify script or a non-templated modify script.

The main benefit of the simpler source auto is that if your modify script doesn't need to be a template for any other reason, it will speed up execution, as chezmoi no longer needs to run its template engine.

Overriding auto detection

Auto-detection has one downside though: What if you use multiple versions of chezmoi (such as an old version from Debian stable on some server but an up-to-date version on your personal computer). In that case you don't want to use the newer syntax for compatibility reasons.

The workaround is to export an environment variable CHEZMOI_MODIFY_MANAGER_ASSUME_CHEZMOI_VERSION set to the oldest version that you use. E.g:

CHEZMOI_MODIFY_MANAGER_ASSUME_CHEZMOI_VERSION=2.46.0

This could be set in your .bashrc/.zshrc/.profile or similar file (the details of how to best set environment variables for a particular platform and shell is out of scope of this documentation).

Actions & directives

This is a high level overview of how chezmoi_modify_manager applies your config. For the details on specific directives see chezmoi_modify_manager --help-syntax.

Glossary

  • Directive: Things like source, ignore, transform, set, add:hide, etc that you put in your config file. They are documented in the output of chezmoi_modify_manager --help-syntax.
  • Actions: The directives are internally translated into a ruleset of actions. These are very similar to the directives, but may not correspond 1:1. For example:
    • set becomes a special transform internally.
    • source doesn't enter the actions, it is only used to figure out what file to load.
    • etc.

Contexts

There are two different "contexts" for evaluating actions:

  • Merging: This is the normal algorithm, used during chezmoi apply (and diff etc)
  • Filtering: This is using when re-adding an existing file (chezmoi_modify_manager -a or -s).

See Algorithms for details of how these work, in this file we are only concerned with how the directives and rules matching works.

These have separate directive to action translators. Not all directives apply to all contexts. Some examples:

  • set is unused when filtering
  • add:hide is unused when merging
  • ignore translates to the same as add:remove when filtering.
  • etc.

Order of action matching

Actions come in three flavours:

  1. Section matches (always literal matches)
  2. Literal section+key matches
  3. Regular expression section+key matches

Not every rule can exist in every variant. For example:

  • Merge section matches only support ignore and remove.
  • set will only ever exist as a literal section+key match
  • etc.

Chezmoi_modify_manager uses a single regex to match both the section and key. This is done by constructing a combined string for these, using the 0-byte (\0) as a separator. For example a regex directive ignore regex "Section|OtherSection" "SomePrefix.*" is compiled down to (?:Section|OtherSection)\0(?:SomePrefix.*). This can be visible if you attempt to use ^ or $ (don't do that).

The special string <NO_SECTION> is used to match keys that appear before the first section (hopefully no one has an ini-file with a section with that name in it).

When matching actions:

  1. We first check if any section action applies. If so we are done. These are always literal matches.
  2. Then we check if there is a literal section+key match. If so it applies and we are done.
  3. Otherwise, we check if any regex action matches. If so we take the first result. This will be the same as first in source order in your config file.

Additionally, chezmoi_modify_manager will warn if there are multiple regex matches that match. This can be disabled (per file) with a no-warn-multiple-key-matches directive, in case you want this behaviour.

Algorithms

This documents a high level overview of the algorithms chezmoi_modify_manager uses for merging or filtering INI files.

In general these algorithms are single-pass, processing one line at a time from the input INI file. This makes them quite fast in practice.

The code for these are implemented in the ini-merge crate. The explanation here is intended for users, and as such leaves out a lot of gnarly details around for example: empty sections, sections with only comments in them, etc. If you are interested in that, go read the code.

The actual INI parser is in the ini-roundtrip crate. A custom INI parser is used to ensure that writing it back out doesn't change formatting.

Filtering

This is used when re-adding an existing file (chezmoi_modify_manager -a or -s). This is the simpler of the two algorithms.

Relevant directives from your config for this algorithm:

  • add:hide (replaces the value with HIDDEN when re-adding)
  • add:remove (removes the line when re-adding)
  • ignore (removes the line when re-adding)

(The reason there are two directives with the same effect is that they do different things when merging.)

When a user passes -s or -a a bunch of things happen:

  1. We figure out if the file is already managed or not. Depending on -a or -s we will then do different things. This is not the focus of this page though.
  2. Assuming we decided that we should add the file and manage it using chezmoi_modify_manager (instead of plain chezmoi), and that the file was already managed by us before, we then need to filter:
    1. Load the ruleset that the user wrote into an Actions structure. Currently, this does not take chezmoi templates into account (though this might change).
    2. For each line in the file being filtered:
      • If it is a new section header, check section actions to determine if it should be removed entirely, otherwise keep it.
      • If it is a comment or blank line keep it (unless the entire section is being removed)
      • If it is a key, check actions to determine if it should be hidden, removed or kept.

Note evaluation order of actions documented in Actions, section matches take priority, then literal matches, then regex matches (in order).

Merging

This is used for normal chezmoi apply (and chezmoi diff etc). This is a more complicated case: there are now three files involved.

Relevant directives from your config for this algorithm:

  • ignore (keeps the system state and ignores whatever is in the source state)
  • set (sets to a specific key and value)
  • remove (entirely removes the match)
  • transform (applies a custom transform to the match, see --help-transforms, custom semantics apply to each)
  1. Load the ruleset that the user wrote into an Actions structure. Chezmoi has already processed any templates for us.
  2. Load the .src.ini file into a fast data structure for looking things up in it.
  3. For each line in the system state (as provided by chezmoi on stdin):
    • If it is a comment or blank line, keep it (unless it is in a section that we are not outputting).
    • If it is a section header, check:
      • If the entire section is ignored, keep it as is from the system state.
      • If the section is being removed by remove, remove it.
      • If the section exists in the .src.ini, keep it.
      • If the section doesn't exist in the .src.ini, remove it.
      • (There is also some additional logic to deal with entirely empty sections etc, so we don't actually emit the section on stdout until we are sure later on, there is a concept of "pending lines" to implement that.)
    • If it is a key, find the first action that applies if any. Then:
      • If no action applies, take the value from the .src.ini file.
      • If no action applies and the line is not in the .src.ini file, remove the line.
      • If the action is to ignore, leave the system value as is.
      • If the action is to remove, remove it.
      • If the action is to set, set it.
      • If a transform applies, apply it (see each transform for more details).
    • Before we start a new section, check if there are any lines in the .src.ini that didn't exist in the system state (or any such set directives), if so emit them.
    • Before the end of the file, check for entire sections (or set directives in such sections) in the .src.ini that didn't exist in the system state, if so emit them.

The newly emitted keys or sections from the last two bullet points will generally be weirdly formatted. The assumption is the program that owns this file will reformat it on next use.

Packaging for Linux distros etc

Do you want to package chezmoi_modify_manager in your favourite package manager? Awesome! Here are some helpful notes for you.

  • Please tell me about your package (file a github issue): I will link to any suitable package from the README (with a written caveat that I don't maintain them and thus cannot guarantee that they are up-to-date or safe to use). I maintain the AUR package myself as of writing this (thus there is no disclaimer there).
  • If you need some inspiration for how to build and install properly, you might want to take a look at my AUR PKGBUILD. Arch Linux uses a relatively simply format (bash scripts), so it should be easy to decode and adapt.

How to build

  • When building, please export the environment variable CHEZMOI_MODIFY_MANAGER_BUILDER. This will be reported in chezmoi_modify_manager --doctor, and can be very helpful in bug reports. If you don't set this, --doctor will report it as a warning (assuming that it is an unknown local build on someone's computer).
    • Please set it to a short and truthful value (such as "debian", "homebrew", "aur" etc) that identifies the package ecosystem. If your package build the latest and greatest git version, please add a suffix indicating so (e.g. "aur-git")
    • Especially don't claim to be "github-ci" or "github-release" as those are used for the official binary releases. That would be Not Cool!
  • Build a release build (--release). Rust is really quite slow in debug builds, but very speedy in release builds. The difference between debug and release builds is much larger than for C/C++.
  • Build with --locked: This ensures that the versions of dependencies are exactly the same as upstream. Otherwise, you might get newer supposedly compatible versions. Those may or may not work.
  • Build with the stable Rust toolchain: Nightly or beta is not needed and is just asking for potential issues.
  • You likely want to exclude the built-in self updater (that downloads from Github releases), as your package manager should be used instead. This is easy: pass --no-default-features --features=keyring to cargo build. This will also avoid vendoring C dependencies (in particular libdbus) and instead link them dynamically. If you don't want that, add the vendored feature as well (i.e. --features=keyring,vendored).

And so we arrive at the final (reliable regardless of what environment the user might have) build command:

export CHEZMOI_MODIFY_MANAGER_BUILDER="<your package ecosystem name>"
export RUSTUP_TOOLCHAIN=stable
export CARGO_TARGET_DIR=target
cargo build --locked --release --no-default-features --features=keyring

If you need to download dependencies first (as is best practise for some build systems) this gets split into two phases:

# Download deps
export RUSTUP_TOOLCHAIN=stable
cargo fetch --locked --target "$(rustc -vV | sed -n 's/host: //p')"

# Build
export CHEZMOI_MODIFY_MANAGER_BUILDER=aur
export RUSTUP_TOOLCHAIN=stable
export CARGO_TARGET_DIR=target
cargo build --frozen --release --no-default-features --features=keyring

Note the change from --locked to --frozen in the second command here.

What to install

You should of course install the binary itself chezmoi_modify_manager. However, you might also want to install shell completion files relevant to your platform. These can be generated by executing the built binary with --bpaf-complete-style-<name of shell>. Here is the code from the AUR PKGBUILD to do the entire install:

local _cmd_name="target/release/${pkgname}"
install -Dm0755 -t "$pkgdir/usr/bin/" "$_cmd_name"
mkdir -p "$pkgdir/usr/share/bash-completion/completions/"
mkdir -p "$pkgdir/usr/share/zsh/site-functions/"
mkdir -p "$pkgdir/usr/share/fish/vendor_completions.d/"
"$_cmd_name" --bpaf-complete-style-zsh > "$pkgdir/usr/share/zsh/site-functions/_$pkgname"
"$_cmd_name" --bpaf-complete-style-bash > "$pkgdir/usr/share/bash-completion/completions/$pkgname"
"$_cmd_name" --bpaf-complete-style-fish > "$pkgdir/usr/share/fish/vendor_completions.d/${pkgname}.fish"
# No support to install distro completions in elvish.
# See https://github.com/elves/elvish/issues/1739
#"$_cmd_name" --bpaf-complete-style-elvish

For more info on supported shells, see the bpaf documentation, which is the library used by chezmoi_modify_manager to handle command line parsing.

Design decisions

This file documents motives some design decisions.

Why did you implement a custom INI parser?

I ended up writing my own INI parser for rust: ini-roundtrip. This had to be done because standard INI parsers don't support preserving the formatting. This is not acceptable when trying to minimise the diff. We want to not change the formatting applied by the program that writes the settings file. For example KDE writes key=value while PrusaSlicer writes key = value.

It also does minimal parsing, meaning it can handle weird non-standard syntax such as [Colors:Header][Inactive] (a real example from kdeglobals).

Why Rust?

This code used to be written in Python, but each invocation of the command would take on the order of 95 ms. Per managed file. As I was getting up to around 20 managed INI files, this started to add up. The rewrite in Rust takes (on the same computer) 2 ms. This is a 46x speedup. On another (faster) computer I got a 63x speedup (54 ms vs 0.9 ms).

Fast path

The most time critical operation is to compute the new system state when chezmoi invokes us. This is the "fast path" in the code. All other operations such as --add, --update etc are less important from a performance perspective. This should be kept in mind when adding new features.