A long time ago I spent some time thinking about how I structure files on my computer. Here's what I've been using for over 7 years now and it's still going strong.

Organized Chaos

The directory structure is:

    x/<broad-category>/<projects (or "package")>

In other words, there's a single-letter directory that contains a set of directories that designate broad categories. Here are some broad categories:

    coding
    music
    pics
    temp
    sensitive
    base
    nixos

Under these broad categories, all I create are directories that designate some "project". A project is (in this use of the word) just a way to designate files that belong together. Typically, a project will be worked on for some time, after which the directory is just stored there. As an example, "pics/vacation-20XX", "base/.vim", or "coding/blog" (where this blog resides). Every "project" has its own internal structure but that really bears no importance on the parent or sibling directory structure. The broad category + project structure is a way to get a simple and scalable overview. There's basically no mental overhead once the broad categories are established, and they're easily changed.

Coming back to a project is easy. The projects are easily listed with a single ls invocation. An lsa alias lists the 20 last modified projects. This is very practical. It's also easy to search globally from the top-level directory.

Why the top-level directory "x"?

Because it makes rsync-ing easy and it keeps it from getting polluted by programs that put stuff in $HOME.

What about configurations that need to be in $HOME?

I use a tool to symlink these (a custom script listed below). I've tried using GNU Stow but it's just inadequate.

Problems with GNU Stow

I have two directories, one with sensitive configurations (x/sensitive), and one with non-sensitive configurations (x/base). These overlap in the files they contain; both contain ".ssh/authorized_keys". On a system where I can allow sensitive configurations, I will first symlink from $HOME to everything in x/base, and then do the same for x/sensitive (overwriting anything that has the same name).

This is intended behavior, as I want to "expand" my capabilities when on a sensitive-allowed system by overwriting non-sensitive files with sensitive files. Because I have some files in base that are not in "x/sensitive", I need to perform this overwrite step. GNU Stow does not allow this. A workaround is to symlink from sensitive to all files in "x/base" that "x/sensitive" doesn't re-define, and just use GNU Stow to `stow sensitive`, but I don't want to do the bookkeeping for putting symlinks in "x/sensitive" to these files. That defeats the point of computing: making life easier.

At that point I'd be using the wrong tool for the job. Not only do I want to symlink a directory, but also sometimes only its contents. For instance for directories like ".config", I don't want my dotfiles polluted by programs that store into "~/.config"

GNU Stow is also not available everywhere. I just want to rsync and run the script and get my entire home setup.

Finally, this script permits me to have different files for different DPI screens. The typical flow is as follows for getting it working on a new machine:

    $ rsync -auv x/ other-machine:~/x/
    $ ssh other-machine
    $ cd x/base  # Stored generally useful scripts that can be shared, configuration that can be viewed publicly
    $ ./install
    $ cd ../sensitive  # Stores SSH keys, passwords, private data
    $ ./install
    $ cd ../nixos/other-machine  # Stores some per-system config, (screen DPI, input setups)
    $ ./install

On systems without sensitive access I do not rsync the "sensitive" directory" and hence not `./install` it either.

Code

#! /usr/bin/env bash

# ARG_HELP([Link to the current directory tree from a target. Will link files recursively only. Directories containing a .link file will be symbolically linked in their entirety.])
# ARG_OPTIONAL_SINGLE([target],[t],[Target directory to link from.],["$HOME"])
# ARG_OPTIONAL_BOOLEAN([dry-run],[d],[Only print intended actions.])
# ARG_OPTIONAL_SINGLE([except],[e],[Items to skip by regex, reads from .except by default.],[])
# ARGBASH_GO()
# needed because of Argbash --> m4_ignore([
### START OF CODE GENERATED BY Argbash v2.10.0 one line above ###
# Argbash is a bash code generator used to get arguments parsing right.
# Argbash is FREE SOFTWARE, see https://argbash.io for more info


die()
{
	local _ret="${2:-1}"
	test "${_PRINT_HELP:-no}" = yes && print_help >&2
	echo "$1" >&2
	exit "${_ret}"
}


begins_with_short_option()
{
	local first_option all_short_options='htde'
	first_option="${1:0:1}"
	test "$all_short_options" = "${all_short_options/$first_option/}" && return 1 || return 0
}

# THE DEFAULTS INITIALIZATION - OPTIONALS
_arg_target="$HOME"
_arg_dry_run="off"
_arg_except=


print_help()
{
	printf '%s\n' "Link to the current directory tree from a target. Will link files recursively only. Directories containing a .link file will be symbolically linked in their entirety."
	printf 'Usage: %s [-h|--help] [-t|--target ] [-d|--(no-)dry-run] [-e|--except ]\n' "$0"
	printf '\t%s\n' "-h, --help: Prints help"
	printf '\t%s\n' "-t, --target: Target directory to link from. (default: '"$HOME"')"
	printf '\t%s\n' "-d, --dry-run, --no-dry-run: Only print intended actions. (off by default)"
	printf '\t%s\n' "-e, --except: Items to skip by regex, reads from .except by default. (no default)"
}


parse_commandline()
{
	while test $# -gt 0
	do
		_key="$1"
		case "$_key" in
			-h|--help)
				print_help
				exit 0
				;;
			-h*)
				print_help
				exit 0
				;;
			-t|--target)
				test $# -lt 2 && die "Missing value for the optional argument '$_key'." 1
				_arg_target="$2"
				shift
				;;
			--target=*)
				_arg_target="${_key##--target=}"
				;;
			-t*)
				_arg_target="${_key##-t}"
				;;
			-d|--no-dry-run|--dry-run)
				_arg_dry_run="on"
				test "${1:0:5}" = "--no-" && _arg_dry_run="off"
				;;
			-d*)
				_arg_dry_run="on"
				_next="${_key##-d}"
				if test -n "$_next" -a "$_next" != "$_key"
				then
					{ begins_with_short_option "$_next" && shift && set -- "-d" "-${_next}" "$@"; } || die "The short option '$_key' can't be decomposed to ${_key:0:2} and -${_key:2}, because ${_key:0:2} doesn't accept value and '-${_key:2:1}' doesn't correspond to a short option."
				fi
				;;
			-e|--except)
				test $# -lt 2 && die "Missing value for the optional argument '$_key'." 1
				_arg_except="$2"
				shift
				;;
			--except=*)
				_arg_except="${_key##--except=}"
				;;
			-e*)
				_arg_except="${_key##-e}"
				;;
			*)
				_PRINT_HELP=yes die "FATAL ERROR: Got an unexpected argument '$1'" 1
				;;
		esac
		shift
	done
}

parse_commandline "$@"

# OTHER STUFF GENERATED BY Argbash

### END OF CODE GENERATED BY Argbash (sortof) ### ])
# [ <-- needed because of Argbash

set -Eeuo pipefail

shopt -s extglob

if [ -z "$_arg_except" ] && [ -e .except ] && [ -r .except ]; then
	except=$(cat .except)
elif [ -z "$_arg_except" ]; then
	except='^$'
else
	except=$_arg_except
fi

dry=$_arg_dry_run
self=$(realpath "${BASH_SOURCE[0]}")
if ! [ -e "$_arg_target" ] && [ "$dry" = 'on' ]; then
	exit
fi

source "${self%install}/.bash_colors"

target=$_arg_target

prefix() {
	sed "s/^/$1: /"
}

is_dry() {
	[ "$dry" = 'on' ]
}

dry() {
    if ! is_dry; then
        "$@"
    else
        echo "DRY: PWD=$PWD $@"
    fi
}

link() {
	from=$(echo "$1/$2" | sed $'s/\'/\'\'/g')
	to=$(echo "$PWD/$2" | sed $'s/\'/\'\'/g')
	postfix=${3:-FILE}

	if [ -L "$from" ] && [ "$(readlink "$from")" = "$to" ]; then
		echo "${BCyan}ALREADY LINKED $postfix: '$from' -> '$to'${Color_Off}"
		return
	elif [ -L "$from" ] || [ -e "$from" ]; then
		echo "${BRed}REMOVING: $(file "$from") ($from)${Color_Off}"
		dry rm -rf "$from"
	fi

	echo "${Green}LINKING $postfix: '$from' -> '$to'${Color_Off}"
	dry ln -s "$to" "$from"
}

handle_directory() {
	from="$1/$2"
	to="$PWD/$2"
	if [ -e "$2/.link" ]; then
		link "$1" "$2" "DIRECTORY"
		link=$2/.link
		if [ -f "$link" ] && [ -r "$link" ] && [ -x "$link" ]; then
			dry "$link" "$from" |& prefix LINK
		fi
	elif [ -e "$2/.skip" ]; then
	 	echo "${Purple}SKIP: $to${Color_Off}"
	else
		if [ -L "$from" ] && [ ! -e "$from" ]; then
			echo "${Yellow}REMOVING BROKEN SYMLINK: $(file "$from")${Color_Off}"
			dry rm "$from"
		fi
		if ! [ -d "$from" ]; then
			dry mkdir "$from"
		fi
		if is_dry; then
			(cd "$2"; "$self" -t "$from" -e "$except" -d)
		else
			(cd "$2"; "$self" -t "$from" -e "$except")
		fi
	fi
}

for file in .* *; do
	if [ "$file" = '.' ] || [ "$file" = '..' ]; then
		continue
	fi
	if echo "$file" | grep -P "$except" > /dev/null; then
		echo "${Purple}SKIP: $file - part of --except: $except${Color_Off}"
		continue
	fi
	if [ -f "$file" ] && [ "$(realpath "$file")" != "$self" ] ; then
		link "$target" "$file"
	elif [ -d "$file" ]; then
		handle_directory "$target" "$file"
	fi
done

# ] <-- needed because of Argbash