In this post we will walk you through setting up a server for hosting Git repositories with NixOs and NixOps. We’ll write a configuration file for Nix that can be used to deploy to a server using NixOps. In this case we will deploy a virtual server using VirtualBox (this can be changed to deploy to a server hosted by a cloud provider if desired). You’ll need to have Nix, NixOps, & VirtualBox (you may need to configure VirtualBox correctly to work with NixOps) installed for this guide.
Let’s first start with a description of what the final configuration will look like. The server will have a git user named git that will handle all the git operations. The git repositories themselves will be stored in the git user’s home directory. The git users login shell will be set to the git-shell. Git-shell is a shell that comes with git that only allows running git commands. This will prevent git users from doing anything fishy on the server. Cloning git repositories from the git server will be as simple as:
git clone git@<IP_ADDRESS_OF_SERVER>:<NAME_OF_GIT_REPOSITORY>We will supply our server with shell scripts that can be used to backup repositories, create a new repository, delete a repository, and list all the git repositories stored on the server. A cron job will be configured to run the backup script once a day. Nix will be used to package our shell scripts and if a change is made to any of the scripts we can use NixOps to deploy the changes for us with no change to our configuration files.
The git-server.nix file contains the configuration of our network (which in this case is simply the git server itself). It is as follows:
{
network.description = "Git server";
git-server =
{ config, pkgs, ... }:
let
repos-dir = "/home/git"; #set up a directory to hold the git repos, this will also be the git users home directory
repos = import ./repos-packages.nix { inherit pkgs repos-dir; };
in
{
imports = [
./virtualbox.nix
];
time.timeZone = "UTC";
services.openssh.enable = true;
services.cron.enable = true;
services.cron.systemCronJobs = [ "30 9 * * * root repos-backup" ]; #run repos-backup once a day at 9:30
nix.gc.automatic = true;
environment.systemPackages = with pkgs;
[
vim git
#custom scripts
repos.repos-backup repos.repos-create repos.repos-delete repos.repos-list repos.repos-setenvvars
];
users.mutableUsers = false;
users.users.root.openssh.authorizedKeys.keys = [
"your public rsa key goes here"
];
users.users.git = {
isNormalUser = true;
description = "git user";
createHome = true;
home = "${repos-dir}";
shell = "${pkgs.git}/bin/git-shell";
openssh.authorizedKeys.keys = ["your public rsa key goes here"];
};
};
}Let’s go through this config piece by piece to explain what is going on.
Describe the network as well as define the git-server (the only server in our network).
{ config, pkgs, ... }:
let
repos-dir = "/home/git"; #set up a directory to hold the git repos, this will also be the git users home directory
repos = import ./repos-packages.nix { inherit pkgs repos-dir; };Input the config/pkgs in nix. The let binding here defines variables to be used in the following block of code. repos-dir defines the directory to store the git repositories. repos contains the nix derivations that build our shell scripts that are imported from repos-packages.nix (which we will write later). We need to give it the pkgs & repos-dir (so that the directory to store git repositories in is defined in one place & easily changeable in the configuration).
Begin the block of code defining the git server.
Import the VirtualBox deployment (which we will write later). This can be swapped out for another deployment method (for example if you wanted to deploy to the cloud instead).
time.timeZone = "UTC";
services.openssh.enable = true;
services.cron.enable = true;
services.cron.systemCronJobs = [ "30 9 * * * root repos-backup" ]; #run repos-backup once a day at 9:30Set the timezone to UTC (Note: the timezone must be set or cron won’t work right with NixOs). Feel free to change it to a different time zone. Enable openssh so that your site can be connected to with SSH. Enable cron and set it up with a job to backup all of your git repositories once a day at 9:30. repos-backup is the name of the script to back up the git repositories we will write later.
nix.gc.automatic = true;
environment.systemPackages = with pkgs;
[
vim git
#custom scripts
repos.repos-backup repos.repos-create repos.repos-delete repos.repos-list repos.repos-setenvvars
];Tell NixOs to automatically run Garbage Collection (this removes unused packages from the system periodically). environment.systemPackages is where we tell Nix which programs we would like installed and available on the system path. We’ll tell it that we would like to have vim & git installed as well as all of our custom scripts that we’ll define later. Vim is not necessary, but is useful for editing/viewing of files.
This sets it so that our users can only be configured through our configuration file.
Input the public rsa keys that can be used to log in as root to openssh. Make sure to set this to your key or you won’t be able to log in.
users.users.git = {
isNormalUser = true;
description = "git user";
createHome = true;
home = "${repos-dir}";
shell = "${pkgs.git}/bin/git-shell";
openssh.authorizedKeys.keys = ["your public rsa key goes here"];
};This defines a user called git for the server. This user will be used for all git functionality. It is a normal user & its home directory is where we will store all of the git repositories. ${repos-dir} will evaluate the repos-dir variable declared earlier. We make sure to set the git users login shell to the git-shell so that it can’t run any non git commands. ${pkgs.git} will evaluate to the directory that git is installed into in NixOs. This ensures that the correct path to git-shell is used. openssh.authorizedKeys.keys contains the public rsa keys that can be used to run git commands on the server. Put whichever keys in here you would like to grant access to. If you don’t put any keys here no one will be able to use the git server.
This is actually pretty simple.
{pkgs, ...}: let
targetEnv = "virtualbox";
virtualbox = {
memorySize = 1024;
vcpu = 1;
headless = true;
};
in {
deployment = {
targetEnv = targetEnv;
virtualbox = virtualbox;
};
}Feel free to change the options (ie. memory, cpus used, or whether or not it’s headless) as you wish. deployment refers to the environment used to deploy the server.
This config file defines Nix derivations for our shell scripts used to help manage our git server. These derivations essentially tell Nix how to build our scripts. In our case building will be fairly simple as we only need to copy the scripts over to the server and ensure that they are on the system path.
{ pkgs ? import <nixpkgs> {}, repos-dir ? "/home/git" }:
let
stdenv = pkgs.stdenv;
sh = pkgs.sh;
coreutils = pkgs.coreutils;
in {
repos-backup = stdenv.mkDerivation rec {
name = "repos-backup";
builder = "${sh}/bin/sh";
args = [ ./shell-script-builder.sh ];
src = ./repos-backup.sh;
buildInputs = [ coreutils ];
system = builtins.currentSystem;
};
repos-create = stdenv.mkDerivation rec {
name = "repos-create";
builder = "${sh}/bin/sh";
args = [ ./shell-script-builder.sh ];
src = ./repos-create.sh;
buildInputs = [ coreutils ];
system = builtins.currentSystem;
};
repos-delete = stdenv.mkDerivation rec {
name = "repos-delete";
builder = "${sh}/bin/sh";
args = [ ./shell-script-builder.sh ];
src = ./repos-delete.sh;
buildInputs = [ coreutils ];
system = builtins.currentSystem;
};
repos-list = stdenv.mkDerivation rec {
name = "repos-list";
builder = "${sh}/bin/sh";
args = [ ./shell-string-script-builder.sh ];
src = ''
#!/bin/sh
. repos-setenvvars
ls -d $reposDir/*.git | xargs -n1 basename
'';
buildInputs = [ coreutils ];
system = builtins.currentSystem;
};
repos-setenvvars = stdenv.mkDerivation rec {
name = "repos-setenvvars";
builder = "${sh}/bin/sh";
args = [ ./shell-string-script-builder.sh ];
src = ''
#!/bin/sh
# set environment variables for use in repos scripts
reposDir="${repos-dir}" #directory containing the git repos
reposBackupDir=$reposDir/repobackups #directory containing the git repos backups
'';
buildInputs = [ coreutils ];
system = builtins.currentSystem;
};
}Lets go through this line by line.
Input two optional variables into our Nix function. These determine what package repository to use (defaulting to nixpkgs) and what the repos-dir should be set to (defaulting to “/home/git”).
Define variables in a let block for use in the upcoming code block. These are simply used to define a shorthand way of referring to a few packages.
Start the code block.
repos-backup = stdenv.mkDerivation rec {
name = "repos-backup";
builder = "${sh}/bin/sh";
args = [ ./shell-script-builder.sh ];
src = ./repos-backup.sh;
buildInputs = [ coreutils ];
system = builtins.currentSystem;
};
repos-create = stdenv.mkDerivation rec {
name = "repos-create";
builder = "${sh}/bin/sh";
args = [ ./shell-script-builder.sh ];
src = ./repos-create.sh;
buildInputs = [ coreutils ];
system = builtins.currentSystem;
};
repos-delete = stdenv.mkDerivation rec {
name = "repos-delete";
builder = "${sh}/bin/sh";
args = [ ./shell-script-builder.sh ];
src = ./repos-delete.sh;
buildInputs = [ coreutils ];
system = builtins.currentSystem;
};Define Nix derivations for our backup, create, and delete scripts for handling git repositories. These are all fairly similar except they have different names & scripts. name is the name of the derivation. This will also end up being the name of the command that can be run in the terminal to run the script. builder is what will be used to run the builder script. In our case it is sh. args contains the script that will be run by sh in order to build the Nix derivation. we’ll define shell-script-builder.sh in a moment. src is the source to be built. The shell scripts in all of these scripts will be defined later in the post. buildInputs contains the packages that are needed in order to build this derivation. coreutils contains basic commands like cp that we need.
Let’s go over the shell-script-builder.sh script so that we can understand how it is doing the building.
#!/bin/sh
set -e
unset PATH
for p in $buildInputs; do
export PATH=$p/bin${PATH:+:}$PATH
done
mkdir -p $out/bin
cp $src $out/bin/$nameset -e makes it so that any subsequent commands that fail will make the script fail. Next we build a PATH so that the correct binaries are on it. First clear the path with unset. The for loop adds the bin directory for each pkg in buildInputs to our path. This way the correct binaries are available during our build stage.
The out variable contains the output directory path for our derivation. This variable is set by Nix. We use mkdir to create a bin directory in the output directory for our script.
Note:
Executables/scripts must be put in the bin directory of the derivation or they won’t be put on the system path (this took me quite a while to figure out so I’m pointing it out to you).
Finally copy the the src code file to the bin directory of the derivation with the name defined in the derivation with the cp command.
Ok with that out of the way let’s go back to the rest of the repos-packages.nix file.
repos-list = stdenv.mkDerivation rec {
name = "repos-list";
builder = "${sh}/bin/sh";
args = [ ./shell-string-script-builder.sh ];
src = ''
#!/bin/sh
. repos-setenvvars
ls -d $reposDir/*.git | xargs -n1 basename
'';
buildInputs = [ coreutils ];
system = builtins.currentSystem;
};
repos-setenvvars = stdenv.mkDerivation rec {
name = "repos-setenvvars";
builder = "${sh}/bin/sh";
args = [ ./shell-string-script-builder.sh ];
src = ''
#!/bin/sh
# set environment variables for use in repos scripts
reposDir="${repos-dir}" #directory containing the git repos
reposBackupDir=$reposDir/repobackups #directory containing the git repos backups
'';
buildInputs = [ coreutils ];
system = builtins.currentSystem;
};This defines the Nix derivations for our setenvvars & list scripts. The repos-list script simply lists out the current git repositories using the ls command. The repos-setenvvars script defines two variables containing the location of the git repositories as well as the location of the git repositories backups to be called by the other scripts. There’s two key differences here between these derivations are our other three derivations.
repos-setenvvars script in our configuration file allows the location of the reposDir to be defined in the Nix configuration. This keeps us from having to keep multiple files in multiple locations synced. The other scripts can then call the repos-setenvvars scripts to keep the locations consistent.shell-script-builder.sh has been replaced with shell-string-script-builder.sh which is used to build these derivations. It is similar to shell-script-builder.sh with some small differences.shell-string-script-builder.sh is as follows:
#!/bin/sh
set -e
unset PATH
for p in $buildInputs; do
export PATH=$p/bin${PATH:+:}$PATH
done
mkdir -p $out/bin
echo "$src" &> $out/bin/$name
chmod a+xr $out/bin/$nameThis is very similar to shell-script-builder.sh, but here we echo the contents of the src variable (which contains the contents of the shell script that is set by Nix) to a file. We must then make sure to call chmod on the created file to give the newly created file read and executable rights (else you won’t be able to run the script).
A test-build for
repos-packages.nixcan be done using nix-build with the following command:
nix-build repos-packages.nixLet’s go ahead and define the repos-backup.sh, repos-create.sh, and repos-delete.sh scripts. These will be similar to their names and will backup your repositories, create a new repository, & delete a repository. On the git server these scripts will be ran as such: repos-backup, repos-create, & repos-delete.
The repos-backup.sh script is as follows:
#!/bin/sh
#backs up git repos in your repo directory
. repos-setenvvars #set environment variables
if [ -z "$reposDir" ] || [ ! -d "$reposDir" ]; then
echo "exiting, can't find reposDir or env variables not set"
exit 1
fi
mkdir -p $reposBackupDir
for repo in $(ls -d $reposDir/*.git)
do
repobase=$(basename $repo)
checkSumFile=$reposBackupDir/sha-$repobase.sha256
checkSumFileContents=$(cat $checkSumFile)
checkSum=$(find $repo -type f -exec sha256sum {} \; | sha256sum)
#check the old checksum vs the new one to see if the repo has changed
if [ "$checkSum" == "$checkSumFileContents" ]
then
echo "repo: '$repo' has not changed..."
#don't need to do anything as the repo hasn't been updated
else
#tar the repo and then back it up to the cloud... remove the tar file when we're done
echo "repo: '$repo' has changed... backing up..."
tarFile=$reposBackupDir/$repobase.tar.gz
tar -zcvf $tarFile $repo
#TODO: transfer the tar file to cloud storage here
rm -f $tarFile
echo "finished backing up repo: '$repo'"
fi
#update the checksums
rm -f $checkSumFile
echo "$checkSum" &> $checkSumFile
doneThe script first loads the repos-setenvvars script in order to set the environment variables. It exits if either the reposDir variable is not set or the location referenced by reposDir does not exist on the file system. Next it makes the repos backup directory if it does not exist via mkdir.
Loop through each git repo in reposDir. Each git repo ends with the prefix .git.
repobase=$(basename $repo)
checkSumFile=$reposBackupDir/sha-$repobase.sha256
checkSumFileContents=$(cat $checkSumFile)
checkSum=$(find $repo -type f -exec sha256sum {} \; | sha256sum)For each repo we’ll compute a checksum based upon the repos contents and store it in a file. If the current computed checksum is different then the stored checksum then it means the repo has changed and that we’ll need to back it up. The checkSum variable computes the current checksum of the repo (using a sha256 hash) & the checkSumFileContents contains the old checksum of the repo.
#check the old checksum vs the new one to see if the repo has changed
if [ "$checkSum" == "$checkSumFileContents" ]
then
echo "repo: '$repo' has not changed..."
#don't need to do anything as the repo hasn't been updated
else
#tar the repo and then back it up to the cloud... remove the tar file when we're done
echo "repo: '$repo' has changed... backing up..."
tarFile=$reposBackupDir/$repobase.tar.gz
tar -zcvf $tarFile $repo
#TODO: transfer the tar file to cloud storage here
rm -f $tarFile
echo "finished backing up repo: '$repo'"
fiCheck if the checksums match. If they do we don’t have to do anything. If they don’t then we’ll use tar to create a compressed tar archive and then back it up as a change has occurred.
Note:
You’ll need to write code to transfer the tar archive to your storage and place it where the todo note is. Otherwise it won’t actually back up anything.
After transferring the tar file we remove it so that we’re not taking up unnecessary space on the server.
Finally make sure to update the checkSumFile with a hash of the repositories current contents.
The repos-create.sh script is as follows:
#!/bin/sh
#creates a git repo in your repo directory
#input a cli argument with the git repo to create
set -e
. repos-setenvvars #set environment variables
if [ -z "$reposDir" ] || [ ! -d "$reposDir" ]; then
echo "exiting, can't find reposDir or env variables not set"
exit 1
fi
if [ -z "$1" ]; then
echo "exiting, no repo name input to create"
echo "usage: 'repos-create <name-of-repo-to-create>'"
exit 1
fi
case "$1" in
*.git)
newRepoDir=$reposDir/$1 ;;
*)
newRepoDir=$reposDir/$1.git ;;
esac
if [ -d "$newRepoDir" ]; then
echo "repo: '$newRepoDir' already exists, exiting..."
exit 1
else
echo "creating new repo: '$newRepoDir'"
mkdir -p $newRepoDir
cd $newRepoDir
git init --bare
echo "new repo: '$newRepoDir' created"
fiThis script starts off the same way as the last one by getting setting the environment variables appropriately.
if [ -z "$1" ]; then
echo "exiting, no repo name input to create"
echo "usage: 'repos-create <name-of-repo-to-create>'"
exit 1
fiIf there is no repo name input into the script as a command line argument then quit & print the usage. $1 refers to the first command line argument.
Set up a variable containing the path of the new git repo. This will also ensure that the repo ends in .git by adding it to the end if necessary. This allows the script to accept an argument that may or may not end in .git and standardize it to end in .git.
if [ -d "$newRepoDir" ]; then
echo "repo: '$newRepoDir' already exists, exiting..."
exit 1
else
echo "creating new repo: '$newRepoDir'"
mkdir -p $newRepoDir
cd $newRepoDir
git init --bare
echo "new repo: '$newRepoDir' created"
fiCreate the a new git repo if it does not exists. All that needs to be done is creating a directory to hold the repo and then calling git init --bare inside the newly created directory. This is the git command to initialize an empty repository.
The repos-delete.sh script is as follows:
#!/bin/sh
#deletes a git repo in your repo directory
#input a cli argument with the git repo to delete
set -e
. repos-setenvvars #set environment variables
if [ -z "$reposDir" ] || [ ! -d "$reposDir" ]; then
echo "exiting, can't find reposDir or env variables not set"
exit 1
elif [ -z "$1" ]; then
echo "exiting, no repo name input to delete"
echo "usage: 'repos-delete <name-of-repo-to-delete>'"
exit 1
fi
case "$1" in
*.git)
deleteRepoDir=$reposDir/$1
deleteRepoBackupHash=$reposDir/sha-$1.sha256 ;;
*)
deleteRepoDir=$reposDir/$1.git
deleteRepoBackupHash=$reposBackupDir/sha-$1.git.sha256 ;;
esac
if [ ! -d "$deleteRepoDir" ]; then
echo "exiting, git repo at: '$deleteRepoDir' does not exist"
exit 1
fi
#ask for confirmation that the use actually wants to delete the repo
read -p "Confirmation, would you like to delete the git repo at: '$deleteRepoDir' (y/n)? " choice
case "$choice" in
y|Y ) echo "Yes selected, deleting: '$deleteRepoDir'" ;;
n|N ) echo "No selected, exiting" && exit 0 ;;
* ) echo "invalid option, exiting" && exit 1 ;;
esac
rm -rf $deleteRepoDir
if [ -f "$deleteRepoBackupHash" ]; then
rm $deleteRepoBackupHash
fi
echo "git repo '$deleteRepoDir' has been deleted"This script again starts off by sourcing the environment variables. It then checks to make sure that a repo name was input into the script as a command line argument (else it exits and prints usage).
case "$1" in
*.git)
deleteRepoDir=$reposDir/$1
deleteRepoBackupHash=$reposDir/sha-$1.sha256 ;;
*)
deleteRepoDir=$reposDir/$1.git
deleteRepoBackupHash=$reposBackupDir/sha-$1.git.sha256 ;;
esacThis sets the variables containing the git repo directory to delete (as well as the backup hash to delete). Similarly to the create script this allows us to accept command line arguments that may or may not end in .git.
if [ ! -d "$deleteRepoDir" ]; then
echo "exiting, git repo at: '$deleteRepoDir' does not exist"
exit 1
fiCheck that the repo to delete actually exists. If it doesn’t exist then exit the script.
Now that all of the files needed for the server have been defined let’s go over how to create a deployment with NixOps. To create a deployment for our git server use the create command:
nixops create git-server.nix -d git-serverThis creates a deployment using our git-server.nix configuration named git-server. To build & deploy the configuration you’ll need to use the deploy command:
nixops deploy -d git-serverThe first time running the deploy command will provision a virtual machine using VirtualBox & then build/deploy the nix configuration to the virtual machine. After deploying you can use the info command to get info about the current deployments:
nixops infoTake note of the IP Address listed here as you’ll need it in order to connect to the server. In the future if you need to make any modifications to the configuration you’ll first need to edit the git-server.nix file and then run the modify command:
nixops modify git-server.nix -d git-serverRunning the modify command will only modify the configuration it will not deploy it. In order to deploy the newly modified configuration you’ll need to run the deploy command after the modify command.
Now that the server is running you should be able to ssh in as the root user and use the repos-create command to create a new git repository. After the repository is created you should be able to clone it on your local computer by running:
git clone git@<IP_ADDRESS_OF_SERVER>:<NAME_OF_GIT_REPO>Then you should be able to run all local git commands & then push the changes to the server.
In the future if you want to delete the git server you can use the nixops destroy -d git-server & nixops delete -d git-server commands.