Posts on Random Stuff - Too Many Machines

The state of post-quantum cryptography at the end of 2024

Sat, 14 Dec 2024 21:00:00 +0100

The recent hype about Google Willow inspired me to recheck what’s happening in the world of post-quantum cryptography.

You probably already heard the alarming news - all of our encryption is becoming useless. The reality is not as tragic. What is true is that most asymmetric algorithms (those with public and private key) are very weak against quantum computers. They are widely used for two purposes:

Most encrypted protocols (that utilize a much faster symmetric encryption for the vast majority of the traffic) randomly generate a key at the beginning of the session and use asymmetric encryption to send this key to the other party. This is called KEX (Key EXchange) or KEM (Key Encapsulation Mechanism), depending on the exact way.
Digital signatures, for example those that confirm who’s the owner of a TLS certificate or a publisher of a software package.

But the alarming articles routinely fail to mention the good news:

We don’t have a quantum computer that can break a modern encryption (although it doesn’t exist, it already has a name: CRQC, Cryptographically-Relevant Quantum Computer), current ones are either too small or too unstable. We are getting closer, experts predict CRQC will be available in the next few years.
Symmetric algorithms, which do almost 100% of the work, are safe. There are some quantum-based attacks that slightly reduce the effectiveness of the cyphers, but simply increasing the key length (which we had to do anyway due to the widespread use of GPU computing for cracking encryption) mitigates the risk.
Hash algorithms (used, among other things, for storing encrypted passwords) are also safe. Again, the real danger came from GPUs, but this has already been taken care of.
Current software can be easily modified for post-quantum secrecy. It’s modular, algorithms are changed every few years anyway, due to increase in CPU/GPU power and sometimes when the weaknesses are found.
We already have some quantum-resistant asymmetric algorithms in use and adoption is rising very fast - according to Cloudflare, from 2% in January to 13% in December 2024 in the traffic they see.

Official standards

In August 2024 NIST, (National Institute of Standards and Technology, a US government agency) published the first 3 official standards and more are coming:

FIPS203 for key exchange.
FIPS204 and FIPS205 for digital signatures.

Of those, at least FIPS203 was already (relatively) widely used, although not always called that. NIST doesn’t develop new algorithms, it chooses from the submitted proposals. The one that became FIPS203 is mostly known as Kyber or ML-KEM. Adversaries can (and do) harvest encrypted data now to decrypt it in future (just as you can easily crack 1990s encryption with a PC of today), therefore it’s a good idea to move to post-quantum standards as soon as possible.

Securing digital signatures against the quantum computing is not as urgent. Once CRQC are available, they will be a huge threat. Quick refresher: signatures can be checked with a public key, but to make them you need a private key, which should be well guarded. Obtaining the private key from the public key is, for all practical purposes, impossible with a classical computer (as in: even if you use all the world computing power to break just one key, you’d still need more time than the life expectancy of the Sun). But it’s fast with a quantum computer. However, it’s not much use retroactively. You could use a CRQC to decrypt TLS traffic harvested now, that’s a real threat for some use cases. But forging 2024 certificate in 2034 will be useless. We just need to switch to quantum-safe algorithms before CRQC.

Client support - mostly done

Chrome has implemented Kyber/FIPS203 since version 115 released in August 2023. That also includes all derivative browsers such as Edge, Opera and Vivaldi. Firefox added support in January 2024. The only major browser that doesn’t support it is Safari.

Messaging apps such as Signal, WhatsApp and iMessage also support at least one form of post-quantum cryptography.

Server support - not as good

I found it peculiar that server-side lags behind client-side in security standards. But Google decided to introduce Kyber early, other browsers followed and since they aggressively auto-update (for good reasons), pretty much all browsers support a quantum-resistant key encapsulation, and most client-side software uses browser components to do the connection. But, until servers get the support, new protocols won’t see much use. And servers tend to run older software, also for good reasons: new algorithms mean new security risks. It would be ironic to increase security to the future theoretical threats only to decrease to the current threats - and yet it happened. Few years ago SIDH was a top contender for a quantum-resistant KEM, but it was broken in 2022. And not by a quantum computer, not even by a powerful GPU, a puny single-core CPU was enough - yikes.

So where do we stand now, in December 2024?

OpenSSL, the library responsible for something like 50% of the TLS support [citation needed], has a policy of only supporting official standards. They are now working on adding FIPS203, but current versions are missing it. However, OpenSSL can be extended with “providers” - external libraries that add additional crypto algorithms.
Go language, probably the second most common technology in the server world, has its own crypto libraries and you guessed it: official versions don’t support Kyber.
OpenSSH supports FIPS203, but only in the latest version 9.9, released in September 2024. It is, of course, too new to be included in mainstream Linux distros (Kali is a notable exception, though I’m stretching the definition of mainstream).

Cloudflare to the rescue

Cloudflare is one of the early adopters. The company provides many services, including TLS termination. Incidentally, I use it for my photo gallery. Nice to know that my completely insignificant website that only contains 100% public information is well protected against state-level adversaries of the future! They also publish stats and articles about the topic.

Open Quantum Safe

Open Quantum Safe or OQS is a research project of the Linux Foundation. One of their works is liboqs, a C library providing a large number of quantum-resistant algorithms, plus integrations into current software, such as OpenSSL or OpenSSH. If you want to get started with it, the fastest way is compiling a provider for OpenSSL. It is available on the GitHub repo.

I checked a few recent Linux distro and none had a liboqs package. Why, if it’s free, opensource, and comes from a reputable source? One of the reasons might be, ironically, security. The goal of the project is to help with research and software testing, it’s not intended for production use. The library contains numerous algorithms, not only the official ones or top contenders, but also those that aren’t well tested, or even those that were rejected. Which means a large attack surface. In fact, there are known security vulnerabilities in liboqs. If you want to help with testing, then go ahead. If you want to have a full-featured client, be careful. But if you want to protect your server - don’t do it, you will be introducing more vulnerabilities than you’re fixing.

How to check common software for post-quantum support

Browser: go to Cloudflare Research Post-Quantum
SSH client: type ssh -Q kex and look for a line starting with mlkem
SSH server: connect with -vv option and check “peer server KEXINIT proposal” (as a bonus “local client KEXINIT proposal” will show what your client supports). Alternatively, nmap --script ssh2-enum-algos -sV -p 22 your.ssh.host. In any case, look for mlkem.
OpenSSL: openssl list -kem-algorithms and look for mlkem or kyber.

Secret files in Git - transparent encryption and history rewriting

Thu, 10 Oct 2024 16:00:00 +0200

I accidentally posted on GitHub some files I shouldn’t: my input files from Advent of Code. I thought it doesn’t matter since the files are generated for each player, but AoC author specifically prohibits this.

I don’t think it’s a big deal: few people visit my GitHub anyway. But let’s solve it. I might need that skill one day for a more serious problem.

This is a two part problem:

I still like to keep my input files on GitHub. I use several computers and sharing files through the repo is convenient. I just have to encrypt them.
But first, I need to remove the files I posted. It’s not enough to delete them, they would still be available in the git history.

How to really delete files from git

You need an extension called Filter Repo. On Debian or Ubuntu, you can simply run sudo apt install git-filter-repo.
Make sure the state is as clean as possible: no changes, untracked files, open PRs, nobody pushing other changes.
Next, get a fresh clone of the repo in a new place - for two reasons. Filter Repo insists on working on a fresh clone. And, you want a copy anyway in case something goes wrong.
Now, the main part: git filter-repo --path <path to the file or directory> --invert-paths. Don’t forget the last argument - the default is to keep only files matched by the filter, but you want to delete those and keep everything else. Instead of --path you can use --path-glob or --path-regex to match multiple files. You can also add --dry-run to see what files would be deleted. Verify that you deleted everything you wanted to and nothing else.
Filter Repo also intentionally removes links to remote repositories. There are two ways to deal with it:
- If the repo is used by many people (or worse, CI systems), the recommended option is to create a new remote repo and inform everyone to switch to it (preferably by cloning a fresh copy).
- Another possibility is to set the remote repo to the same value as it was before and push the changes (with –force) back to the original upstream repo. Be aware that the files you wanted gone will be deleted from your local copy and from the upstream, but any other copy of the repo will still have them, plus the original, incompatible history. So, all other copies will need to be made clean, either with some heavy git magic, involving rebasing and a lot of –force, or better deleted and cloned again. Since I’m the only user (that I know of), I went with the second option.

The whole set of commands looks like this:

 git clone git@github.com:igorwaw/advent22.git
 cp -a advent22 advent22.bak # backup
 cd advent22
 ls day*/*-input.txt
 git filter-repo --invert-paths --path-glob 'day*/*-input.txt' --force
 ls day*/*-input.txt # verification - files are gone
 git remote add origin git@github.com:igorwaw/advent22.git
 git pull # to get the remote branches - most of my repos use main, older ones still use master
 git branch --set-upstream-to=origin/main main
 git pull # verification - this is expected to fail as I have incompatible changes
 git push --force # rewrite the remote
 git pull # verification - no incoming changes

A quick check on GitHub - the files are gone and there are no new commits, no traces of what just happened. This is scary stuff, a proper removal from history. Luckily, the real world history can’t be so easily rewritten (OR CAN IT?).

How to transparently encrypt files in the repo

There are millions of ways to encrypt files, but the best solution for this use case is something completely transparent: files are encrypted on git push and decrypted on git pull.

The tool called git-crypt fits perfectly. Easy to get (simple sudo apt install git-crypt), fast and secure (uses symmetric encryption with AES-256 by default, multi-user setup with GPG keys is also possible). It has some limitations too:

By far the biggest limitation: no good mechanism for revoking the key. If you simply change it, only new commits will use it, old versions of files will still be encrypted with the old key. Reencrypting old versions is doable, but complex (see above).
It doesn’t hide any metadata. Filenames, modification dates etc. are plainly visible.
Some git GUIs fail with git-crypt.

None of these is relevant for me and advantages clearly outweigh the potential issues. I don’t need to hide metadata and if the key leaks, I can just delete and recreate the whole repo.

First, go to your repo, generate and export the key:

git-crypt init
git-crypt export-key ../git-crypt-key

Next, create a file .gitattributes. It’s similar to well-known .gitignore, but informs git to use other transformations than simply ignoring the file. For this case, I needed the following content:

 day*/*-input.txt filter=git-crypt diff=git-crypt

Of course I had to put the input files back. Then, git-crypt status -e shows what files will be encrypted:

➜ advent22 git:(main) ✗ git-crypt status -e
 encrypted: day1-9/1-input.txt
 encrypted: day1-9/2-input.txt
 encrypted: day1-9/3-input.txt
 encrypted: day1-9/4-input.txt
 encrypted: day1-9/5-input.txt
 encrypted: day1-9/6-input.txt
 encrypted: day1-9/7-input.txt
 encrypted: day1-9/8-input.txt
 encrypted: day1-9/9-input.txt
 encrypted: day10-14/10-input.txt
 encrypted: day10-14/11-input.txt
 encrypted: day10-14/12-input.txt
 encrypted: day10-14/13-input.txt
 encrypted: day10-14/14-input.txt

You can encrypt them with git-crypt lock, but there’s no need. Simply add the new files, commit them and push to the remote repo:

git add .gitattributes day*/*input.txt
git commit -m "encrypted inputs"
git push

Now, go to GitHub and see what was committed - the files will look like this: GITCRYPT r5”4Â)ùKåæÜ¿„ÚÊ¶.

Using the encrypted repo on another machine

Copy the exported keyfile to another machine in a secure way (eg. using sftp or rsync -e ssh). Place the keyfile in any convenient place outside the repo (one directory level up is convenient), then run the command: git-crypt unlock ../git-crypt-key.

Reusing the encryption key

I’m going to use the same key for all of my Advent of Code repos. No, it’s not the same as using the same password for many websites. These are all under my control, just split into several repos for convenience. How do you do that? Simple: the same way as unlocking the repo on another machine in the previous step.

cd ../advent15
git-crypt unlock ../git-crypt-key

There’s nothing to unlock yet, but the command also initializes the hidden git-crypt files if they don’t exist, only it doesn’t create the new keyfile. Confusing, yes. Then, create the .gitattributes, add, commit and push files exactly the same way as in the previous repo.

Switching from Docker to Podman

Sat, 21 Sep 2024 16:00:00 +0200

Many distributions have switched from Docker to other container engines. There were different reasons for it - licensing or design choices - but for years I ignored the trend. I thought: container engines are supposedly 99% compatible with Docker, after all they all claim to support OCI standard, but somehow I always run into this 1% of incompatibility. So I just always installed Docker to avoid the issues.

Few months ago the principal engineer at my workplace, who’s a big fan of Podman, encouraged me to give it another try - to spend a few hours learning about the differences before I decide that something doesn’t work with Podman. I quickly discovered things that are obvious to any Podman user: while the command line tool is compatible and you can do “alias docker=podman”, there are some deeper differences. Once you learn them, you can do all the same things as with Docker, plus some more.

Short version: whenever there’s a choice between convenience and security, Docker defaults to convenience and Podman to security.

Container registries

Let’s start with the simplest one. Check any tutorial on Docker and it will start with “docker run -it ubuntu” or something similar. Such command will probably fail with Podman. Docker tries to run a locally available image, if it doesn’t find one, it downloads it from docker.io registry. You can point it to another registry by giving full path, but that’s the default one.

Anyone can upload their software to docker.io registry. Obviously it has some good sides - a huge range of images to choose from and a place to store your own stuff. But that also means you can accidentally download some low quality code. Or worse, malicious code that steals your data. Many companies forbid the use of public container registries.

Podman has no default container registry (unless your distro already changed the default configuration). You can configure one (or multiple) in /etc/containers/registries.conf, it might be your company’s internal server, might be docker.io or another public one eg. quay.io. Or you can provide full path: “podman run -it docker.io/ubuntu”. Or you can configure shortnames only for specific images - for example Ubuntu does that with their Podman package, meaning that “podman run” will work with ubuntu, rhel, alpine, python and many other well-known images, but not with some random stuff found on the net. All of these don’t prevent the problem of downloading the bad stuff, but it makes it a bit harder to do by accident. If you don’t agree, it’s easy to change the default to the same as Docker’s.

Networks and pods

Let’s now move from Docker to Docker Compose tutorial. It usually involves running several containers, let’s say: frontend, backend and database. Each one can connect to others using their names. That won’t work with Podman - unless you inform Podman that this is your intention. For example, by placing them in the same pod.

The concept of pods comes from Kubernetes. Pods contain several containers grouped together - and they obviously are supposed to communicate. They can also be started/stopped together. But they really show their strength when you move to Kubernetes - see a few points below.

There are also other ways to connect the containers, you’re not forced to use pods. You can use the host’s hostname if the ports are exposed. Or configure a private network, they have name resolution. Or configure a service mesh or a discovery service. Or, if you insist, you can enable dnsname plugin on the default network (the one simply called podman). Podman is just missing one shortcut option that’s present in Docker because, again, it makes it harder to do insecure things by accident.

Rootless

The biggest difference between Docker and Podman. By default, Docker runs as root. Even if you run docker command as a user, it talks with the daemon that runs as root - and so the containers are started as root. Podman, on the other hand, really runs with the privileges of the user who invoked it. This has some serious implications:

The good: Podman is much more secure by design. If Docker fails to properly isolate the container (and there were such vulnerabilities in the past), the container can get full root rights to the host system and other containers. With Podman, the worst case scenario is getting access to one user account.
The bad: some containers need more than 1 user id. Podman can provide a full set of 2^16 ids using subids. No big deal, but: you have to have a proper configuration in /etc/subuid and /etc/subgid - with any modern distro and the standard way of creating accounts, you usually do, if you do something uncommon you might have to manually edit the file. And if your containers bind mount directories from the host filesystem, you may run into permission problems. Again, quite easily fixable with “podman unshare chown” or “:U” option for the volume, but more than one person ran into this and decided that Podman doesn’t work.
The ugly: some, but very few containers, really need extra privileges. If that is the case, you know what you’re doing and you’re accepting security implications, you can simply run podman as root. You’re opting out of extra isolation, but at least it’s for the selected few containers, not all of them. Even better, you can fine-tune the security options by adding and removing capabilities and seccomp filters, it’s not only the choice between full root access and nothing.

Daemonless

Podman doesn’t run a daemon by default. When you run a podman command, it will do what you requested (eg. run a container, delete an image, query for information) and simply exit. Again, security - no huge binaries running all the time, no exposed ports. You can choose to run a daemon if it’s needed for better Docker compatibility eg. with a 3rd party tool.

Systemd for starting containers

Docker doesn’t play well with Systemd. It’s the design choice - both tools try to configure cgroups and namespaces, restart services etc. Docker developers plainly refuse to cooperate, they want complete control. On the other hand, Podman authors decided it’s not the job of the container engine to autostart services. It’s either done with an orchestrator such as Kubernetes, or, for running on a single machine, you can just use the same software that starts other services, which is systemd.

Just one command, “podman generate systemd” will create the config file for autostarting your service. And you can even run it as user (yes, systemd supports that). Instead of autostarting at boot, you can also attach it to a socket and only activate it to handle incoming connection (if you’re as old as me, you remember inetd doing the same thing).

Fine print: “generate systemd” is now deprecated and Podman wants you to use quadlets. Quadlets are more powerful, but also more complex, which spoils my narrative. But the old command still works.

Systemd inside the containers

The most common way to use containers today is to isolate a single application. One container, one process, if it exits, the whole container exits (and possibly gets restarted). But there is another approach - treat container as something like a VM, running multiple daemons. Anything in between is also possible.

Personally, I’m in a single process camp. It’s been several years since I ran multiple processes (and that was because I didn’t know how to handle multi-container setup). But I think it’s better to have a choice. If you need to run systemd inside the container, Podman will detect it and automatically set the environment properly.

Kubernetes

Kubernetes used to use Docker, but switched to other container engines (CRI-O by default). Yet it’s Podman that works better with it.

The idea is: you use Podman to develop and test software and then you can easily run it in production on Kubernetes. Podman, unlike Docker, has a concept of pods. If you’re ready to move your software from a single machine to Kubernetes cluster, you can use “podman generate kube” to create the YAML file in the Kubernetes format. Since the format is not easy to create by hand, it saves a lot of time.

But there’s more. What if you have some software already running on Kubernetes, but want to test it on a single machine? There’s “podman play kube” which reads Kubernetes YAML. Of course it only accepts a subset of options as Podman can’t do scaling and multi-node deployment.

Running IBM Storage Scale on VMs

Thu, 23 Nov 2023 19:00:00 +0100

IBM Storage Scale (formerly Spectrum Scale, formerly GPFS) is a distributed file system often found in HPC clusters, Machine Learning platforms etc. It can scale up to an unimaginable size, it can provide higher throughput than any physical drive, it allows concurrent access from many nodes in the cluster. It supports distributed locking. It has all the standard features of Unix filesystems such as quotas or ACLs. Nodes and disks can be added and removed on the fly. There’s no single point of failure. It can also be accessed with S3, NFS or Hadoop compatible interface. And that’s not even a full list of features.

Which also means it’s not something you would run at home, not even a usual datacenter. Unless, of course, you need a playground for learning or testing. I felt quite uncomfortable running commands on a production system, that stored some petabytes of data and was used by thousands of people, without testing them first.

Creating VMs

I used Vagrant with a default provider (Virtualbox). Vagrant makes it easy to launch VMs for test environments. No need for configuration and installation, just vagrant init name-of-your-image ; vagrant up and you’re done. That is, if the default configuration works for you. If not, then a simple modification to the Vagrantfile takes only a few minutes, and that’s including time spent on searching the web for examples.

It’s getting a bit tricky if you need to test a distributed system:

the syntax is a little different for multiple VMs in one file,
you should probably configure a private network,
if you need port forwarding (eg. for SSH access), you need to use a different port for each VM.

I configured one client and three servers, sr1, sr2 and sr3. Servers both have extra hard drives in addition to the standard system drive. All machines have 2GB of RAM and 2 vCPUs (except the first server which needs more power), all connect to the same private network. All use the same operating system. Spectrum Scale supports several versions of RHEL, Ubuntu and SLES, it should work with other distributions, but no guarantees, and you’d be forced to use manual installation.

Here is my Vagrantfile. It’s not the initial version, that’s where I arrived after some experiments.

Vagrant.configure("2") do |config|

 config.vm.define "sr1" do |sr1|
 sr1.vm.box = "ubuntu/jammy64"
 sr1.vm.hostname = 'sr1'
 sr1.vm.provision "shell", path: "boot-tasks.sh"
 sr1.vm.disk :disk, size: "1GB", name: "extra_storage1"
 sr1.vm.disk :disk, size: "1GB", name: "extra_storage2"

 sr1.vm.network :private_network, ip: "192.168.56.101"
 #sr1.vm.network :forwarded_port, guest: 22, host: 2021, id: "ssh"

 sr1.vm.provider :virtualbox do |v|
 v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
 v.memory = 4096
 v.cpus = 4
 end
 end

 config.vm.define "sr2" do |sr2|
 sr2.vm.box = "ubuntu/jammy64"
 sr2.vm.hostname = 'sr2'
 sr2.vm.provision "shell", path: "boot-tasks.sh"

 sr2.vm.network :private_network, ip: "192.168.56.102"
 #sr2.vm.network :forwarded_port, guest: 22, host: 2022, id: "ssh"
 sr2.vm.disk :disk, size: "1GB", name: "extra_storage1"
 sr2.vm.disk :disk, size: "1GB", name: "extra_storage2"

 sr2.vm.provider :virtualbox do |v|
 v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
 v.memory = 2048
 v.cpus = 2
 end
 end


 config.vm.define "sr3" do |sr3|
 sr3.vm.box = "ubuntu/jammy64"
 sr3.vm.hostname = 'sr3'
 sr3.vm.provision "shell", path: "boot-tasks.sh"

 sr3.vm.network :private_network, ip: "192.168.56.103"
 #sr3.vm.network :forwarded_port, guest: 22, host: 2023, id: "ssh"
 sr3.vm.disk :disk, size: "1GB", name: "extra_storage1"
 sr3.vm.disk :disk, size: "1GB", name: "extra_storage2"

 sr3.vm.provider :virtualbox do |v|
 v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
 v.memory = 2048
 v.cpus = 2
 end
 end



 config.vm.define "client" do |client|
 client.vm.box = "ubuntu/jammy64"
 client.vm.hostname = 'client'
 client.vm.provision "shell", path: "boot-tasks.sh"

 client.vm.network :private_network, ip: "192.168.56.104"
 #client.vm.network :forwarded_port, guest: 22, host: 2024, id: "ssh"

 client.vm.provider :virtualbox do |v|
 v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
 v.memory = 2048
 v.cpus = 2
 end
 end

end

Vagrantfile references another file, boot-tasks.sh, which needs to be in the same directory. You can remove this line from Vagrantfile for now, we’ll use it in a few minutes. Run the first VM with vagrant up sr1. We’ll use it as the installer machine. Other VMs can be started later.

Preventing installation errors

If you’re running the VMs on your own computer, you might be tempted to give them less resources. Unfortunately, GPFS is a memory hog. Just the main daemon gpfsd consumes 1.2GB of RAM at startup. You need at least 2GB of RAM on all nodes, I also recommend to assign 4GB to one of the nodes and use it for running GUI and installer. That means that for 3 servers and client you need at least 10GB - a bit tough if your computer only has 16GB. But if your nodes don’t have enough memory, daemons will fail to start or crash, what’s worse, they might not even show any meaningful error message.

I also had problems, quite ironically, with storage performance. I’m running several VMs out of one hard drive (not even SSD). Linux block layer by default timeouts if it can’t finish a disk operation in 30 seconds. A reasonable value for a physical system, but if several VMs are doing write-heavy operations, it might not be enough. I’d rather wait a bit more than risk data corruption. I prepared a script time.sh and put it in the same directory as Vagrantfile. I configured Vagrant to run it automatically while rebuilding the VMs, I can also run it manually if I need to change the value.

#!/bin/bash

TIMEOUT=600

echo "Increasing disk timeout"

for f in /sys/block/sd?/device/timeout; do
 echo $TIMEOUT >"$f"
done

Getting Storage Scale

Storage Scale Developer Edition is available for free, but you need to register for an IBM account and accept countless license agreements. You’ll get a 1.7G zip file. Put it in the directory where you have your Vagrantfile, it will be available for VMs under /vagrant. Connect to the first server with vagrant ssh sr1. Unzip and add executable bit to the installer: chmod a+x Storage_Scale_Developer-5.1.9.0-x86_64-Linux-install and run it with root privileges: sudo ./Storage_Scale_Developer-5.1.9.0-x86_64-Linux-install.

It will list a few installation options. First one is:

 To install a cluster or deploy protocols with the IBM Storage Scale Installation Toolkit:
 /usr/lpp/mmfs/5.1.9.0/ansible-toolkit/spectrumscale -h

Ansible? Now you’ve got my attention! Last time I created GPFS testbed I installed the packages manually, but I like this option more. Let’s read the documentation for the IBM Storage Scale Installation Toolkit.

The guide lists some requirements:

obviously, we need Ansible on the machine that will start the installation: sudo apt install ansible
we need to resolve hostnames of all nodes to IPs, either a local DNS or /etc/hosts,
installer needs to be run as root,
we need password-less SSH (as root! abomination!),
we need to install make, C, and C++ compiler - on all nodes, including the client

Configuring VMs

Connect to the first server with vagrant ssh sr1. Edit /etc/hosts by adding these lines:

192.168.56.101 sr1
192.168.56.102 sr2
192.168.56.103 sr3
192.168.56.104 client

Copy the file to /vagrant. Then, generate a password-less keypair. Now, we’re going to do something against the normal security rules: share the same keypair. It’s fine for a temporary test environment only available on the home LAN. Copy both key files to /vagrant.

ssh-keygen
cp /etc/hosts /vagrant
cp ~vagrant/.ssh/id_rsa* /vagrant

Next, we’ll add a script so Vagrant will automatically configure all VMs. It will set the ssh keys, install required packages, and remove unattended-upgrades (this daemon has a habit of getting in the way of package installation). In the same directory where you placed Vagrantfile, add file boot-tasks.sh with the following contents:

#!/bin/sh

mkdir -p ~vagrant/.ssh
cd ~vagrant/.ssh
cp /vagrant/hosts /etc
cp /vagrant/id_rsa* .
cat id_rsa.pub >> authorized_keys
chown -R vagrant:vagrant ~vagrant/.ssh
cd /root/.ssh
cp ~vagrant/.ssh/* .
chown root:root *
apt -y remove unattended-upgrades
apt update
apt -y install g++ make
. /vagrant/time.sh

Restart machines with vagrant reload. Then try to ssh from root@sr1 to all VMs, including sr1. It should work.

Installing Storage Scale

First thing is to prepare the installer node. All commands below require root privileges:

cd /usr/lpp/mmfs/5.1.9.0/ansible-toolkit
./spectrumscale setup -s 192.168.56.101

First server will have a GUI and an admin role (-a -g). All servers will have a manager role, NSD role and will form quorum (-m -n -q) . Client is a node without any role (no extra arguments).

./spectrumscale node add sr1 -m -q -n -a -g 
./spectrumscale node add sr2 -m -q -n
./spectrumscale node add sr3 -m -q -n
./spectrumscale node add client
./spectrumscale node list

NSD means Network Shared Disk. There are several possible cluster topologies for Storage Scale, many of them require some kind of shared storage. We’re going to use the simplest form, where each NSD node will use its local disks. Let’s define the disks now. And at the same time, define the filesystem that uses all these disks.

./spectrumscale nsd add -p sr1 /dev/sdc /dev/sdd -fs filesystem1
./spectrumscale nsd add -p sr2 /dev/sdc /dev/sdd -fs filesystem1
./spectrumscale nsd add -p sr3 /dev/sdc /dev/sdd -fs filesystem1

Now, check the filesystem configuration:

./spectrumscale nsd list
./spectrumscale filesystem list

Last thing to configure. By default, Spectrum Scale will send information to IBM, but this function is not configured. We need to either provide information about the customer, country etc. or simply turn it off with ./spectrumscale callhome disable.

We’ve got nodes, disks and filesystem, ready for deployment. But first, let’s save VM snapshots in case we mess up something and need to revert, by running this command on the host: vagrant snapshot save before-installation.

Finally, run ./spectrumscale install. If all goes well, you might go for lunch at this point as the whole process will take at least 30 minutes. More likely, it will fail at some point. IBM Storage Scale Installation Toolkit runs some pre-checks and creates a temporary repo during installation, unfortunately it needs to repeat it during every attempt and it takes (on my slow inadequate hardware) about 10 minutes. Very annoying. Further tasks are more Ansible-style: if something fails, fix the problem and run the installer again, tasks already done will be checked and skipped, failed tasks will be attempted again.