Description:
Sometimes I remember my practice in a school workshop, that time I was trained to process wood and metal. It was good time. Those days I understood one important thing - if you have bad tools - your work will not bring you pleasure.
What have been changed:
- Storage: RAID 50 and ZFS have been replaced by RAID 10 and LVM/EXT4/XFS/BTRFS. 1T SSD was added.
- Scripts: All scripts now use Hashicorp Vault.
- IaaS: Ovirt has been replaced by Openstack completely.
- Automation: Stackstorm has been replaced by Ansible Tower.
Storage:
As I wrote before, I have a server with a RAID controller (8 HDD 1TB disks what gives 4TB of usable capacity). Long time I used RAID 50 with ZFS (send/receive/snapshots/compression) in my day by day work. It was reliably, but slow.
Now I use:
- RAID 10 for common things and 1TB SSD for speed critical operations.
- LVM snapshots (Ovirt doesn't support LVM directly as a storage type, ZFS filesystem cannot be used because of O_DIRECT syscall checking (ZVOL only)).
- EXT4 for usual things.
- XFS for Openstack Swift.
- BTRFS with compression for long lived things.
All of these permutations gave me nice improvement of disk operations. Elasticsearch and other data now placed on 1T SSD.
Scripts:
I have a lot of scripts and tools, majority of them live in a private Gogs repository, but some lies in "~/.local/bin". Some of those scripts contain private data like: passwords, passphrases, encryption keys etc.
Very long time I promised to myself to change situation and start using some kind of secret management solution (instead of just DAC. It happend. Now every scripts in my infrastructure (that contains private data) use Hashicorp Vault for sensitive data retriving.
As a storage backend I use etcd and Vault server itself was installed:
- On two working laptops.
- On one server.
- On Raspeberry Pi 2.
It means that I always have access to my secret storage in spite of inaccessibility some of nodes. In addition I started using Jenkins plugin for Hashicorp Vault.
IaaS:
I guess it was 2012 when I started to use Ovirt. Since that time I installed Ovirt on many servers and majority of them still work, but my home server doesn't have Ovirt anymore, I have replaced it with Openstack. Why ? There are some important reasons:
- Lack of orchestration such as Heat.
- Lack of backup solution out of the box.
- Lack of flexible network features.
In my work I use many environments with different services, which change very often, for that purpose most suitable way - to use orchestration that provides by Heat.
During Ovirt usage I made backups of virtual machines with help of ZFS snapshots and "send/receive" features. Now I use Cinder "backup" feature which save virtual machines to Swift object store (separate replicas on different storages).
Not necessary to describe rich of network features of Openstack, but I should mention that I use VXLAN, VLAN and bridges.
Despite these shortcomings Ovirt is more convinient and easy to use than Openstack and most suitable for classic IaaS than Clouds.
Automation:
One and half year ago I wrote a note about Stackstorm, all this time I have been successfully used Stackstorm in my work. Mistral workflows - it's a great tool for complex tasks. If everything is so good then why I replaced Stackstorm with Ansible Tower:
- Doesn't have convinient way to store and operate sensitive data.
- Lack of workflows visualization in free version.
- MongoDB as a dependencie.
I have changed all workflows and tasks to Ansible playbooks and now I have an uniform and feature rich (Ansible modules) solution for my daily work. In addition I bought a keypad that I use as a fast launcher of Ansible Tower tasks.
As a "conclude":
Good tools and uniform methods give nice results and make our lives more simple and handy :)