Almost five years ago the University where I work suffered a fire in the main library building on campus. This building also houses, in the basement, the (then) primary data center. Two holes in the ceiling of the data center, which had been made to allow conduit runs, had not been properly sealed. We had a major water intrusion.
In the months that followed, as we worked to restore services and find ways to move out of the now compromised data center, we used VMware Cloud on AWS. Most of our production servers were virtual machines running in vSphere in our vBlock. With VMC we could “lift and shift” workloads from on-premises to AWS relatively quickly.
This spring we are working to upgrade all our Ubuntu servers to version 22.04. This required some
adjustments to our mod_cluster configuration and our Apache configuration. Fortunately we have a
test environment and an infrastructure automation tool to help make test and prod the same.
Our infrastructure automation tool is Chef. One of the features of Chef is the ability to “pin” the version of resources through the environment. All the nodes (Chef parlance for server) exist within an environment, with clever names like “test”, or “prod”. When we set out to migrate to VMC a new environment was created, called “prodvmc”. Catchy.
At about the same time, some version pinning occurred in the “prod” environment. Pinning that was never revisited. Pinning that was committed to the repository with no meaningful comments as to why the versions were pinned.
All of our Ubuntu 22 testing occurred, naturally enough, in the “test” environment. We never exercised the pinned versions. Also, apparently to reduce setup time in Vagrant, some resource definitions had code that said, “don’t do this if the environment is ’test’”.
Both of these decisions have produced some technical debt we now have to pay. The infrastructure automation code that is stepped around for the test environment hasn’t proved to be hard to overcome. Frustrating perhaps, but not insurmountable.
Unwinding the version pinning is a much larger debt. Our applications are Java-based web applications running in Wildfly application server. One of the resources that was pinned is the primary definition for our Wildfly clusters. A definition used by all of our major applications.
Not all of the servers belong to the “prod” environment. Slightly more than half belong to the “prodvmc” environment, where there was no pinning. For the rest, we now need to revisit four years of commits to determine what will happen to these production resources, when we remove the pinning constraint and update to the latest version of the infrastructure definition.
One measurement of an organization’s maturity, is how well things are documented. In our haste to move to VMC we made a decision to pin some versions, most likely thinking, “This is temporary for reason “x”, we’ll come back and address this once the move is completed.” And then we never went back. Not going back apparently hasn’t had too large an impact since all our applications are working. But not documenting the reason for the decision is going to cost us some time.
Lessons to learn:
I spent the better part of two work days this week mob programming. Solo programming is easy, you and a computer, and your favorite text editor. Pair programming (which I have never done) sounds like it would be interesting. Two minds, two differing approaches, one problem. Done right, it could be very empowering. Mob programming is what happens when there are three, four, and even five people on the Zoom call, watching one person “drive” and offering ideas, suggestions, and feedback.
I found it exhausting. It was very easy to lose track of the current thread, of the current approach to solving the problem. Depending on who was driving at the time it was very difficult to keep up with what they were doing. (Imagine an editor with 5 open tabs, each titled “UNNAMED” that the person is jumping back and forth between.)
We ultimately resolved the issue, but not efficiently, and not in a reproducible way. Maybe not reproducing it is actually a plus. I think with two, or maybe three people, you can focus on a problem and make steady progress. If one member of the group gets too close to the tree, at least one other person and pull them back to see the forest. Somehow with four people, and also with five, you end up with scattered approach. Two or three forests aren’t being seen for two or three trees.
If I find myself headed into a similar situation-two of us working and deciding to bring in another for assistance-I think the designated driver needs to be very up-front about directing what is going on. I also think it would help if the driver narrated what they were doing, even to the point of saying things like, “I’m going to open a new tab to pull up this code …”.
No more mob programming for me.
Recently at work I had to split a sub-folder in a repository into its own repository. I wanted to keep the commit history. My search online led me to Splitting a subfolder out into a new repository on the GitHub Dos site.
The instructions there are very good. In a nutshell what I did was this.
git filter-repo --path FOLDER-NAME/ command.In all the process took about 5 minutes. Not something I’ll do every day, but it is nice to know it is possible.
For the past ten years or so, my primary terminal emulator has been iTerm2. Recently I have been experimenting with other terminal emulators. The current test subject is Alacritty. Switching to a different piece of software has exposed a learned behavior that I’m not sure I’m willing to give up.
iTerm2 employs what I think of as a document model. You open the software once, but can have multiple “documents” open simultaneously. In this case each document is another terminal session. Since they are all children of the application you can cycle through them using the CMD-` keyboard shortcut.
On the desktop where I keep my terminal windows (multiple desktops is another topic for another day), I typically have three terminal windows open. In the upper left quadrant I have a window where I keep my daily log open in a text editor. The lower left quadrant has a window where my ICR client (Weechat) and Mutt email live, each in their own tmux window. Both Mutt and Weechat run on a small server in my home, so I ssh into that machine and attach to a tmux session which holds those two applications. The right side of my screen is a single terminal window where I do most of my work. I usually have several tmux session defined, each centered around either a long running task, or some recurring administration function I want to return to again and again.
Since iTerm2 uses a document model I can use the CMD-` shortcut to hop from terminal to terminal. Alacritty does not employ a document model. Each of the three terminal windows described above is a unique instance of Alacritty. In order to switch from one to another I need to use CMD-Tab. CMD-Tab brings up the application switcher, and sometimes there are several applications I have to tab past to get to the next Alacritty window. Furthermore, from the application switcher you can’t tell which Alacritty window you are about to open. When you CMD-` you are taken to the window so you know immediately which window you are at.
For now, I’m going to return to iTerm2 so I can keep my in-application window switching ability.
As a programmer / site reliability engineer / cloud infrastructure administrator I am very familiar with processes and procedures. Some are anecdotal and others are formalized. The best ones are automated, so that they are precisely repeatable.
My wife is having hip replacement surgery today. As with her first hip replacement last year, there are a number of processes and procedures at play, and a few checklists. We both appreciate the thoroughness of the preparation. Having each stage mapped out, knowing what to expect, and having people verify and explain as they go, all make the experience better.
One of the procedures requires no food or drink within six hours of the surgery. Usually they tell you nothing after midnight, regardless of the procedure time, as it is a definite, unambiguous time.
As a part of the pre-op preparation, there were several pills my wife needed to take. She is unable to swallow pills using water alone. We explained this prior to the first surgery to the hospital intake nurse, to the anesthesiologist who called the day before, and to the nurses preparing her for surgery. In order to swallow pills my wife use a small bite of soft food: a cracker chewed up, or a bite of banana.
On the morning of the first surgery the nurse presented her with a little cup full of pills and a small cup of water. We explained once again the inabilty to swallow with only water. So the pills didn’t happen.
We explained this requirement again during all of the pre-op visits and calls. During the pre-op anesthesiologist phone call yesterday, we again explained about the need for a small bite of soften food to swallow pills. This morning we explained that to the nurse. She offered a saltine cracker, which did all the pills to be taken. An hour later, as we were waiting in the holding bay outside the operating room, we were told that due to the cracker surgery would have to be delayed. By six hours.
The procedure wasn’t followed. Had the nurse contacted the anesthesiologist for today’s surgery and asked, it might have been allowed, or they may have said, “Nope, no food”. Either way the surgery would have happened at the original 9 am time. Since there wasn’t any communication to verify the use of a cracker, the surgery was delayed.
I understand and appreciate the importance of procedures and checklists, and that there are consequences for not following them. That the surgeon and anesthesiologist are paying that much attention bolsters my confidence in their focus and dedication to my wife’s care.
The consequence of not following the no food edict is a six hour delay. The irony is that the efficacy of the medication taken at 7 am with a bite of cracker will have worn off by the start time of the delayed surgery.
Whenever my wife or I sees something like this, we’ll say, “When do people think up stuff like this?”, to which the other will say, “Not between 9 and 5.”
This house is wonderfully situated to embrace its location. I especially like that different parts of the house have distinct views.
After watching 0 to LSP: Neovim RC From Scratch I wanted to experiment with my own Neovim configuration. Fortunately, someone tweeted a very nice way to have a scratch environment, totally separate from your current Neovim configuration, for experimentation.
Bonus tip: If you want to try @ThePrimeagen's 0-to-LSP without affecting your current nvim conf, just use a scratch container
— Vure (@vure89) December 16, 2022
docker run -it --rm -v {$PWD}:/root/.config/nvim -w /root/.config/nvim --net host alpine sh
(don't forget to `apk add neovim neovim-doc`)
🐴🤝
Start by creating a scratch directory on your host machine. Then using that directory run this command:
docker run -it --rm -v {$PWD}:/root/.config/nvim -w /root/.config/nvim --net host alpine sh
After the image is started, run
apk add neovim neovim-doc
to install Neovim and its documentation.
The Docker instance will start surprisingly quickly, and since it is mapped to a directory on your host machine, the configuration you make will be saved. I can see myself using this pattern for other experiements.