Last Tweets of the Krell
Many readers are no doubt familiar with Forbidden Planet, the 1956 documentary film about the Krell — a civilization that came to an unfortunate end just at the launch of what could have been their biggest achievement. Ever since the film’s release, xenoanthropologists have been stymied by a lack of source material on the Krell.
But on April 1, researchers from the OpenNMT institute announced a stunning success in translating previously undecipherable Krell messages, thanks to deep neural machine translation technology.
They note that the Krell language has three grammatical pluractionality markers, one for public speech addressed to everyone, one for cohort speech among a group of peers, and one for private speech between individuals. The researchers joked that these modes correspond to our Twitter, Slack, and text messaging. Krell names are unpronounceable for us, so we have chosen single-letter initials for the participants.
The Participants
P: The president of an organization that we will translate as KrellTech.
A: The alpha technical expert at KrellTech.
S: A site reliability engineer.
X: A user experience designer.
The Messages
P(to public) KrellTech is thrilled to announce this morning the launch of CognoMaterializer 1.0! The basic version of this service is available free of charge to all citizens of Altair IV. The possibilities are equal to the number ten raised almost literally to the power of infinity!
P(to cohort) Time to party! All KrellTech employees, friends, and family invited to the big bash at the Level 700 meeting room.
P(to cohort) Congratulations everyone, and special thanks to A for leading this project. Never have I seen a single individual successfully do so much, on their own, to create a complex project like this! The drinks are on me, A!
A(to cohort) Thanks, P! I’ll toast you for one drink but then I’ll have to crash — too many late nights bringing this project home! I’m fried!
Later That Night
P(to public) After midnight … we start day two of the CognoMaterializer era. A few reports of problems, due to user error. I promise we will have a fix by tomorrow for all affected users!
S (to cohort) @A, you better get in here fast. Some of these user reports look serious …
S (to cohort) @A, WHERE ARE YOU!!?? You don’t respond to messages, pages, or calls. I’m going to start a rollback to the previous version.
S (to cohort) @A, or @X, or anyone in @cognoteam, what’s up with the rollback script??? I tried to run it, and it just gave an error message.
X(to cohort) @A, why do you have unreviewed code from experimental running in production? I didn’t even know that was possible!!??
X(to cohort) I warned everyone not to cut the budget for user testing, but did you listen? Noooo, @P said we had no budget left. We brought a few users in for one-hour tests, but we never did the week-long home tests I specified as essential. We’re going to have to pull the plug on this whole thing.
S(to cohort) I tried the shutdown script, but the ping-and-restart daemons restart the servers faster than I can shut them down. Didn’t anyone try this before?? Can we contact our brick service provider and have them shut it down? Could we run around and pull all the plugs?
X(to cohort) umm, the brick is a 8000 cubic mile array of klystron relays powered by 9,200 thermonuclear reactors. I don’t think we can do a manual shutdown.
P (to public) Everyone, we’ve got a few minor issues — I recommend taking a pause from enjoying the CognoMaterializer until we get a quick update out. Just try not to think about anything bad. Or anything at all, really. And whatever you do, don’t go to sleep!!
X(to cohort) we’re 3-way-copulated
S (to cohort) we’re triple-3-way-copulated
Last Known Krell Message
S(to public) this is so bad … I don’t know if there is anyone left out there to see this. I’m sorry for the damage we’ve done to our civilization; I know we could have protected it if we had followed these practices:
- Many eyes. Don’t let one person work alone. All changes should be reviewed and approved by others.
- Understand use cases. The temptation is to concentrate on what you, as an engineer, do in building your service, but the truly important thing is what your customers will do when they use it.
- Engineers are judged by failure. Product designers are judged by how useful the product is when it is functioning. Engineers are judged by how gracefully the system degrades when it fails.
- Test what you build and build what you test. Make sure the use cases are tested, and that what you run is what was tested. Hermetic builds assure that you know where everything came from, and can reproduce it.
- Adversarial testing. Be devious in imagining what could go wrong, and fix it before it does. Hire red teams to try to uncover vulnerabilities.
- Software Release Checklist. Don’t release until you check every box: code coverage and tests passing; bugs fixed; feature checklist; security review; privacy review; fairness review; accessibility review; performance review; documentation review; whatever else is part of your development process. Yes, sometimes you will release software with known bugs, but that should always be part of an intentional decision to do so.
- Product Launch Checklist. Engineers often have the mindset of “if the software works, ship it.” But working code by itself is not a product. Before launching, have a checklist for the whole product: sales, marketing, communication, and support teams have to be trained and ready to go.
- Landing Checklist. Everyone remembers the date that our species first landed on our moon–the day they landed, not the day they launched. So it is with software; prepare for the landing, when the software is in the hands of many customers, not just for the launch, when it is in the hands of a few enthusiastic early adopters.
- Progressive rollouts. Trial the system with a small number of users before releasing to a large number.
- Monitor. Have enough people, and automatic systems, to gather feedback on how you are doing and sound an alert when there are problems.
- On Call: Make sure there is a schedule of qualified people on call to deal with problems, especially at the start and after any major change.
- Instant shut down and rollback. Make sure you have the ability to cleanly shut down the system if something goes wrong, and in case of error to roll back to the previous working system. Bonus points if you can selectively cherry pick properly-working improvements and reliably add them to a previous version.
- Safe User Interface. Make it hard for users to do unsafe things. P insisted on a “one-thought” interface (and even patented it). It is safer to require a specific prompt to initiate action (“OK, Cogno”), to have the system echo back its interpretation of the user’s command, and to use a confirmation dialog for actions with big irreversible effects (“Are you sure you want to conjure a Monster from the Id?”).
- Culture. X and I (and others) had concerns all along, but the culture encouraged us to keep quiet about problems, discouraged discussion, and didn’t have a process to fix the problems.
- Practice emergencies. Don’t wait until a real emergency to see how good your response is; practice in advance.
- Premortems. Before launch, ask everyone on the team “Imagine the project turns out to be a miserable failure. What could have caused that?” Then fix the problems before they happen.
- Postmortems. When something goes wrong, analyze and understand it after the fact … that is, if there is anyone left alive.
Articles in this Series
April 1 2016: Go, Dog. Gauguin! The P. D. Eastman Story
April 1 2017: Last Tweets of the Krell
April 1 2018: Functional Lifestyles
April 1 2020: Hope I Incorporate Before I Get Old
April 1 2021: Hamilton: The Mathematical Musical
April 1 2023: Packet-Switching Patent Revealed!
April 1 2023: New Python Operators!