Changing State Safely

I've started using a pattern at work that has been very helpful for me when it comes to making state changes, and I wanted to share my short thoughts. State changes are inherently dangerous in software because you produce an observable side effect. Bugs in this area can often be costly errors. For all code below I'll be using Python as that's what I've been using at work, and also because it's quite easy to read - I'll even provide type annotations for you!

Consider this short snippet from a simple service that might implement some bank transaction logic, assuming the user will pass in well typed inputs:

import json


class BankService:
    def __init__(self) -> None:
        self.accounts: dict[str, float] = {
            "alice": 1000,
            "bob": 500,
        }
        self.logs: list[str] = []

    def transfer(
        self,
        from_acct: str,
        to_acct: str,
        amount: float,
        memo: Any,
    ) -> None:
        # check for potential errors
        if from_acct not in self.accounts:
            raise KeyError(f"Account {from_acct} not found")
        if to_acct not in self.accounts:
            raise KeyError(f"Account {to_acct} not found")
        if self.accounts[from_acct] < amount:
            raise ValueError("Insufficient funds")

        # remove money from this account
        self.accounts[from_acct] -= amount

        # log
        log_entry = {
            "from": from_acct,
            "to": to_acct,
            "amount": amount,
            "memo": memo,
        }
        log_str = json.dumps(log_entry)
        self.logs.append(log_str)
        print(log_str)

        # add money to other account
        self.accounts[to_acct] += amount

On first mention of a bank service, the obvious error is moving money before potentially failing code. However in this case, the logic is in the right order and the transaction with the logging only happens after we confirm the entire interaction is valid.

The true source of failure in this program actually comes not from any of the transaction logic, but the logging. Even with users only passing in well formed input, the Any annotation on the memo means in theory, a user could do something like what is shown below without breaking any type annotations:

class Message():
    pass

bank = BankService()
bank.transfer("alice", "bob", 50, Message())
# error! a class can't be serialized! ^

Now we have a dangerous case where if our logging fails, we have already moved money out of an account, but not into any other account! This showcases one of my first realizations - that operations which should happen together and appear as one atomic transaction should always happen directly next to each other in code. If we had done this, at least it would have been possible to roll the change back on detection of a failure elsewhere in the program like the logging. Now we are sad because money has magically disappeared.

Two transactions of moving money to and from an account are obviously linked, but many times in software there are less obviously related components that make up a transaction that form a single atomic state change. For example consider something like updating a database with content and also moving the same content to a GCS bucket. It would be a problem if we added the data to the db without uploading it to GCS, because if we try again and succeed, we will now have two copies in the db and only one copy in storage. These two changes should happen directly adjacent to each other so that any failures in other parts of the code will not impact their atomicity. You could also consider the fact that a cloud upload can fail - in which case it might make sense to do the db update after the cloud update has been confirmed to succeed. Any transformation that we do between updating the db and uploading to GCS should happen before either of those steps. You can also think about this as saving state updates until later when all work has been done for the overall change.

This might seem like a very odd case that you could catch quite easily in practice, but the logging stands as a substitution for any potentially failing part of software, of which there are unlimited in any real project. Good engineering means our code should be robust and easy to maintain - so after a bit of thinking I came up with one rule for safely making any kind of state change in software.

1. State changes must happen together after any failing code.

If you break this rule, prepare to have a strong rollback system. In practice, I think all potentially failing blocks of code should be written something like this:

variables = ...
try:
    # work for state changes
    # store results into variables
except:
    # handle failures 

# change state using variables

I don't think there's anything particularly remarkable about this observation, but it's something I never concretely put into words and thought about when implementing software. When dealing with real systems, it's important to be tactical about how we go about changing state so we do exactly what we want and nothing else. Let's see how the original code would look like with these guidelines:

def transfer(
    self,
    from_acct: str,
    to_acct: str,
    amount: float,
    memo: Any,
) -> None:
    try:
        # check for potential errors
        if from_acct not in self.accounts:
            raise KeyError(f"Account {from_acct} not found")
        if to_acct not in self.accounts:
            raise KeyError(f"Account {to_acct} not found")
        if self.accounts[from_acct] < amount:
            raise ValueError("Insufficient funds")


        # log
        log_entry = {
            "from": from_acct,
            "to": to_acct,
            "amount": amount,
            "memo": memo,
        }
        log_str = json.dumps(log_entry)
        self.logs.append(log_str)
        print(log_str)
    except Exception as e:
        raise

    # move money
    self.accounts[to_acct] += amount
    self.accounts[from_acct] -= amount