Log everything, except when you shouldn't

Logging in python has always confused me. It takes a long time to set-up, but is extremely configurable, I ended up coding a small Git repo to deepen my understanding of Python logging:

The key takeaways were:

• Separate your understanding of Python logging into two core components: The logger, and loghandler(s)
  • You may have multiple log handlers, each set at a different log level
  • E.g. You could configure your SMS log handler to only care about critical log message. Sweet!
• After learning all the gory details of python logging, you can easily achieve what matters: Coloured logged, and emoji support depending on the severity of your logs
• An unexpected benefit of doing all this effort includes
  • You now get to see the log messages bubble-up from your dependencies (this is a good thing when you're debugging
  • Other logging tools will know how to integrate with your SaaS (yay standards!)

tldr; Every logging start-up you see is a consequence of the complexity of setting up good logging and an unwillingnes to go that deep.

At Subscribie, we have the standard logging you'd expect to stdout, and have configured another handler to log to Telegram.

# Telegram logging
if os.getenv("FLASK_ENV", None) != "development":
    # See https://docs.python.org/3/howto/logging-cookbook.html#dealing-with-handlers-that-block # noqa
    que = queue.Queue(-1)  # no limit on size
    queue_handler = QueueHandler(que)

    telegram_token = os.getenv("TELEGRAM_TOKEN", None)
    telegram_chat_id = os.getenv("TELEGRAM_CHAT_ID", None)
    telegramHandlerHost = "api.telegram.org"

    telegramHandlerUrl = (
        f"bot{telegram_token}/sendMessage?chat_id={telegram_chat_id}&text="
    )

    telegramHandler = TelegramHTTPHandler(
        telegramHandlerHost, url=telegramHandlerUrl, secure=True
    )
    logger.info(f"Setting TELEGRAM_PYTHON_LOG_LEVEL to {TELEGRAM_PYTHON_LOG_LEVEL}")
    telegramHandler.setLevel(TELEGRAM_PYTHON_LOG_LEVEL)
    listener = QueueListener(que, telegramHandler, respect_handler_level=True)
    logger.addHandler(queue_handler)
    listener.start()

Except when you shouldn't. There's an excellent talk (link long lost) which calls out the excellent idea of marking any fields of your data as pii (personally identifiable information) to make compliance easier. The idea being you can greater control, redact as needed.

See also:

You don't need Kubernetes, you need documentation

Money collection is not actually very complex compared to information management.

You might say that information management about money collection is hard (see opensanctions), but now we're in a bind.

What's more important, is knowing that other people can know what you have built, and have them work safely on it, without needing you in the room to answer questions.

Opensource is a fantastic driver of that objective, because it often involves asynchronous communication (see remote working communication tips).

In First Round Captial's interview, it's David Nunez* who sums up the developer perspective best:

It’s the middle of the night when the on-call engineer jolts awake as their phone flashes and chimes with a flurry of push notifications, text messages, and phone calls. The production server is down and they’re responsible for getting it back up as fast as possible. With bleary eyes and a foggy mind, the engineer frantically searches for answers in Slack, Google Docs, and GitHub but they only turn up vague mentions of the error. With no meaningful information to orient around, they’re forced to wait anxiously for one of the veteran engineers to wake up as the outage drags on.
- Investing in Internal Documentation: A Brick-by-Brick Guide for Startups (First Round Review)
*It's not clear if David actually said the above, or this was written by the author ( author information is not visible on the blog sadly).

We know this. You know this. We're developers. We know Stripe succeeded for developers because Paypal didn't serve developers when Stripe did. In darker times, Payment developer docs were a confusing mix-mash of Paypal.com and X.com undoubtedly as a consequence of Peter Thiel and Elon Musk's merger of Paypal and X.com. Thank you Nunez, thank you docs writers & marketers.

What David is talking about now though, is internal docs. Shining a light on what we developers already know but just accept non-docs as the norm. Why do we do this? You Have Agency.

You need up to date external and internal documentation. It's really not about the tools, but for the record we settled on docsy by Google (which is ironically used by the lovely Kubernetes docs team) and https://docs.subscribie.co.uk/ is the result of that work in progress. We also evaluated mkdocs, building our own (who doesn't), and Sphinx (but nobody enjoys writing reStructuredText).

Joe Armstrong on information management

The late Joe Armstrong eludes to information management in his fantastic talk, 'The forgotten ideas in computer science" (slides, video - Computer Science - A Guide for the Perplexed  Joe Armstrong).

The unsolved ideas in Computer Science according to Joe:

• How to find things*
• How to store things
• How to program things

*Google has had a good go, and ChatGTP, well, just makes it up)

As a remote team, we've always had to put docs first it but still takes practise, and frankly you have to lead by example- if you're not documenting your repos, how can you expect your team to do so? There's also a communication style which is learnable. See our Remove Working Communication tips:

Key takeaways are: read the document. tldr; it's in the same vein as nohello.net but with more real-life examples.

Developers might not understand the reason for your SaaS existing, and that's a solvable problem

Harvard Business Review reported "Know Your Customers’ “Jobs to Be Done”. It's nothing complicated: Don't built it unless it's solving a job someones doing (and ideally doesn't want to be doing). There's a reason nobody is buying your jump-to-conclusions matt. Oh wait.

The point is, it helps if your colleagues appreciate, and have experienced the pain your SaaS is solving. Subscribie is essentially a debt management system, dressed up as a Subscription payment system. It's billing software. Boring- but exciting if you're a small business owner or financial controller who knows what it's like to:

• Chase a late payment from a customer
• Manually attach repeat invoices
• Consumer payments are easy 'tap to pay' commercial payments should be the same

Developers typically have never sent anybody an invoice and the only payments they receive are payroll, so it's important to be empathetic, and show developers how payments and invoicing are done. This does two things:

1. The developer can visualise the problem they're solving
2. See the pain that they are solving- for some developers this is exceptionally motivating because they can directly see how their work improves the lives of others

Which is the perfect Segway to- Involve the team in your customers

Show the team how customers are using the SaaS

We give a lot of value to real users of our software. It's incredibly motivating to see your code helping another business out, and the best software is software written by teams who have directly seen their users struggle with something. I'm not going to go all manifesto and cash-in on an Enterprise acronym flow chart- no, just look for opportunities to 'be in the room' with your customer and automate what you can with software. Nibble at your piece of the software world, and deskill rampantly. (Make the next washing machines).

I've already written about SaaS based architectures so won't re-hash that here but will point out the consequences of some of these choices.

and also:

Some consequences are:

• Pro: You're possibly building an asset of your own, rather than bolstering someone else's (this is the buy vs build debate)
• Con: You're building more than you need to, more of what you want to
• Pro: You gain a fundamental understanding of the challenges in the SaaS space
  detail: Every internal process you see at work is a piece of software which will soon be turned into a service

Why "lessons observed" and not "lessons learnt"?

An ex RAF officer once told me, "We used to have regular 'lessons learnt' sessions. But after many years we realised we weren't learning from them, so we re-named the sessions 'lessons observed' instead.