Designing Python Projects For Multi Environment
2023 Edition
Every time I greenfield a project, I’m struck by how many pitfalls I find as I promote from proof of concept to production. I wanted to sit down and do a brief exercise circa February 2023 on running a simple two actor system (backend, database) from local development out through GKE.
I. Introduction
It is important to design systems to be tested because testing is an essential part of the development process that ensures the quality and reliability of the system. Testing helps to catch bugs, ensure that requirements are met, and validate that the system behaves as expected. Designing systems to be tested from the start makes it easier to identify and fix issues early in the development process, reducing the risk of costly and time-consuming bugs in production.
The Focus On Testing
It is important to design systems to be tested because testing is an essential part of the development process that ensures the quality and reliability of the system. Testing helps to catch bugs, ensure that requirements are met, and validate that the system behaves as expected. Designing systems to be tested from the start makes it easier to identify and fix issues early in the development process, reducing the risk of costly and time-consuming bugs in production. I’ve seen projects stall before they could ship because complicated query systems couldn’t be tested and the results only ‘looked’ valid until testers came on. Here’s what the internet says we’ll focus on:
Modular Design:
Systems should be designed as a collection of small, modular components that can be tested in isolation. This promotes better testability and makes it easier to identify and fix issues.
We’ll use a containerized Fast API application and standard postgres server for testing. We’ll run tests independently as well, meaning bash scripted testing on the database side, unfortunately for me. Help me ChatGPT and CoPilot.
Separation of Concerns:
The design should separate concerns such as data processing, data storage, and user interfaces into separate components that can be tested independently.
We’ll use the OpenAPI frontend that comes with FastAPI to proxy a frontend for this project. Data storage will be be controlled by the database here. Processing will be done by the backend or via simple SQL at the database.
Dependency Injection:
Engineers should make use of dependency injection techniques to allow components to be easily substituted during testing, making it easier to test individual components in isolation.
SQL Alchemy allows for dynamic engine creation at runtime. By allowing the connection to be defined in the environment, we can ‘switch’ databases to some extent out of the box. This, we’ll leave alone for this project. Implementation is a bit heavy to ensure continuity between databases. We will want to allow backends to ‘find’ the database in every environment, however. This will impact environment configuration management.
Test-Driven Development:
Engineers should adopt a test-driven development approach, where tests are written before the implementation of a feature, ensuring that all requirements are met and that the implementation is correct.
Pytest can be wired in out of the gate. The ‘frontend’ we’ll test manually as ‘up’. For PostgreSQL, we can borrow bash scripts from the interweb and sneak them in throughout I imagine. We can also have the backend test the database connection on startup. Logging may be an issue as we progress (always is).
Automated Testing: Engineers should automate testing as much as possible, using tools such as continuous integration and continuous deployment, to quickly detect and fix issues.
Git Actions and GCP CloudBuild have a lot of functionality we can take advantage of. This may take a second to set up though.
That’s enough pre-thought, I think. In Part 2: sketching and build everything out on the local level.
Resources
Extra Images
