Skip to content

msflib (core)

Purpose

msflib is the foundation every other MSFLib package builds on, and the only package you need to install for a CRUD API on your own models. It provides namespace-aware settings, base model classes, a generic CRUD action layer with lifecycle events, a typed router helper, an app-scoped event bus, scope and policy primitives, schema setup and migration helpers, a seeding framework, and storage, email and security utilities.

The core does not mount any routes of its own and does not create your FastAPI app, engine or session. You assemble those in the host app; the core gives you the parts.

For a hands-on walk through the core, follow the Quick Start.

Install

[tool.poetry.dependencies]
msflib = { git = "https://github.com/msflib/fastapi.git", subdirectory = "core", rev = "core-v0.2.1" }
Extra Enables
migrations Alembic, used by msflib.db.migrations on non-SQLite databases

Alembic is imported only when a migration actually runs through Alembic. SQLite is managed with create_all and needs no extra.

MSFLib 0.2.0 and later require Pydantic v2 and pydantic-settings v2.

What is in the package

Area Package path What it does
Settings msflib.core.config ModuleSettingsBase, SettingsBase, CoreSettings, compose_settings_model, tiered config helpers
Models msflib.models ModelBase, SchemaBase, BaseEnum, Option, ServerEvent, AccountStub
Actions msflib.actions ModelAction, action, write hooks, after-commit helpers
API helpers msflib.api DependencyNamespace, create_enhanced_router, session and keystore dependency factories
Event bus msflib.eventbus App-scoped emitters, on, HookRegistry, emit_optional and emit_required
Scope msflib.scope ScopeEnvelope, ScopeCompiler, dimension registry, get_scope_dependencies
Policy msflib.policy PolicyEnvelope, PolicyResolutionService, resolution traces, decision logging
Database msflib.db migrate, MigrationConfig, run_initial_data_cli, reset_database, SQLite savepoint support
Seeding msflib.seed YAML-driven SeedRunner, SeederBase, ActionSeeder, CLI (see the note below the table)
Security msflib.core.security bcrypt password helpers, JWT creation, token revocation
Storage and email msflib.utils.uploads, msflib.services.email Local, S3, GCS, Azure and Cloudinary uploads; Jinja-rendered email
Other msflib.core.pii, msflib.core.records, msflib.core.validators, msflib.schemas Sensitive-key redaction, structured record helpers, validators, Msg and Token schemas

Import from the subpackage that owns the name. msflib/__init__.py exports nothing. msflib.models resolves its names lazily, so from msflib.models import ModelBase loads only what it needs. msflib.db and msflib.utils have empty __init__ files; import from their submodules (msflib.db.migrations, msflib.utils.uploads, and so on).

Known issue (#276)

msflib.seed exports only the command-line helpers (run_seed_cli, build_parser, parse_args, is_production_db, confirm_production_run). Import SeedRunner from msflib.seed.runner, and SeederBase and ActionSeeder from msflib.seed.base.

Wiring into a host app

A host app does five things with the core: compose its settings, build an engine and session dependency, register models, create the schema, and include routers. The shortest complete version is below; the Quick Start builds the same app step by step.

# app/settings.py
from msflib.core.config import CoreSettings, SettingsBase


class Settings(CoreSettings, SettingsBase):
    pass


settings = Settings()

with a .env file next to where you run the app:

PROJECT_NAME=Tasks
SQLITE_DATABASE_URI=sqlite:///./tasks.db
SECRET_KEY=change-me

# app/settings.py
from msflib.core.config import CoreSettings, SettingsBase, compose_settings_model

Settings = compose_settings_model("Settings", SettingsBase, [CoreSettings])

settings = Settings()

with a .env file next to where you run the app:

CORE__PROJECT_NAME=Tasks
CORE__SQLITE_DATABASE_URI=sqlite:///./tasks.db
CORE__SECRET_KEY=change-me

# app/db.py
from msflib.api.deps import get_session_factory
from msflib.db.sqlite import enable_savepoints
from sqlmodel import create_engine

from .settings import settings

core = settings.scope("CORE")
engine = enable_savepoints(
    create_engine(core.SQLITE_DATABASE_URI, connect_args={"check_same_thread": False})
)
get_session = get_session_factory(engine)

# app/main.py (excerpt; `lifespan`, `get_owner`, `task_router` and the `FastAPI` import are from the Quick Start)
from msflib.eventbus import bind_app_emitter

app = FastAPI(title=core.PROJECT_NAME, lifespan=lifespan)
app_emitter = bind_app_emitter(app)
app.include_router(task_router(get_session=get_session, get_owner=get_owner), prefix=core.API_V1_STR)

Module routers follow the same pattern as task_router in the Quick Start: they are factories that take your session and identity dependencies and return an APIRouter. See Mounting routes.

Known issue (#276)

msflib.db.session builds an engine from app.core.config.settings (it reads USE_SQLITE, SQLITE_DATABASE_URI, SQLALCHEMY_DATABASE_URI and DB_DEBUG_MODE off a top-level settings object) and so only imports in a host app that has an app.core.config module with those attributes. Building your own engine, as above, avoids that coupling.

Configuration

Settings are Pydantic models grouped by namespace. Each module defines a ModuleSettingsBase subclass with a namespace class variable (CORE, AUTH, AI_CORE, and so on). The host app puts the ones it uses into one settings class and reads each through settings.scope("NAMESPACE"), which works in both layouts (not every module reads settings through scope; see the known issue below).

There are two ways to build that class. Both are supported, and neither is deprecated.

Inherit the module settings classes together with SettingsBase, as the repository's testsite does. Fields are top-level and environment variables use the plain field names.

from msflib.auth.config import AuthSettings
from msflib.core.config import CoreSettings, SettingsBase


class Settings(AuthSettings, CoreSettings, SettingsBase):
    PROJECT_NAME: str = "Tasks"


settings = Settings()
settings.PROJECT_NAME                          # also settings.scope("CORE").PROJECT_NAME
settings.scope("AUTH").ENABLE_GOOGLE_OAUTH

PROJECT_NAME=Tasks
ENABLE_GOOGLE_OAUTH=true

compose_settings_model(name, SettingsBase, [CoreSettings, AuthSettings, ...]) creates a settings class with one field per namespace. Environment variables are nested with a double underscore.

from msflib.auth.config import AuthSettings
from msflib.core.config import CoreSettings, SettingsBase, compose_settings_model

Settings = compose_settings_model("Settings", SettingsBase, [CoreSettings, AuthSettings])

settings = Settings()
settings.CORE.PROJECT_NAME                     # also settings.scope("CORE").PROJECT_NAME
settings.scope("AUTH").ENABLE_GOOGLE_OAUTH

CORE__PROJECT_NAME=Tasks
AUTH__ENABLE_GOOGLE_OAUTH=true

A subclass of the class compose_settings_model returns can also declare a namespace field with a default instance (CORE: CoreSettings = CoreSettings(PROJECT_NAME="Tasks")) to change defaults in code, with the caveat in the known issue below.

Subclassed Composed
Environment variables Plain field names: PROJECT_NAME Nested: CORE__PROJECT_NAME
Read a value settings.PROJECT_NAME or settings.scope("CORE") settings.CORE.PROJECT_NAME or settings.scope("CORE")
Change a default in code Assign the field in the class body Declare a namespace field with a default instance (see the known issue)
Fields with the same name in several modules One shared field (every module defines ENABLED, so a flat ENABLED applies to all of them) Separate nested fields per namespace
Code that reads a top-level attribute Works, for example the auth router reads settings.SECRET_KEY Needs a matching top-level attribute (#287)

Reading ENABLED through settings.scope(...) can mix values between modules in either layout (#280).

Known issue (#287)

The msflib-auth router reads settings.SECRET_KEY from the top level of the settings object. With composed (nested) settings POST /login and the password-recovery endpoints fail with AttributeError. Use the subclassed style when you mount the auth router.

Behaviour may change (#280)

The NAMESPACE__KEY environment variable style only works for composed (field) settings, not for subclassed hosts. In a subclassed host use the plain field names.

Known issue (#276)

When a subclass of a compose_settings_model result declares a namespace field with a default instance (CORE: CoreSettings = CoreSettings(PROJECT_NAME="Tasks")), every environment variable for that namespace is ignored, because SettingsBase.__init__ re-applies the subclass default after the environment has been read. See the troubleshooting entry for workarounds.

Modules that need to keep older flat variable names working declare flat_aliases; nested keys win over flat aliases when both are set. SettingsBase reads .env and uses __ as the nested delimiter.

Every ModuleSettingsBase has an ENABLED: bool = True field. The core namespace is CORE (CoreSettings):

Key Default Notes
API_V1_STR /api/v1 Prefix to give include_router
PROJECT_NAME MSFLib
SECRET_KEY random per process Set it explicitly in production, or tokens do not survive a restart
ENVIRONMENT development production or prod disables destructive DB operations
SERVER_HOST, CLIENT_HOST http://localhost:8000, http://localhost:3000 Used to build absolute URLs
BACKEND_CORS_ORIGINS [] A JSON list string such as ["http://localhost:3000"]. Only the JSON form is parsed (known issue, #276)
USE_SQLITE, SQLITE_DATABASE_URI True, sqlite:///./tmp/msflib.db
POSTGRES_SERVER, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB, POSTGRES_PORT localhost, postgres, empty, empty, 5432 Assembled into SQLALCHEMY_DATABASE_URI when the settings come from the environment or composed settings. If you build CoreSettings(...) directly with keyword arguments and omit the URI, it stays None (#276)
SQLALCHEMY_DATABASE_URI, DB_DEBUG_MODE None, False
SMTP_HOST, SMTP_PORT, SMTP_TLS, SMTP_USER, SMTP_PASSWORD localhost, 1025, False, empty, empty
EMAILS_FROM_EMAIL, EMAILS_FROM_NAME, EMAILS_ENABLED, EMAILS_USE_SENDMAIL, EMAIL_TEMPLATES_DIR see source EMAILS_ENABLED defaults to true when host, port and from-address are all set
STORAGE_METHOD, STORAGE_BASE_URL, STORAGE_PATH file, /uploads, uploads STORAGE_METHOD is one of file, cloudinary, s3, gcs, azure
CLOUDINARY_*, AWS_*, GCS_CREDENTIALS_PATH, AZURE_STORAGE_CONNECTION_STRING empty Credentials for the non-local storage methods
SENTRY_DSN None

The full list is CoreSettings in core/msflib/core/config.py. For tests, assign through the scope (settings.scope("CORE").EMAILS_ENABLED = True). The tiered configuration model (default, environment, tenant, workspace, user, request override) is described in Tiered configuration.

Key concepts

Models

ModelBase is a SQLModel base with id, created_at and updated_at; subclass it with table=True to get a table. SchemaBase is the base for create, update and read schemas: it serialises enums as plain values and coerces numbers to strings where the field is a string. BaseEnum is a str enum whose members compare equal to their string value, for use in both. ModelBase.add_relationship(...) attaches a SQLAlchemy relationship after the classes are defined, which is how modules link to tables defined elsewhere. See Models, data and tables.

Actions

ModelAction[Model, CreateSchema, UpdateSchema] is the CRUD layer: get, get_by_all, get_multi_by_all, create, update, delete, their *_multi and *_by_expressions variants, and random and create_random for test data. Actions take a Session as the first argument and commit by default; pass commit=False to take part in a larger transaction. Every write emits lifecycle events. See Actions and CRUD and Subclassing actions.

Lifecycle events and write hooks

Each ModelAction write emits events named <model>-<op>-pre-commit (after flush, before commit, inside the transaction) and <model>-<op>-post-commit (once the outermost commit succeeds), where <model> is the lowercase class name and <op> is create, update or delete. A generic model-<op>-... event fires as well. Listeners receive the row and an options dict containing the session. Pre-commit listeners that raise abort the write; post-commit listeners are best-effort. Post-commit listeners get a plain snapshot of the row, not the live ORM object.

on_model_write(model, callback, emitter=..., key=...) is a shorter way to run callback(session, row, operation) inside the write's transaction, in a savepoint so a failing callback undoes only its own writes. run_after_commit(session, callback) and emit_after_commit(session, name, ...) run work only after the outermost commit, correctly across savepoints.

Event bus

bind_app_emitter(app) gives each FastAPI app its own emitter and adds middleware that makes it the current emitter for each request. Register listeners with @app_emitter.on("event-name"); get_app_emitter(app) returns the same emitter later. The app emitter is active in request handlers, in BackgroundTasks run after the response, and in call_later or create_task work started from a request. In scripts, threads (a threading.Thread does not inherit it) and lifespan code, wrap the code in with use_app_emitter(app):. A worker process has no app: register its listeners on AppEmitter(get_emitter()). Listener functions take (payload, options, logger=None). emit_optional is best-effort; emit_required raises when no listener handled the event. Modules that need event wiring expose a register_event_hooks(..., emitter=app_emitter) function; HookRegistry makes such registration idempotent. See Event bus.

Scope

ScopeEnvelope is an immutable description of whose data an operation touches: dimensions such as tenant_id, workspace_id and account_id, plus an optional principal. ScopeCompiler turns an envelope into a canonical string, segments or mapping for use in cache keys, vector-store filters and similar. get_scope_dependencies(...) takes your identity dependencies (get_current_account, get_current_workspace, get_current_active_user, optionally get_current_tenant) and returns a DependencyNamespace of request-scoped dependencies: get_account_scope, get_workspace_scope (when a workspace tier exists) and get_tenant_scope. Which callables to pass is set out in the docstring of get_scope_dependencies. See Scopes, tenancy and workspaces.

Policy

PolicyEnvelope is a common shape (enabled, defaults, constraints, overrides, selection, fallbacks, observability) for policy-bearing settings. PolicyResolutionService resolves a module's effective settings across tenant, workspace and user layers plus a request override, returning the settings and a PolicyResolutionTrace that records which layer won each key. Scoped values come from a PolicyScopedConfigStore, which msflib-workspace-config implements. See Policy.

Database setup

msflib.db.migrations.migrate(engine, MigrationConfig(...)) brings any database to the current schema and returns the action it took: create_all for SQLite or an app with no Alembic revisions, created, baselined or upgraded for Alembic-managed databases. It builds the schema from config.metadata as it stands, so import every module that defines tables before calling migrate; otherwise it silently creates zero tables. It runs in one transaction, and on PostgreSQL under an advisory lock so concurrent starts take turns. run_initial_data_cli(...) wraps migrate plus your seed function into a python -m app.initial_data entry point with --check and --reset. See Models, data and tables.

Seeding

SeedRunner reads a seeders.yml that names an action class (class_name) or a factory function (class_factory), plus records and dependencies between seeders, and creates the rows through the actions. Custom seeders subclass SeederBase. run_seed_cli(...) provides the command line. See Seeding.

Dependencies and routers

get_session_factory(engine) returns a FastAPI dependency that yields a Session. get_keystore_factory(...) returns a dependency for a token store (Redis when host and password are given, otherwise in-memory). DependencyNamespace is the container module dependency factories return. create_enhanced_router(router) adds a validation_types argument to route decorators so the request schema can be chosen when the router is built, which is how modules let you substitute your own create and update schemas. See Endpoints and routers and Dependency injection.

Examples

All of these were run against the core on SQLite in memory.

Listen to lifecycle events

from fastapi import FastAPI
from msflib.eventbus import bind_app_emitter, use_app_emitter
from sqlmodel import Session, SQLModel, create_engine
from msflib.db.sqlite import enable_savepoints

from app.actions import task_action
from app.models import TaskCreate

app = FastAPI()
app_emitter = bind_app_emitter(app)
engine = enable_savepoints(create_engine("sqlite://"))
SQLModel.metadata.create_all(engine)

seen = []


@app_emitter.on("task-create-pre-commit")
def before_commit(task, options, logger=None):
    seen.append(("pre", task.title, "session" in options))


@app_emitter.on("model-create-post-commit")
def after_commit(task, options, logger=None):
    seen.append(("post", task.title))


with use_app_emitter(app), Session(engine) as session:
    task_action.create(session, data=TaskCreate(title="x"), update={"owner": "ann"})

print(seen)  # [('pre', 'x', True), ('post', 'x')]

use_app_emitter(app) is needed here because there is no request: inside a request the middleware does this for you. The Task model and task_action are from the Quick Start.

Run work inside every write

from msflib.actions import on_model_write

on_model_write(
    "task",
    lambda session, row, operation: print(operation, row.id),
    emitter=app_emitter,
    key="audit",
)

app_emitter is the app emitter from the previous example. With this registered, creating a task prints create and the new row id before the commit. on_error="raise" makes a failing callback abort the write; the default logs it and lets the write go ahead. Writes that bypass ModelAction (raw session.add) emit no events and run no hooks.

Build and compile a scope

from msflib.scope import ScopeCompiler, build_context_scope, create_default_registry

scope = build_context_scope(tenant_id=1, workspace_id=7, account_id=3)
compiler = ScopeCompiler(registry=create_default_registry())
print(compiler.compile_scope_string(scope, operation="retrieval.search"))
# t/1/a/3/w/7

Passwords and tokens

from msflib.core.security import create_access_token, get_password_hash, verify_password

hashed = get_password_hash("correct horse")
assert verify_password("correct horse", hashed)

token = create_access_token("42", secret_key="change-me", expiry_minutes=30)

create_access_token(subject, secret_key, expiry_minutes=11520, claims=None) signs an HS256 JWT whose sub is the string form of subject. Login flows built on it live in msflib-auth.

Troubleshooting

An uploads/ directory appears in the working directory.

Known issue (#276)

CoreSettings creates STORAGE_PATH (default uploads) when it is instantiated and STORAGE_METHOD is file. Importing msflib.core.config instantiates a module-level Settings() as well, so the directory is created on import. Set STORAGE_PATH to a directory you want, or a different STORAGE_METHOD.

An environment variable has no effect. Either the name does not match your layout (a subclassed host reads PROJECT_NAME, a composed host reads CORE__PROJECT_NAME), or your composed settings subclass overrides the namespace default in code. With class Settings(Base) where Base comes from compose_settings_model, and CORE: CoreSettings = CoreSettings(PROJECT_NAME="x"), every CORE__... variable is ignored, not only PROJECT_NAME (see the known issue under Configuration). Put the values in the environment or .env instead, or set them in code with settings.scope("CORE").X = ... after construction.

On SQLite, work done before a savepoint survives a rollback. pysqlite defers BEGIN, so releasing the first savepoint in a transaction commits everything before it. Wrap the engine with msflib.db.sqlite.enable_savepoints(engine). This matters for session.begin_nested(), on_model_write callbacks and commit=False flows. Code that depends on savepoints can call require_sqlite_savepoints(engine, caller=...) to fail fast on an engine that was not wrapped.

TypeError: ... must be instantiated with concrete generic types. ModelAction() was created without type arguments. Use ModelAction[Model, Create, Update]() or subclass it with concrete types.

Lifecycle events do not fire. The listener was registered on a different emitter from the one in use. Outside a request, wrap the code in with use_app_emitter(app): (this includes threads); in a worker process, which has no app, register the listener on AppEmitter(get_emitter()); a listener on the global emitter is not seen by app-bound code; in tests, bind the same app emitter you listen on.

Alembic errors on PostgreSQL. Install the migrations extra. If the app has no revisions yet, migrate falls back to create_all and logs a warning.

Importing msflib.db.session raises ModuleNotFoundError: app. See the warning under Wiring into a host app.

API reference

See the generated API reference for the core package. core/README.md in the repository has more detail on the migration scenarios and the seeding folder layout.

See also