Skip to content

refactor: Migrate table column ComponentRow.text from TEXT to MEDIUMTEXT length#300

Draft
yuechao-qin wants to merge 1 commit into
masterfrom
ycq/refactor-componentrow-text-mediumtext
Draft

refactor: Migrate table column ComponentRow.text from TEXT to MEDIUMTEXT length#300
yuechao-qin wants to merge 1 commit into
masterfrom
ycq/refactor-componentrow-text-mediumtext

Conversation

@yuechao-qin

@yuechao-qin yuechao-qin commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

Migrate component.text from TEXT to MEDIUMTEXT

What

Widens the component.text column from TEXT (64 KB) to MEDIUMTEXT (16 MB) on MySQL. PostgreSQL and SQLite are unaffected (TEXT is already unlimited).

Why

Component YAML specs can exceed 64 KB for complex pipelines with many parameters/outputs. The current TEXT limit silently truncates on MySQL.

How

  • Model update: ComponentRow.text now uses sql.Text(length=2**24) which SQLAlchemy maps to MEDIUMTEXT on MySQL
  • Migration function: migrate_component_text_column() in database_migrations.py — idempotent, inspects actual column length before deciding to ALTER
  • Startup wiring: Runs automatically on deploy (after migrate_secret_value_column), no-op on subsequent deploys

Production impact

Factor Value
Rows 13,702
Table size ~240 MB
MySQL algorithm INPLACE
Estimated time 5–30 seconds
Concurrent DML Allowed (reads/writes continue)
Lock Brief metadata lock at start/end only

Testing

  • 2 new unit tests: idempotency + large text round-trip (SQLite)
  • All 56 existing tests pass

Files changed

File Change
cloud_pipelines_backend/backend_types_sql.py Add _MEDIUMTEXT_LENGTH constant
cloud_pipelines_backend/component_library_api_server.py Update column type
cloud_pipelines_backend/database_migrations.py Add migration function
cloud_pipelines_backend/database_ops.py Wire into startup
cloud_pipelines_backend/api_router.py Add 3 temporary admin endpoints
tests/test_database_migrations.py Add 2 tests

Before merging

Remove the temporary admin endpoints at the bottom of api_router.py (marked with DO NOT MERGE INTO master/main banner).

Copy link
Copy Markdown
Collaborator Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

# MySQL MEDIUMTEXT: 2^24 = 16,777,215 bytes
# With utf8mb4 charset (4 bytes/char worst case): ~4,194,303 characters
# With ASCII-only content (1 byte/char): ~16,777,215 characters
_MEDIUMTEXT_LENGTH: Final[int] = 2**24
@yuechao-qin
yuechao-qin force-pushed the ycq/refactor-componentrow-text-mediumtext branch from 63ff127 to 0bcd0b2 Compare July 18, 2026 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant