Skip to content

Limit character-set support to UTF-8, ASCII, and binary #456

Description

@JanJakes

Problem

The SQLite driver currently accepts MySQL character sets that it cannot represent safely. The SQL lexer expects UTF-8 input, and SQLite text values are stored as UTF-8.

This becomes visible in WordPress when column metadata reports a non-UTF-8 character set. wpdb::strip_invalid_text() then generates CONVERT(... USING charset) queries containing the original encoded bytes, which the driver cannot parse or convert correctly.

Supported character sets

Limit text support to character sets whose encoded values are valid UTF-8:

  • utf8mb4
  • utf8mb3 / utf8
  • ascii
  • binary values and types

Other character sets, including Big5, KOI8-R, Latin-1, CP1251, Hebrew, and TIS-620, are unsupported. They should not be accepted for columns or CONVERT(... USING charset) expressions.

Expected behavior

  • Reject unsupported character sets in CREATE TABLE and ALTER TABLE with a clear driver error.
  • Reject unsupported character sets in CONVERT(... USING charset).
  • Implement correct behavior for supported UTF-8, ASCII, and binary conversions instead of treating every conversion as a no-op.
  • Ensure column metadata only exposes supported character sets.
  • Define the behavior for existing SQLite databases whose emulated metadata contains unsupported character sets.
  • Add coverage for DDL, metadata, CONVERT, and WordPress CRUD behavior at the supported-character-set boundary.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions