Skip to content

Escape special characters in Enum/DateTime type names#538

Open
slabko wants to merge 1 commit into
masterfrom
escape-in-identifiers
Open

Escape special characters in Enum/DateTime type names#538
slabko wants to merge 1 commit into
masterfrom
escape-in-identifiers

Conversation

@slabko

@slabko slabko commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

ClickHouse renders string literals inside type names - Enum labels (Enum8('foo' = 1)) and DateTime/DateTime64 timezones (DateTime('Europe/Amsterdam')) - with C-style escape sequences for control and grammar-significant bytes. Previously clickhouse-cpp neither produced these escapes when generating a type name (Type::GetName()) nor decoded them when parsing a type string, so any label/timezone containing ', \, or a control byte could be corrupted or mis-parsed.

This PR makes escaping symmetric in both directions for the following bytes:

byte escaped
0x00 NUL \0
0x08 BS \b
0x09 TAB \t
0x0A LF \n
0x0C FF \f
0x0D CR \r
0x5C \ \\
0x27 ' \'

Decode (clickhouse/types/type_parser.cpp)

  • The single-quoted-string lexer now unescapes escape sequences instead of copying bytes verbatim, via a new kUnescapeMap plus explicit handling of \' and \0.

Encode (clickhouse/types/types.cpp)

  • New EscapeStringLiteral() helper backed by kEscapeMap.

  • Applied in EnumType::GetName(), DateTimeType::GetName(), and DateTime64Type::GetName().

  • Tuples are intentionally out of scope. Tuple field-name identifiers use a different quoting scheme (backticks) and were reverted to the existing implementation; only a benign non-escape tuple round-trip test is included.

  • DateTime/DateTime64 escaping is client-side only in practice. The server validates timezone names against the tz database, so a timezone containing escape sequences can't be stored server-side; the escape/unescape logic is still implemented and unit-tested for correctness.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds symmetric escaping/unescaping of ClickHouse C-style escape sequences inside single-quoted string literals that appear in type names (Enum labels and DateTime/DateTime64 timezones), preventing mis-parsing/corruption for values containing ', \, NUL, or control bytes.

Changes:

  • Encode: introduce EscapeStringLiteral() and apply it to EnumType::GetName(), DateTimeType::GetName(), and DateTime64Type::GetName().
  • Decode: update TypeParser’s single-quoted-string lexer to unescape supported sequences into a scratch buffer.
  • Tests: add unit + integration coverage for escaped Enum labels and DateTime/DateTime64 timezones, plus CreateColumnByType round-trips.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
clickhouse/types/types.cpp Adds escaping map + helper and applies escaping when rendering Enum/DateTime type names.
clickhouse/types/type_parser.cpp Adds unescape map and updates quoted-string tokenization to decode escape sequences.
clickhouse/types/type_parser.h Clarifies token StringView lifetime and scratch-buffer purpose for processed identifiers/strings.
ut/types_ut.cpp Adds unit tests verifying escaping in Type::GetName() for Enum labels and DateTime timezones.
ut/type_parser_ut.cpp Adds parsing tests for escaped Enum labels and escaped DateTime/DateTime64 timezones (including all supported escapes).
ut/CreateColumnByType_ut.cpp Adds CreateColumnByType tests for escaped inputs and render→parse round-trips; includes tuple header for tuple round-trip test.
ut/client_ut.cpp Adds live-server integration test for Enum escape round-trip via server-reported schema.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +262 to +266
scratch_.clear();
scratch_ += *(cur_++); // quotes must be included
for (; end_ - cur_ > 1; ++cur_) {
if (*cur_ == '\\' && end_ - cur_ > 1 && kUnescapeMap[(uint8_t)*(cur_ + 1)] != 0) {
scratch_ += kUnescapeMap[(uint8_t)*(cur_ + 1)];

@slabko slabko Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is, unfortunately, true and can result in errors different from what they really are if the type name is incorrect. However this fix require a separate PR.

Comment thread clickhouse/types/types.cpp Outdated
Comment thread ut/client_ut.cpp Outdated
Comment thread clickhouse/types/type_parser.h Outdated
@slabko
slabko force-pushed the escape-in-identifiers branch 4 times, most recently from d2a8b1f to d3258fe Compare July 21, 2026 21:08
@slabko slabko changed the title Escape special characters in Enum/DateTime type names (encode + decode) Escape special characters in Enum/DateTime type names Jul 22, 2026
ClickHouse renders string literals inside type names — Enum labels (`Enum8('foo' = 1)`) and
DateTime/DateTime64 timezones (`DateTime('Europe/Amsterdam')`) — with C-style escape sequences for
control and grammar-significant bytes. Previously `clickhouse-cpp` neither produced these escapes
when generating a type name (`Type::GetName()`) nor decoded them when parsing a type string, so any
label/timezone containing `'`, `\`, or a control byte could be corrupted or mis-parsed.

This PR makes escaping symmetric in both directions for the following bytes:

| byte | escaped |
|------|---------|
| `0x00` NUL | `\0` |
| `0x08` BS  | `\b` |
| `0x09` TAB | `\t` |
| `0x0A` LF  | `\n` |
| `0x0C` FF  | `\f` |
| `0x0D` CR  | `\r` |
| `0x5C` `\` | `\\` |
| `0x27` `'` | `\'` |

**Decode (`clickhouse/types/type_parser.cpp`)**
- The single-quoted-string lexer now unescapes escape sequences instead of copying bytes verbatim,
  via a new `kUnescapeMap` plus explicit handling of `\'` and `\0`.

**Encode (`clickhouse/types/types.cpp`)**
- New `EscapeStringLiteral()` helper backed by `kEscapeMap`.
- Applied in `EnumType::GetName()`, `DateTimeType::GetName()`, and `DateTime64Type::GetName()`.

- **Tuples are intentionally out of scope.** Tuple field-name identifiers use a different quoting
  scheme (backticks) and were reverted to the existing implementation; only a benign non-escape tuple
  round-trip test is included.
- **DateTime/DateTime64 escaping is client-side only in practice.** The server validates timezone
  names against the tz database, so a timezone containing escape sequences can't be stored
  server-side; the escape/unescape logic is still implemented and unit-tested for correctness.
@slabko
slabko force-pushed the escape-in-identifiers branch from d3258fe to 2cc7c18 Compare July 22, 2026 07:41
@slabko
slabko marked this pull request as ready for review July 22, 2026 09:06
@slabko
slabko requested a review from mzitnik as a code owner July 22, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants