Errors parsing UTF-8 data

[ivvil]

When I try to query tables that contain UTF-8 data I get a C-STRING-DECODING-ERROR.

For example, I have a table with a column named Tamaño, when I execute:

(clsql:print-query "select top(1) * from VgeXML")

I get the following error:

:UTF-8 c-string decoding error:
  the octet sequence #(241 111 0 0) cannot be decoded.
   [Condition of type SB-INT:C-STRING-DECODING-ERROR]

Restarts:
 0: [RETRY] Retry SLY mREPL evaluation request.
 1: [*ABORT] Return to SLY's top level.
 2: [ABORT] abort thread (#<THREAD tid=294045 "sly-channel-1-mrepl-remote-1" RUNNING {120397E083}>)

Backtrace:
 0: (SB-IMPL::FIND-BAD-UTF8 #.(SB-SYS:INT-SAP #X76856031CF60))
 1: (SB-IMPL::READ-FROM-C-STRING/UTF-8/LF #.(SB-SYS:INT-SAP #X76856031CF60) CHARACTER)
 2: (ODBC:%DESCRIBE-COLUMN #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #X768560371600 :TYPE (* T)> 9)
 3: ((LABELS ODBC-DBI::INITIALIZE-COLUMN :IN ODBC-DBI::%INITIALIZE-QUERY) 8)
 4: ((:METHOD ODBC-DBI::%INITIALIZE-QUERY (ODBC-DBI::ODBC-QUERY T T)) #<ODBC-DBI::ODBC-QUERY {1202112C83}> NIL NIL :RESULT-TYPES NIL :WIDTH NIL) [fast-method]
 5: (ODBC-DBI::DB-QUERY #<ODBC-DBI::ODBC-DB {1201C9E943}> "select top(1) * from VgeXML" :RESULT-TYPES NIL :WIDTH NIL)
 6: (ODBC-DBI:SQL "select top(1) * from VgeXML" :DB #<ODBC-DBI::ODBC-DB {1201C9E943}> :RESULT-TYPES NIL :ROW-COUNT NIL :COLUMN-NAMES NIL :QUERY NIL :HSTMT #<unused argument> :WIDTH NIL)
 7: ((SB-PCL::EMF CLSQL-SYS:DATABASE-QUERY) #<unused argument> #<unused argument> "select top(1) * from VgeXML" #<CLSQL-ODBC:ODBC-DATABASE / OPEN {1200038783}> NIL NIL)
 8: ((:METHOD CLSQL-SYS:QUERY (STRING)) "select top(1) * from VgeXML" :DATABASE #<CLSQL-ODBC:ODBC-DATABASE / OPEN {1200038783}> :RESULT-TYPES NIL :FLATP NIL :FIELD-NAMES NIL) [fast-method]
 9: (CLSQL-SYS:PRINT-QUERY "select top(1) * from VgeXML" :TITLES NIL :FORMATS T :SIZES T :STREAM T :DATABASE NIL)
10: (SB-INT:SIMPLE-EVAL-IN-LEXENV (CLSQL-SYS:PRINT-QUERY "select top(1) * from VgeXML") #<NULL-LEXENV>)
11: (EVAL (CLSQL-SYS:PRINT-QUERY "select top(1) * from VgeXML"))
12: ((LAMBDA NIL :IN SLYNK-MREPL::MREPL-EVAL-1))
 --more--

I can see that the rest of the column names get parsed correctly, except for this one

[dieggsy]

Some thoughts:

• Are you sure you're using this version of the library? I'm not sure this is being distributed anywhere at the moment. Fair either way though, since the old sources appear to be gone.
• That is, in fact, an invalid UTF-8 byte sequence. It's definitely not in the UTF-8 encoding of Tamaño:

  CL-USER> (babel:string-to-octets "Tamaño" :encoding :utf-8)
  #(84 97 109 97 195 177 111)
  

  ...or the UTF-16 encoding, for that matter? hmmm:

  CL-USER> (babel:string-to-octets "Tamaño" :encoding :utf-16)
  #(255 254 84 0 97 0 109 0 97 0 241 0 111 0)
  

  (edit: tried the combining character version too, just in case)
• What DB system are you using? What encoding is the DB using for column names?

[dieggsy]

Huh, Latin-1 or iso-8859-1 encoding (ISO/IEC 8859-1) does have that sequence of bytes (I think the zeros would be from a pre-allocated buffer):

(babel:string-to-octets "Tamaño" :encoding :latin-1)
#(84 97 109 97 241 111)

So, it might be that's the encoding the DB is storing the column names in? I'm wondering if adding something like :encoding :iso-8859-1 or so to your connect statement would help, assuming that's really the issue. Though, that may have other side effects you don't want.