Fix Issue #330

[soegaard]

Ignore non-valid unicode characters.

[github-actions]

Resyntax analyzed 1 file in this pull request and found no issues.

[mflatt]

I imagine these values are generated for characters in supplementary planes. Should input handling track surrogate pairs and join them to form those characters?

[soegaard]

Studying the documentation [1] for UCKeyTranslate, I found:

The key-code-to-character tables map the virtual key code to UCKeyOutput values, for which there are two possibilities:

If bits 15 and 14 of the UCKeyOutput value are 01, the UCKeyOutput value is an index into the offsets contained in a UCKeyStateRecordsIndex structure. If this occurs, the mapping process for the virtual key code continues on to the next step

Otherwise, the UCKeyOutput value produces one or more Unicode characters, either directly or via reference to a UCKeySequenceDataIndex structure. This ends the mapping process for a given virtual key code.

Checking the problematic number from issue #330 :

> (number->string 55349 2)
"1101100000110101"
> (string-length "1101100000110101")
16

So bit 15-14 is indeed 10.

@lambduli Have you had time to experiment with DrRacket and your custom keyboard layout?
Do the special keys work in DrRacket?

Do dead keys work for you?

[1]
https://developer.apple.com/documentation/coreservices/1390584-uckeytranslate

[lambduli]

@soegaard I have just tried out all the characters in my custom layout. Everything works exactly as I would expect.

As per dead keys — my custom layout doesn't have any. (But my standard Czech layout has two and those work just fine in DrRacket.)

[soegaard]

@lambduli Thanks for the report.

@mflatt Since Jan is the only one that come across the problem, and his layout seems to work,
I am inclined to keep the PR as-is.

[mflatt]

I don't think the bits described in the quoted documentation are the relevant ones. Those seem maybe part of a uchr resource.

Meanwhile, the result is a UniChar array, which is UTF-16, and characters like '𝔸' need surrogate encoding in UTF-16. I think the right change is

(define output-chars (malloc _UniChar (add1 max-string-length))) ; leave room for a terminator

and then

  ;; get the number of characters returned, and convert to string;
  ;; the characters are UTF-16
  (define n (max 0 (min max-string-length (unbox actual-string-length))))
  (ptr-set! output-chars _UniChar n 0) ; ensure nul terminator
  (cast output-chars _pointer _string/utf-16))

(edited to fix off-by-1!)

[soegaard]

Your solution certainly looks better.
I tested it and it works for me.

@lambduli Can you change this last version?

Around line 483 the code now looks like this:

  (UCKeyTranslate layout 
                  virtual-key-code
                  key-action
                  (bitwise-and (>> modifier-key-state 8) #xFF)
                  keyboard-type
                  key-translate-options
                  dead-key-state
                  max-string-length
                  actual-string-length
                  output-chars)
  ;; get the number of characters returned, and convert to string;
  ;; the characters are UTF-16
  (define n (max 0 (min max-string-length (unbox actual-string-length))))
  (ptr-set! output-chars _UniChar n 0) ; ensure nul terminator
  (cast output-chars _pointer _string/utf-16))

And in line 429 an add1 was added:

(define output-chars (malloc _UniChar (add1 max-string-length))) ; leave room for a terminator

[lambduli]

Hi @soegaard, I am sorry for such a long delay! I had to take time off because of some deadlines and such.

I am happy to try that change.
So I replaced the code that began at line 483 with the new one (rewriting the few lines that featured the original fix from #330).

I also replaced the 429th line.

I tried to run DrRacket and it launched fine it also seems that all my special characters work in the editor just fine.

[mflatt]

Closing as fixed by #334.