failure to send octets to a :force-binary stream.

[informatimago]

I'm trying to get a binary stream, but drakma always opens a latin-1 stream

(make-flexi-stream (make-chunked-stream http-stream)
                                          :external-format +latin-1+)

So when I try to write a sequence of bytes, the flexistream complains:

Connecting to ws://localhost:1236/ws
  0: (DRAKMA:HTTP-REQUEST "http://localhost:1236/ws" 
         :PROTOCOL :HTTP/1.1
         :METHOD :GET
         :FORCE-SSL NIL
         :CERTIFICATE NIL
         :KEY NIL
         :CERTIFICATE-PASSWORD NIL
         :VERIFY NIL
         :MAX-DEPTH 10
         :CA-FILE NIL
         :CA-DIRECTORY NIL
         :ADDITIONAL-HEADERS (("Sec-WebSocket-Protocol" . "chat")
                              ("Sec-WebSocket-Version" . 13)
                              ("Sec-WebSocket-Key" . "q4hMuo+AxUxtJltGws387g==")
                              ("Connection" . "Upgrade")
                              ("Upgrade" . "websocket")
                              ("Host" . "localhost:1236"))
         :CLOSE NIL
         :KEEP-ALIVE NIL
         :CONTENT-TYPE "application/octet-stream"
         :FORCE-BINARY T
         :EXTERNAL-FORMAT-OUT :OCTET
         :EXTERNAL-FORMAT-OUT :OCTET
         :WANT-STREAM T)
  0: DRAKMA:HTTP-REQUEST returned
       #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}>
       101
       ((:CONTENT-LENGTH . "0") (:DATE . "Fri, 08 Sep 2023 19:48:57 GMT") (:SERVER . "Hunchentoot 1.3.0")
        (:SEC-WEBSOCKET-ACCEPT . "MhE9MD1ozUhtWj5UFyRr0c5kLRo=")
        (:SEC-WEBSOCKET-LOCATION . "ws://127.0.0.1:1236//ws") (:SEC-WEBSOCKET-PROTOCOL . "chat")
        (:UPGRADE . "WebSocket") (:CONNECTION . "Upgrade") (:STATUS-CODE . "110")
        (:CONTENT-TYPE . "application/octet-stream"))
       #<PURI:URI http://localhost:1236/ws>
       #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}>
       NIL
       "Switching Protocols"
Connected
ENDPOINT-EXTENSIONS               NIL
ENDPOINT-SUBPROTOCOL              "chat"
ENDPOINT-COOKIES                  NIL
LOWER-LAYER                       #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}>
ENDPOINT-LOCAL-ADDRESS            NIL
ENDPOINT-LOCAL-PORT               NIL
ENDPOINT-REMOTE-ADDRESS           NIL
ENDPOINT-REMOTE-PORT              NIL
ENDPOINT-CLOSEDP                  NIL
Enter:  quit   to end, or a message to send to the server.
Client Received a text message "HELLO I will echo text and binary messages"

Text Message: Hello World!

debugger invoked on a TYPE-ERROR @53CD9D82 in thread #<THREAD "main thread" RUNNING {10015E8113}>: The value 110 is not of type CHARACTER

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ABORT] Exit debugger, returning to top level.

((:METHOD FLEXI-STREAMS::WRITE-SEQUENCE* (FLEXI-STREAMS::FLEXI-LATIN-1-FORMAT T T T T)) #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument>) [fast-method]
   source: (CHAR-CODE CHAR-GETTER)
0] backtrace

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10015E8113}>
0: ((:METHOD FLEXI-STREAMS::WRITE-SEQUENCE* (FLEXI-STREAMS::FLEXI-LATIN-1-FORMAT T T T T)) #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument>) [fast-method]
1: ((:METHOD TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE (FLEXI-STREAMS:FLEXI-OUTPUT-STREAM T T T)) #<unavailable argument> #(110 50 52 139) #<unavailable argument> #<unavailable argument>) [fast-method]
2: ((SB-PCL::EMF TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE) #<unused argument> #<unused argument> #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}> #(110 50 52 139) 0 4)
3: ((:METHOD SB-GRAY:STREAM-WRITE-SEQUENCE (TRIVIAL-GRAY-STREAMS:FUNDAMENTAL-OUTPUT-STREAM T)) #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}> #(110 50 52 139) 0 NIL) [fast-method]
4: (SB-IMPL::WRITE-SEQ-IMPL #(110 50 52 139) #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}> 0 NIL)
5: (WRITE-SEQUENCE #(110 50 52 139) #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}> :START 0 :END NIL)
6: ((:METHOD SEND-OCTETS (STREAM T)) #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}> #(110 50 52 139) :START 0 :END NIL) [fast-method]
7: ((SB-PCL::EMF SEND-OCTETS) #<unused argument> #<unused argument> #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}> #(110 50 52 139))
8: ((:METHOD CL-NAIVE-WEBSOCKETS::WRITE-FRAME (T CL-NAIVE-WEBSOCKETS::FRAME)) #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}> #<CL-NAIVE-WEBSOCKETS::FRAME {100647F2A3}>) [fast-method]
9: ((:METHOD CL-NAIVE-WEBSOCKETS::SEND-FRAME (WEBSOCKETS-ENDPOINT CL-NAIVE-WEBSOCKETS::FRAME)) #<WEBSOCKETS-CLIENT-ENDPOINT {100647B953}> #<CL-NAIVE-WEBSOCKETS::FRAME {100647F2A3}>) [fast-method]
10: ((FLET SB-THREAD::WITH-MUTEX-THUNK :IN CL-NAIVE-WEBSOCKETS::SEND-MESSAGE))
11: ((FLET "WITHOUT-INTERRUPTS-BODY-1" :IN SB-THREAD::CALL-WITH-MUTEX))
12: (SB-THREAD::CALL-WITH-MUTEX #<FUNCTION (FLET SB-THREAD::WITH-MUTEX-THUNK :IN CL-NAIVE-WEBSOCKETS::SEND-MESSAGE) {F8FF15B}> #<SB-THREAD:MUTEX "websockets-endpoint %send-message-lock" owner: #<SB-THREAD:THREAD "main thread" RUNNING {10015E8113}>> T NIL)
13: ((:METHOD CL-NAIVE-WEBSOCKETS::SEND-MESSAGE (WEBSOCKETS-ENDPOINT T T T)) #<WEBSOCKETS-CLIENT-ENDPOINT {100647B953}> 1 #(72 101 108 108 111 32 87 111 114 108 100 33) NIL) [fast-method]
14: ((LAMBDA (SB-PCL::.ARG0. SB-PCL::.ARG1. SB-PCL::.ARG2. SB-PCL::.ARG3.)) #<WEBSOCKETS-CLIENT-ENDPOINT {100647B953}> 1 #(72 101 108 108 111 32 87 111 114 108 100 33) NIL)
15: ((:METHOD SEND-TEXT-MESSAGE (WEBSOCKETS-ENDPOINT T)) #<WEBSOCKETS-CLIENT-ENDPOINT {100647B953}> "Hello World!" :MAY-FRAGMENT NIL) [fast-method]
16: ((SB-PCL::EMF SEND-TEXT-MESSAGE) #<unused argument> #<unused argument> #<WEBSOCKETS-CLIENT-ENDPOINT {100647B953}> "Hello World!")
17: ((LAMBDA (SB-PCL::.ARG0. SB-PCL::.ARG1. SB-INT:&MORE SB-PCL::.MORE-CONTEXT. SB-PCL::.MORE-COUNT.) :IN "/Users/pjb/.cache/common-lisp/sbcl-2.3.7.14-b7987efc9-macosx-x64/Users/pjb/quicklisp/dists/quicklisp/software/cl-ppcre-20230618-git/api.fasl") #<WEBSOCKETS-CLIENT-ENDPOINT {100647B953}> "Hello World!")
18: ((FLET "G0" :IN CHAT))
19: (CHAT #<WEBSOCKETS-CLIENT-ENDPOINT {100647B953}>)
20: (WS-CHAT-CLIENT "localhost" 1236 "/ws" :SECURE NIL :SUBPROTOCOL "chat" :ORIGIN NIL)
21: (RUN-CLIENT "localhost" 1236 "/ws" NIL "chat" NIL)
22: (SB-INT:SIMPLE-EVAL-IN-LEXENV (RUN-CLIENT "localhost" 1236 "/ws" NIL "chat") #<NULL-LEXENV>)
23: (EVAL (RUN-CLIENT "localhost" 1236 "/ws" NIL "chat"))
24: (SB-EXT:INTERACTIVE-EVAL (RUN-CLIENT "localhost" 1236 "/ws" NIL "chat") :EVAL NIL)
25: (SB-IMPL::REPL-FUN NIL)
26: ((LAMBDA NIL :IN SB-IMPL::TOPLEVEL-REPL))
27: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #<FUNCTION (LAMBDA NIL :IN SB-IMPL::TOPLEVEL-REPL) {F8FFC1B}>)
28: (SB-IMPL::TOPLEVEL-REPL NIL)
29: (SB-IMPL::TOPLEVEL-INIT)
30: ((FLET SB-UNIX::BODY :IN SB-IMPL::START-LISP))
31: ((FLET "WITHOUT-INTERRUPTS-BODY-3" :IN SB-IMPL::START-LISP))
32: (SB-IMPL::%START-LISP)

0] down
((:METHOD TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE (FLEXI-STREAMS:FLEXI-OUTPUT-STREAM T T T)) #<unavailable argument> #(110 50 52 139) #<unavailable argument> #<unavailable argument>) [fast-method]
1] d
((SB-PCL::EMF TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE) #<unused argument> #<unused argument> #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}> #(110 50 52 139) 0 4)
2] d
((:METHOD SB-GRAY:STREAM-WRITE-SEQUENCE (TRIVIAL-GRAY-STREAMS:FUNDAMENTAL-OUTPUT-STREAM T)) #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}> #(110 50 52 139) 0 NIL) [fast-method]
3] d
(SB-IMPL::WRITE-SEQ-IMPL #(110 50 52 139) #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}> 0 NIL)
4] d
(WRITE-SEQUENCE #(110 50 52 139) #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}> :START 0 :END NIL)

5] list-locals
#:.DEFAULTING-TEMP.  =  0
#:.DEFAULTING-TEMP.#1  =  NIL
SB-IMPL::SEQ  =  #(110 50 52 139)
STREAM  =  #<FLEXI-STREAMS:FLEXI-IO-STREAM {100643B0E3}>

5] (inspect stream)

The object is a STANDARD-OBJECT of type FLEXI-STREAMS:FLEXI-IO-STREAM.
0. OPEN-P: T
1. STREAM: #<CHUNGA:CHUNKED-IO-STREAM {100643ACA3}>
2. EXTERNAL-FORMAT: #<FLEXI-STREAMS::FLEXI-LATIN-1-FORMAT (:ISO-8859-1 :EOL-STYLE :LF) {100712E3F3}>
3. ELEMENT-TYPE: FLEXI-STREAMS:OCTET
4. COLUMN: NIL
5. LAST-CHAR-CODE: NIL
6. LAST-OCTET: NIL
7. OCTET-STACK: NIL
8. POSITION: 415
9. BOUND: NIL
> q

5] 

I reported also the issue to flexi-stream, edicl/flexi-streams#49 since from the name flexi-stream I would also have expected an ambivalent stream, but perhaps it's not how it's supposed to work.

[avodonosov]

Note, in the docs:

If you want to read binary data from this stream, read from the underlying stream which you can get with FLEXI-STREAM-STREAM.

https://edicl.github.io/drakma/#arg-force-binary

I hope this work for writing too.

(Maybe you have a valid point about flexi-stream with element type octect, I am not sure. Although since the doc explicitly suggests the underlying stream for binary IO, it may be by design. At least working with the underlying stream is a workaround for your code.)

[informatimago]

An answer on the flexi-stream issue explains it. edicl/flexi-streams#49

The problem is that flexi-stream interprets a buffer vector of T (or not vector of integer) as non-binary, without looking at the contents of the vector. If we don't pass a buffer vector of integer, it expects characters inside. So we have to copy the buffers to vectors of octets to have it take the binary data.

This means that I have a work-around, but it may be rather costly when we use flexi-stream (and therefore when we use drakma that uses flexi-stream).

The problem is not directly in drakma, but since it uses flexi-stream explicitely in its API, I think the documentation should mention that when working on the http stream, we're restricted to vectors of bytes to send and receive binary data.

Therefore this issue could be converted in a documentation update issue for drakma and be closed. Thank you.

[avodonosov]

@informatimago, you've probably missed the comment above. Writing through the underlying stream may save from the need to copy buffers.

[informatimago]

Ah, it's a possibility. I will try it and report.

[informatimago]

@avodonosov it works indeed using the underlying stream. It's a better solution. Thank you.