Ksql: Changes are not emitted to streaming query (with python/postman)

[wilcoschoneveld]

Describe the bug
Trying to do the ksqlDB quickstart with python, but can't get any stream content to appear.

To Reproduce
Using ksqlDB quickstart docker-compose.yml (version 0.7)
I try the insert statements into a ksqldb-cli shell as explained in the tutorial.

Running either:

import json
import requests

query = """SELECT * FROM riderLocations
  WHERE GEO_DISTANCE(latitude, longitude, 37.4133, -122.1162) <= 5 EMIT CHANGES;"""

host = "http://0.0.0.0:8088"

print("query")
r = requests.post(
    f"{host}/query",
    json={"ksql": query, "streamsProperties": {},},
    headers={
        "Accept": "application/vnd.ksql.v1+json",
        "Content-Type": "application/vnd.ksql.v1+json",
    },
    timeout=None,
    stream=True,
)
print(r)

for chunk in r.iter_content():
    print("chunk")
    print(chunk)

print("done")

or

import httpx
import asyncio

query = """SELECT * FROM riderLocations
  WHERE GEO_DISTANCE(latitude, longitude, 37.4133, -122.1162) <= 5 EMIT CHANGES;"""

host = "http://0.0.0.0:8088"


async def main():
    client = httpx.AsyncClient()

    print("query")
    async with client.stream(
        "POST",
        f"{host}/query",
        json={
            "ksql": query,
            "streamsProperties": {},
        },
        timeout=None,
        headers={
            "Accept": "application/vnd.ksql.v1+json",
            "Content-Type": "application/vnd.ksql.v1+json",
        },
    ) as r:
        print(r)
        async for chunk in r.aiter_raw():
            print("chunk")
            print(chunk)

    print("done")


asyncio.run(main())

Expected behavior
I expect the changes to be emitted in the python print statements

Actual behaviour
Empty stream:

▶ python test2.py
query
<Response [200]>

[... waiting indefinitely]

Additional context
Running the same query in a ksqldb-cli shows a nice table of changes like expected.
Even in postman I don't get any streaming content other than 187 bytes of headers.

[wilcoschoneveld]

@agavra maybe you can point me in a direction? I'm very interested to get started with ksql but I can't get basic usage working.

[lecardozo]

I was having the same issue. I could make it work with urllib3:

import json
import urllib3

http = urllib3.PoolManager() 
body = {"ksql": "SELECT * FROM DOCUMENTS"}
url = "http://localhost:8088/query"
resp = http.request(url=url, body=json.dumps(body), method="POST", preload_content=False)
for i in resp.stream():
    print(i)

[wilcoschoneveld]

@lecardozo thanks! that works for a prototype. But I think this issue is still unsolved, my intention is to re-check with API v2

[purplefox]

I'm happy to look at this, but I'm not a Python developer. @wilcoschoneveld I tried to run your example but it complained about the line "f"{host}/query",", and when I changed that to be a constant string it complained that it couldn't find "requests".
I am have Python 2.7.16 installed.
I'm guessing this is a Python version issue and/or there are other things that need to be installed first to get this to work.
Could you provide some more detailed instructions on the python setup needed so I can reproduce this?

[purplefox]

Ok, I finally managed to run your script by installing Anaconda ;) https://xkcd.com/1987/
Using current master - it receives all rows as expected, so I can't currently reproduce. Please see my output here https://gist.github.com/purplefox/873f82f325a4b0001facfb86cb2faea4 (I removed the print "chunk" as it wasn't adding much.
There is one weirdness though - it seems your python script is displaying each character as a single chunk - don't know why, but I did debug the server to check that each row was being sent as a chunk, and it was.
I also made the same request using curl --raw and the chunks were being sent as expected.

[purplefox]

@wilcoschoneveld Could you try reproducing with current ksqlDB master? There have been some changes recently in the way the API is served that might affect your results.

[wilcoschoneveld]

Hey @purplefox, thanks for looking into this! I stopped evaluating ksqldb but might look into it some time later this year. I saw that there was work being done on a next version of the API so I understand things have changed.

If you tested my script and it works now, it sounds like the issue is resolved. It could be that the python script is displaying each character as a single chunk because of the iter_content() default chunk size of 1, should be easy to change.

I'm closing this issue. Thanks again!