pulsar io hdfs could not execute

[handywc]

Describe the bug

execute method：

sinkconfig.yaml configuration：

but the log shows some error:

if i change the configuration of directory to “/tmp/bar”, it will write the file to the local file system

and even I keep produce messages to the topic, but the local file still is not writen.

pulsar 2.6.0
puslar-io-hdfs2-2.6.0.nar

could you give some advice，please？
Thank you.

[BewareMyPower]

See https://github.com/apache/pulsar/blob/3a298f3404d597e6a94de981c5fbe570264dcba1/pulsar-io/hdfs2/src/main/java/org/apache/pulsar/io/hdfs2/sink/HdfsAbstractSink.java#L92-L99

path is created from your directory and other config params.

The HDFS sink just uses a Hadoop client to create FileSystem and open a file under the configured directory to write or append. So you should not add any prefix to the path of directory.

if i change the configuration of directory to “/tmp/bar”, it will write the file to the local file system

It means there's something wrong with your hdfs-site.xml, it cannot load the Hadoop file system, so it use local file system instead.

You can write a simple Hadoop client example to verify if the issue is related to the pdfs-site.xml, like:

Configuration conf = new Configuration();
conf.addResource(new Path(/* path of your pdfs-site.xml */));
FileSystem fs = FileSystem.get(conf);
FSDataOutputStream stream = fs.create(/* file path */);
// Then use `stream` to write some data

[handywc]

hi，tks for your reply.

I copy the core-site.xml and the hdfs-site.xml from the namenode of hadoop cluster，and the hosts file is set.

I will write a example code to verfiy if the xml configuration is correct or not later.

tks again.

[handywc]

hi
when i execute the command:

It can write to hdfs successfully.
but i execute the command:

It writes to local file system.
The different is between "localrun" and "create".

[BewareMyPower]

localrun is using bin/function-localrunner to run your sink, while create is run your sink in the functions worker.

It looks like that you deploy the functions worker with broker, so it may be different with running local runner. Could you try to deploy your functions worker separately?

[handywc]

hi
I have deployed functions worker separately.

But still like before. I change the admin-url setting to functions worker's IP and Port.

It also writes to local file system.

[BewareMyPower]

Anyway, the reason why HDFS sink wrote to local file system is that the configuration didn't work and the Hadoop FileSystem use file:/// as the default file system.

But it's weird that local runner works but functions worker doesn't work. Could you give some help? @wolfstudy @tuteng @congbobo184

[congbobo184]

can i have a look for you core-site.xml and hdfs-site.xml？

[handywc]

hi，I upload core-site.xml and hdfs-site.xml.
By the way, the two files is copied from my hadoop cluster.

hdfs-site&core-site.zip

[congbobo184]

@handywc can you add the hadoop.tmp.dir = /tmp/bar configuration in core-site.xml and test again?

[handywc]

hi，thanks for your reply.
I add the hadoop.tmp.dir = /tmp/bar configuration in core-site.xml and test again, but still like before.

[congbobo184]

@handywc the fs.defaultFS configuration in your core-site.xml is hdfs://dims32, should we add the port for this? like hdfs://dims32:port

[handywc]

I add the port, but still like before.

[congbobo184]

@handywc can you change the config hadoop.tmp.dir = tmp/bar and change the directory = tmp/bar then test again

[handywc]

Thank you for your patience.
I change the setting, but it's not ok.

[congbobo184]

@handywc I think the pulsar01 don't have core-site.xml, so it will work by local file system. may be you should copy the core-site.xml to pulsar01. You can try and I don't know if that's true. and then I look up the code, if you set hadoop.tmp.dir = /tmp/bar and the directory should be "", you can choose one.

[handywc]

@congbobo184 The pulsar01 doesn't have the core-site.xml. It writes to hdfs after when I copy the file to pulsar01.
Thank you.
But I have another questions.

I set the configuration of syncinterval and maxpendingrecords, but it doesn't seem to work.
I produce five messages

But the file in hdfs is still empty unless I delete the sink .

[BewareMyPower]

@handywc see https://stackoverflow.com/questions/32231105/why-is-hsync-not-flushing-my-hdfs-file

The default HDFS sink doesn't specify the UPDATE_LENGTH flag to update the metadata of namenode, see https://github.com/apache/pulsar/blob/d7f65451dadc573fc2bb75dbb03cce705ed04d0a/pulsar-io/hdfs2/src/main/java/org/apache/pulsar/io/hdfs2/sink/HdfsSyncThread.java#L73