Direct Storage from HS Devices to InfluxDB

[Cor]

Michael, which of the versions of the ddl uses the InfluxDB 1.8 API ? I mean without that /api/v2/write but just /write ?

[Cor]

In the conf file:

[http]
auth-enabled = true
pprof-enabled = true
pprof-auth-enabled = true
ping-auth-enabled = true

OK, back to version 7 of the dll. In the debug:

17-1-2021 23:13:18 119194 | SubGet URL http://192.168.1.55:8086/api/v2/buckets?name=MQTT2
17-1-2021 23:13:18 119480 | SubmitGet Finally no Response
17-1-2021 23:13:18 119483 | InfluxDB Post URL http://192.168.1.55:8086/query Data: q=CREATE DATABASE MQTT2
17-1-2021 23:13:18 119483 | InfluxDB Post Header Authorization: Basic YWRtaW46UGlrbGVfNTU5MUluZg==
17-1-2021 23:13:18 119511 | InfluxDB Post After postStream
17-1-2021 23:13:18 119511 | InfluxDB Post After postStream Finally False
17-1-2021 23:13:18 119513 | InfluxDB Post Response Requested
17-1-2021 23:13:18 119545 | InfluxDB Post Response Success {"results":[{"statement_id":0}]}

17-1-2021 23:14:39 200108 | InfluxDB Write URI http://192.168.1.55:8086/api/v2/write?bucket=MQTT2&orgID=homeseer
17-1-2021 23:14:39 200108 | InfluxDB Write Auth Authorization: Basic YWRtaW46UGlrbGVfNTU5MUluZg==
17-1-2021 23:14:39 200110 | InfluxDB Write Requesting Response
17-1-2021 23:14:39 200113 | InfluxDB Write Finally The remote server returned an error: (400) Bad Request.
17-1-2021 23:14:39 200143 | HSEvent VALUE_SET| 2048| F891BB20-180-Q319| 16,2| 16,2| 388
17-1-2021 23:14:39 200144 | InfluxDB Write URI http://192.168.1.55:8086/api/v2/write?bucket=MQTT2&orgID=homeseer
17-1-2021 23:14:39 200144 | InfluxDB Write Auth Authorization: Basic YWRtaW46UGlrbGVfNTU5MUluZg==
17-1-2021 23:14:39 200146 | InfluxDB Write Requesting Response
17-1-2021 23:14:39 200148 | InfluxDB Write Finally The remote server returned an error: (400) Bad Request.

[Michael McSharry]

(8) indicates that it does not like auth using "Token user : pass" while (7) gets past the auth, but it does not like something about the message. I had similar experience when testing V2.0. (7) is the best candidate at this time. I know you tried it before without success, but could you try again, show the response from the debug and also show what you have in the [http] section of influxDB config file.

17-1-2021 22:24:14 1111745 | InfluxDB Write Finally The remote server returned an error: [B](401) Unauthorized.[/B]

[Cor]

I use version 8 now. I also enabled auth in the conf file. I entered the credentials in the history tab.
Database is made correctly but no data transfer:

17-1-2021 22:23:05 1042195 | SubGet URL http://192.168.1.55:8086/api/v2/buckets?name=MQTT
17-1-2021 22:23:05 1042199 | SubmitGet Finally no Response
17-1-2021 22:23:05 1042199 | InfluxDB Post URL http://192.168.1.55:8086/query Data: q=CREATE DATABASE MQTT
17-1-2021 22:23:05 1042199 | InfluxDB Post Header Authorization: Basic YWRtaW46UGlrbGVfNTU5MUluZg==
17-1-2021 22:23:05 1042201 | InfluxDB Post After postStream
17-1-2021 22:23:05 1042201 | InfluxDB Post After postStream Finally False
17-1-2021 22:23:05 1042201 | InfluxDB Post Response Requested
17-1-2021 22:23:05 1042213 | InfluxDB Post Response Success {"results":[{"statement_id":0}]}

17-1-2021 22:24:14 1111697 | InfluxDB Write URI http://192.168.1.55:8086/api/v2/write?bucket=MQTT&orgID=homeseer
17-1-2021 22:24:14 1111701 | InfluxDB Write Requesting Response
17-1-2021 22:24:14 1111706 | InfluxDB Write Finally The remote server returned an error: (401) Unauthorized.
17-1-2021 22:24:14 1111741 | HSEvent VALUE_SET| 2048| F891BB20-180-Q319| 16,8| 16,8| 388
17-1-2021 22:24:14 1111741 | InfluxDB Write URI http://192.168.1.55:8086/api/v2/write?bucket=MQTT&orgID=homeseer
17-1-2021 22:24:14 1111743 | InfluxDB Write Requesting Response
17-1-2021 22:24:14 1111745 | InfluxDB Write Finally The remote server returned an error: (401) Unauthorized.

[Cor]

I am refering to your pieces of code in #82. I first tried the curl as described in the 1.8 manual. Then I tried your first curl, that went ok as well. The second curl command in your post #82 gave an error. That is in #85.

I use the default conf file fir Influx, no auth yet.

[Michael McSharry]

Michael, BTW, the first piece of code is working OK
The second gives an error :

I am confused what this referenced. Is it the plugin version of a few weeks ago or is it what is in #84? Did you give (8) a try as it should have the same result as #85 and the original plugin release with InfluxDB capability.

Note that (8) changes only the write header. It continues to use Basic auth for the create DB.

Also what are you using in the config file [http] section?

[Cor]

I have auth protection switchet off at the moment

[Cor]

Here is the same as (7) except auth header is the Token username : passwords

I did not see where you used auth in the curl line. Did you remove it for viewing or your database no longer has auth protection?

I used the examples in the 1.8 manual, after that your code.
sorry for that misinderstanding

[Cor]

Michael, BTW, the first piece of code is working OK
The second gives an error :

{"error":"retention policy not found: rp"}

[Michael McSharry]

Here is the same as (7) except auth header is the Token username : passwords

I did not see where you used auth in the curl line. Did you remove it for viewing or your database no longer has auth protection?

[Cor]

yes, they work:

homeseer@RPi4-HS4:~ $ curl -i -XPOST http://192.168.1.55:8086/query --data-urlencode "q=CREATE DATABASE mydb"
HTTP/1.1 200 OK
Content-Type: application/json
Request-Id: c9f3393f-5903-11eb-8001-b827eb3324b2
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.8.3
X-Request-Id: c9f3393f-5903-11eb-8001-b827eb3324b2
Date: Sun, 17 Jan 2021 20:37:17 GMT
Transfer-Encoding: chunked

{"results":[{"statement_id":0}]}
homeseer@RPi4-HS4:~ $
homeseer@RPi4-HS4:~ $ curl -i -XPOST 'http://192.168.1.55:8086/write?db=mydb' --data-binary 'cpu_load_short,host=server01,region=us-west value=0.64 1434055562000000000'
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: 1a4969a2-5904-11eb-800c-b827eb3324b2
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.8.3
X-Request-Id: 1a4969a2-5904-11eb-800c-b827eb3324b2
Date: Sun, 17 Jan 2021 20:39:32 GMT

homeseer@RPi4-HS4:~ $

On the Influx system:

pi@RPI3-Influx:~ $ influx
Connected to http://localhost:8086 version 1.8.3
InfluxDB shell version: 1.8.3
> show databases
name: databases
name
----
_internal
MQTT2
MQTT
mydb
> use mydb
Using database mydb
> show measurements
name: measurements
name
----
cpu_load_short
> select * from cpu_load_short
name: cpu_load_short
time host region value
---- ---- ------ -----
1434055562000000000 server01 us-west 0.64
>

[Michael McSharry]

The 1.8 and 2.0 forms of the write API are shown at Write data with the InfluxDB API | InfluxDB OSS 1.8 Documentation (influxdata.com). Can you try using curl to write using each of the two formats. The plugin is using the 2.0 format which is advertised to work with 1.8 too.

Actually this is a poor reference. I will find a better one for you.

What I had originally (and what I think was working for you) was based upon InfluxDB API reference | InfluxDB OSS 1.8 Documentation (influxdata.com)

The simplified version that I used was:

curl -XPOST "localhost:8086/api/v2/write?bucket=db" -H 'Authorization: Token <username>:<password>' --data-raw "mem used_percent=23.43"

The full form on the reference side is:

curl -XPOST "localhost:8086/api/v2/write?bucket=db/rp&precision=s" -H 'Authorization: Token <username>:<password>'  --data-raw "mem,host=host1 used_percent=23.43234543 1556896326"

Since then I changed to use Basic rather than Token auth. The response code you are getting 400 is a format and not an auth error, but perhaps the format it wants is the Token auth format.

In another reference the endpoint is /write rather than api/v2/write. I never tried this as I wanted to be compatible with both versions with same code.

curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary 'cpu_load_short,host=server01,region=us-west value=0.64 1434055562000000000'

You can also try to use curl with the Basic auth using -H 'Authorization: Basic xxxxxxx' rather than -H 'Authorization: Token <username>:<password>' where you can get the xxxx from some of the prior debug outputs.

[Cor]

With version 7 no change.
Database creation yes, no data transfer.
I have no idea how wireshark works and how I can test that locally, sorry for that.

[Michael McSharry]

Organization ID is only needed when using V2.0. It is ignored on V1.8

Below are three Wireshark screenshots. The first two are the request to write data from mcsMQTT and the response to this request from InfluxDB 1.8. The response code of 204 is what is expected when data is successfully written.

In my testing the (400) bad request was overcome by changing the mix of header tags in the HTTP request. I believe the correct header for content-type is text rather than url-encoded since 1.8 is expected the space to be escaped rather than %20. I changed it to url-encoded because that is what curl sent. You can try (7) to see if it makes any difference.

The third image is with the text content-type request made using the (7) attached. InfluxDB also responded with success and data was written with this header. It is what I will leave in the plugin because I believe it is the correct content-type for this case.

[Cor]

@https://forums.homeseer.com/member/40973-michael-mcsharry
Where is that Organization id for? Has is any use?