Hardware: ESP-12E
Core Version: 2.1.0-rc2
"Internal" time.c functions always return 0 /1970-01-01/ if not synced via NTP.
I want to initialize internal clock via constant or from RTC for log timestamping (so I don't need precise time on startup, but valid measured intervals)
Module: Generic ESP8266 Module
Flash Size: 4MB
CPU Frequency: 80Mhz
Flash Mode: qio
Flash Frequency: 80Mhz
Upload Using: SERIAL
Reset Method: ck
#include <Arduino.h>
#include <time.h>
void setup() {
// WiFi.begin(); // wifi disabled for example
// configTime(3*3600, 0, "not.avail.ntp");
Serial.begin(115200);
}
void loop() {
time_t now = time(nullptr);
Serial.println(ctime(&now));
delay(1000);
}
Thu Jan 01 00:00:00 1970
Thu Jan 01 00:00:00 1970
...
Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.
I use this workaround for now:
static uint32_t starttime = millis() / 1000;
static uint32_t localtime = starttime;
void time_set(uint32_t time) {
starttime = millis() / 1000;
localtime = time;
}
uint32_t time_get(void) {
return (millis() / 1000 - starttime + localtime);
}
I m looking for full software rtc in core libraries too.
Thanks, @everslick!
I have my implementations RTC and/or NTP sync from my "raw C" projects on AVR, but I want to use "embedded" code because it already exists.
@far5893 There is widely known Time library from arduino.cc, but it is not compiled with 2.1.0-rc2 for me. And of course, this is not core lib
@igrr Please take attention to string handling in sntp_xxx code. I have got exception 28 (or sometimes 9) with sntp_asctime[_r] and sntp_sync_xxx approx after 22-24 hours uptime. there is only ctime(&now) calls in my code. One of saved traces:
Decoding 17 resultsException code: 28: LoadProhibited: A load referenced a page mapped with an attribute that does not permit loads
0x402265f9: sntp_asctime_r at ??:?
0x402265f9: sntp_asctime_r at ??:?
0x4020d6cc: TwoWire::requestFrom(unsigned char, unsigned int, bool) at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/libraries/Wire/Wire.cpp:240
0x40226647: sntp_asctime at ??:?
0x4020d6f8: TwoWire::requestFrom(unsigned char, unsigned char) at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/libraries/Wire/Wire.cpp:240
0x40201b44: asctime at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/cores/esp8266/time.c:110
0x4020df5e: Adafruit_BMP280::read24(unsigned char) at /home/av/Arduino/libraries/Adafruit_BMP280_Library/Adafruit_BMP280.cpp:210
0x40209742: check_float(float, float, float&) at /home/av/bin/arduino-1.6.5/WebConfig.ino:311
0x4020986e: readSensors() at /home/av/bin/arduino-1.6.5/WebConfig.ino:311
0x40105c2e: ets_timer_setfn at ??:?
0x4020a534: ESP8266WiFiSTAClass::status() at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/libraries/ESP8266WiFi/src/ESP8266WiFiSTA.cpp:635
0x4020ad5c: WiFiClient::operator=(WiFiClient const&) at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/libraries/ESP8266WiFi/src/WiFiClient.cpp:362
0x40209e08: every_s() at /home/av/bin/arduino-1.6.5/WebConfig.ino:311
0x40209eb8: handle_ms() at /home/av/bin/arduino-1.6.5/WebConfig.ino:323
0x40209ef3: loop at /home/av/bin/arduino-1.6.5/WebConfig.ino:337
0x4020f6f8: loop_wrapper at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/cores/esp8266/core_esp8266_main.cpp:43
0x40100114: cont_norm at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/cores/esp8266/cont.S:109
sntp_xxx functions come from espressif SDK, as far as I can tell.
On Sun, Feb 28, 2016, 13:04 Alexander Voronin [email protected]
wrote:
Thanks, @everslick https://github.com/everslick!
I have my implementations RTC and/or NTP sync from my "raw C" projects on
AVR, but I want to use "embedded" code because it already exists.@far5893 https://github.com/far5893 There is widely known Time library
from arduino.cc, but it is not compiled with 2.1.0-rc2 for me. And of
course, this is not core lib@igrr https://github.com/igrr Please take attention to string handling
in sntp_xxx code. I have got exception 28 (or sometimes 9) with
sntp_asctime[_r] and sntp_sync_xxx approx after 22-24 hours uptime. there
is only ctime(&now) calls in my code. One of saved traces:Decoding 17 resultsException code: 28: LoadProhibited: A load referenced a page mapped with an attribute that does not permit loads
0x402265f9: sntp_asctime_r at ??:?
0x402265f9: sntp_asctime_r at ??:?
0x4020d6cc: TwoWire::requestFrom(unsigned char, unsigned int, bool) at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/libraries/Wire/Wire.cpp:240
0x40226647: sntp_asctime at ??:?
0x4020d6f8: TwoWire::requestFrom(unsigned char, unsigned char) at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/libraries/Wire/Wire.cpp:240
0x40201b44: asctime at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/cores/esp8266/time.c:110
0x4020df5e: Adafruit_BMP280::read24(unsigned char) at /home/av/Arduino/libraries/Adafruit_BMP280_Library/Adafruit_BMP280.cpp:210
0x40209742: check_float(float, float, float&) at /home/av/bin/arduino-1.6.5/WebConfig.ino:311
0x4020986e: readSensors() at /home/av/bin/arduino-1.6.5/WebConfig.ino:311
0x40105c2e: ets_timer_setfn at ??:?
0x4020a534: ESP8266WiFiSTAClass::status() at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/libraries/ESP8266WiFi/src/ESP8266WiFiSTA.cpp:635
0x4020ad5c: WiFiClient::operator=(WiFiClient const&) at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/libraries/ESP8266WiFi/src/WiFiClient.cpp:362
0x40209e08: every_s() at /home/av/bin/arduino-1.6.5/WebConfig.ino:311
0x40209eb8: handle_ms() at /home/av/bin/arduino-1.6.5/WebConfig.ino:323
0x40209ef3: loop at /home/av/bin/arduino-1.6.5/WebConfig.ino:337
0x4020f6f8: loop_wrapper at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/cores/esp8266/core_esp8266_main.cpp:43
0x40100114: cont_norm at /home/av/.arduino15/packages/esp8266/hardware/esp8266/2.1.0-rc2/cores/esp8266/cont.S:109—
Reply to this email directly or view it on GitHub
https://github.com/esp8266/Arduino/issues/1679#issuecomment-189833248.
@igrr Huh :( May be some try/catch over sntp_xxx in esp8266/time.c ? But I'm not sure this may help
Maybe you will have a look at this library:
https://github.com/Juppit/esp8266-SNTPClock
It was developed from a version with ticker library without network
access. Therefore uncommenting the network stuff it should still give
you time functions.
Am 28.02.2016 um 11:30 schrieb Alexander Voronin:
@igrr https://github.com/igrr Huh :( May be some try/catch over
sntp_xxx in esp8266/time.c ? But I'm not sure this may help—
Reply to this email directly or view it on GitHub
https://github.com/esp8266/Arduino/issues/1679#issuecomment-189837221.
Wrong thread, I guess.
There was someone looking for time finctions without network.
@Juppit Your version have same issue as this - system snmp_xxx functions does not provide correct time if no ntp sync was made prior. Another bug is in SNTPClock::begin - while(!m_secsSince1900) loop never ends if no network/success ntp sync made
These are not c++ exceptions, these are CPU exceptions, so try/catch won't help here. Do you have a test case which is failing? I'd rather try to reproduce it and report to Espressif.
@igrr I don't know how to reproduce it a bit stable. I'll try write minimal code only for timesync/print just now, but I have no idea how to speed up testing - 20+ hours before exception.. may be wifi connection issue while sync, may be some string formatting...
20 hours is not that bad, I can probably hook it up to gdb and let it run
until it crashes.
On Sun, Feb 28, 2016, 14:50 Alexander Voronin [email protected]
wrote:
@igrr https://github.com/igrr I don't know how to reproduce it a bit
stable. I'll try write minimal code only for timesync/print just now, but I
have no idea how to speed up testing - 20+ hours before exception.. may be
wifi connection issue while sync, may be some string formatting...—
Reply to this email directly or view it on GitHub
https://github.com/esp8266/Arduino/issues/1679#issuecomment-189851050.
hey, if I got this right you are looking for some RTC capabilities. I have used the "TimeLib.h" for that. To initialize the time, I send a head request to a google server. I sync it every night and it's running since 2 weeks now...
@kaeferfreund Nope. I am looking for system capability to work w/o internet/NTP server. I know about TimeLib (distributed now as arduino.cc's Time.h). My use case is sensor, that may be sometimes offline and may go online w/o internet/NTP access. I plan to use DS3231 as backup RTC in my v3.1 PCB design but I just don't want to use third party lib where an system implementation exists.
@igrr, 5hrs. Still working. Neither blocking port 123 on router, nor disabling wifi access does not reproduce fail. But for now I know time intervals of internal time sync )) - if NTP accessible, then sunc performed every hour, if not - every approx 33seconds. PS: I am not familiar with gdb, but try tomorrow at work (not sure it will work on cubietruck, sot first try on PC)
@igrr Three day test - all ok, no crash. _May_ be an issue with printing asctime(nullptr) - this call immediately throws exception 28. But I have no code with this call, only commented out from old exercises.
I'll try with more complex code and open separate issue if found something.
FYI: I have the feeling, that stack growth has to do with Exception(28). I
could not find out yet, how to measure stack usage. I guess the stack
collides with heap occasionally. I could prevent Exception(28 by moving
stack variables to the heap.
On Wed, Mar 2, 2016 at 4:32 PM, Alexander Voronin [email protected]
wrote:
@igrr https://github.com/igrr Three day test - all ok, no crash. _May_
be an issue with printing asctime(nullptr) - this call immediately throws
exception 28. But I have no code with this call, only commented out from
old exercises.I'll try with more complex code and open separate issue if found something.
—
Reply to this email directly or view it on GitHub
https://github.com/esp8266/Arduino/issues/1679#issuecomment-191288577.
You can measure stack usage for the code which runs on Arduino task (i.e. called from setup and loop). See https://github.com/esp8266/Arduino/blob/master/cores/esp8266/cont.h#L65.
For code which runs on other tasks, there is no way to check this at the moment.
Edit:
the cont_t structure which has to be passed into this function is not currently exposed to the user. It is declared here: https://github.com/esp8266/Arduino/blob/master/cores/esp8266/core_esp8266_main.cpp#L64. I think you can declare it as extern in your sketch as a workaround.
In my case Exception 28 usually indicate incorrect usage of pointers - asctime(NULL) as above and mostly print(PSTR/FPSTR/F(...)). In most cases I use something like print(PSTR("debug")) instead of print(F("debug")) and got error as should.
UPD: tested with cont_get_free_stack(). Got about 3136-3144 of 4096 bytes, looks ok )) and 35200 bytes of heap (I2C, Web Server, 5x100words sensor logs,1500bytes String in web server handler)
@Ivan: thanx for the hint. but i'm not sure, if i do it right:
i create a cont_t object:
static cont_t stack;
in setup() i call:
cont_init(&stack);
in loop() i do:
LogPrint(" free stack: %i bytes\n", cont_get_free_stack(&stack));
LogPrint(" stack guard: %s\n", cont_check(&stack) ? "CORRUPTED" :
"OK");
every two seconds.
but the log always reads:
free stack: 4100 bytes
do i have to spawn a separate main loop with my cont_t object using
cont_run() ?
On Wed, Mar 2, 2016 at 6:13 PM, Alexander Voronin [email protected]
wrote:
In my case Exception 28 usually indicate incorrect usage of pointers -
asctime(NULL) as above and mostly print(PSTR/FPSTR/F(...)). In most cases
I use something like print(PSTR("debug")) instead of print(F("debug"))
and got error as should.—
Reply to this email directly or view it on GitHub
https://github.com/esp8266/Arduino/issues/1679#issuecomment-191329715.
Not exactly, you need to use the existing object.
extern "C" cont_t g_cont;
loop() {
size_t free = cont_get_free_stack(&g_cont);
// etc
}
P.S. for further discussion please visit our gitter chat or open a separate issue, let's not hijack the original issue :)
On Thu, Mar 3, 2016 at 4:32 PM, Ivan Grokhotkov [email protected]
wrote:
extern "C" cont_t g_cont;
ahh, thx. without calling cont_init() on it i guess ?
HI, Is this issue still open or closed?
Having the same problem here, time() api returns "Thu Jan 01 00:00:00 1970".
looks like sntp.c never gets compiled from my Arduino IDE. Had inserted asserts under sntp_request and also enabled LWIP_DEBUGF debug logs. No core or logs seen.
Same problem here. Any workarounds?
it looks like
a) s_bootTime only gets set on first use of _gettimeofday_r() what seems to be some helper function for the newlib libc.
b) s_bootTime is only regarded via libc functions.
time() definitely does not care about s_bootTime.
So no changes when using bare metal sntp functions ...
to me it would somehow make sense if the status of sntp_init() in configTime() would be stored as flag and subsequent requests for a timestamp would return either an actual fresh timestamp or the (s_bootTime +) millis()/1000 depending on this flag.
Optional an input_timestamp (from some rtc_chip) could be provided as parameter in configTime() to be used to calculate s_bootTime if sntp_init() fails.
With current master, the above sketch works
extern "C" void sntp_set_system_time(uint32_t);
and in setup():
sntp_set_system_time(0);
Similar could be done with lwip1.4 but not without patching source and recompiling, just because all functions and variables are statically linked in sntp.c (which has public sntp_get_current_timestamp() used by ctime()).
edit: The main goal here, in both case, is to install the os_timer_setfn(sntp_time_inc) function.
somehow the whole time() thing looks pretty uncared of and broken ...
best to just use bare metall functions and 3rd party time(zone)lib.
P.S. with sdk sntp functions I can't connect to timeservice on M$servers, so I have to use "hand coded" timeservice after all.
P.P.S.: I am using current master and get the same result as the OP (lwip variant set to V2)
With example sketches, we can try and help.
PS: How do you use your ntp servers ?
PPS: The OP has to be modified with the two lines I showed.
We could propose a PR to make this automatic (without these two lines).
sntp timer would be initialized at boot and reinitialized when dhcp or user gives a ntp server.
Note that lwipv2 automatically uses dhcp-advertised ntp servers. Initially for me the OP sketch was working unmodified with current NTP time, and I had to ESP.eraseConfig() to go back 50 years ago.
ok, I can confirm that sntp_set_system_time(0); helps a lot.
I modified time.c / configTime()and inserted this after sntp_set_timezone() and now the call to configTime() starts the "fake rtc" and - if WiFi is working and the server responds - eventually the time will be correct even if WiFi drops later on.
Still an optional parameter to configTime() to set time to a value taken from rtc chip would be nice to have. After this simple single initialisation all time operations could be done via standard functions.
now someone would have to set up a fake ntp server to check what dalightsaving rules espressif have implemented ..
P.S.: I use the M$ server exactly the same: just use its name instead of pool.ntp.org.
With hand coded udp requests I get an answer, sdk functions don't ...
Unfortunately I can't use wireshark, my IT-dept ...
#include <ESP8266WiFi.h>
#include <time.h>
#define ssid "your_ssid"
#define wpwd "your_secret"
// extern "C" void sntp_set_system_time(uint32_t);
void setup() {
Serial.begin(74880);
WiFi.persistent(false);
WiFi.mode(WIFI_STA);
WiFi.begin(ssid, wpwd);
configTime(3600, 3600, "pool.ntp.org");
// sntp_set_system_time(0);
}
void loop() {
time_t tnow = time(nullptr);
Serial.println(ctime(&tnow));
delay(1000);
if (millis()==30000) {
WiFi.disconnect();
WiFi.mode(WIFI_OFF);
Serial.println("WiFi off");
}
}
You can fake using an external RTC with sntp_set_system_time(secs-since-epoch).
I fixed lwip2 so:
sntp_set_system_time() is exposed.sntp_set_system_time(0) is called at lwip init.configTime() can be used. Currently they are overridden by config to ntp.pool.org, it explains why you could not reach your preferred/authorized ntp server.a PR is soon on its way
ok, so I see that the issue will be fixed with bells-and-whistles.
To spare you of further issues like this I suggest to do at least some rudimentary documentation on how to use the core time.c and the necessary preconditions.
@d-a-v
Hi, had finally time to check your changes (used your fork) and time now starts w/o prior contact to ntp server, BUT
a) I was not able to use sntp_set_system_time(0); without prior declaration
b) TZ is somehow wrong.
if you test my small sketch with master, you will get following output:
with correct offset for MEZ before & after sync with ntp server as done on Sun Nov 12 12:09:06 2017 (shortly after noon).
edit: BUT even with "sntp_set_system_time(1499893466)" (a timestamp during DST period) it shows only localtime as MEZ not MESZ (ignoring dst offset), so proofing #2505 as valid.
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
load 0x4010f000, len 1384, room 16
tail 8
chksum 0x2d
csum 0x2d
v00000000
~ld
Thu Jan 1 01:00:00 1970
Thu Jan 1 01:00:01 1970
Thu Jan 1 01:00:02 1970
Thu Jan 1 01:00:03 1970
Thu Jan 1 01:00:04 1970
Thu Jan 1 01:00:05 1970
Thu Jan 1 01:00:06 1970
Sun Nov 12 12:09:06 2017
Sun Nov 12 12:09:07 2017
Sun Nov 12 12:09:08 2017
with your git it gives (with / without sntp_set_system_time(0))
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
load 0x4010f000, len 1384, room 16
tail 8
chksum 0x2d
csum 0x2d
v00000000
~ld
Thu Jan 1 02:00:00 1970
Thu Jan 1 02:00:01 1970
Thu Jan 1 02:00:02 1970
Thu Jan 1 02:00:03 1970
Thu Jan 1 02:00:04 1970
Thu Jan 1 02:00:05 1970
Thu Jan 1 02:00:06 1970
Sun Nov 12 13:13:14 2017
Sun Nov 12 13:13:15 2017
******************************************************
ets Jan 8 2013,rst cause:2, boot mode:(3,7)
load 0x4010f000, len 1384, room 16
tail 8
chksum 0x2d
csum 0x2d
v00000000
~ld
Thu Jan 1 08:00:00 1970
Thu Jan 1 08:00:01 1970
Thu Jan 1 08:00:02 1970
Thu Jan 1 08:00:03 1970
Thu Jan 1 08:00:04 1970
Thu Jan 1 08:00:05 1970
Thu Jan 1 08:00:06 1970
Sun Nov 12 13:14:18 2017
Sun Nov 12 13:14:19 2017
as you can see in first case ds offset is wrongly applied
in second case TZ seems to remain standard sdk value first, correct TZ only after sync with ntp server but ds offset again wrongly applied
sntp_set_system_time() is exposed in lwip include files.
You must include <arch/cc.h> or anything from lwip2 to have it.
I don't know if I should make it exposed for lwip1.4 too...
About DST, I just tried it again:
It is now Mon Nov 13 01:39:01 CET 2017 on my laptop and my wall-clock (though the latter does not tell me TZ, and DST was sadly set about 2 weeks ago by hand... but agrees with my computers).
With winter time configTime(3600,0,"ntp.pool.org") which is MEZ (and CET), it esp-sketchly is Mon Nov 13 01:42:02 2017.
With summer time configuration configTime(3600,3600,"ntp.pool.org") which is MESZ (and CEST too) esp says Mon Nov 13 02:42:48 2017.
In your two last examples, what where configTime() parameters ?
How could you verify the validity of the patch with 1499893466 ? I guess you have the matching local time. It was in July'17, did you use it with (TZ,DST) = (3600,0)=(edit)MESZMEZ or (3600,3600)=MEZMESZ ?
HI,
ok there is not much sense in just adding dst-offset if it is present and not according to a dst rule.
If I have to decide if dst applies or not prior to call of configTime() I can just add it to the TZ value...
So to sum it up: as long as (underlying sdk) function does not use dst-rule for applying dst-offset (and this is plausible explanation for observed problems) it is of no use
DST is a difficult problem which seems not to be properly solved though I am not a specialist. There are two main cases, when you can access to internet, or when you can't. Check this arduino lib and also this.
If someone comes with a useful library, it might be worth a PR.
Take a look at this.
So what about timestamps in periods with other rules?
I have just learned from it that e.g. in China was no DST after 1991.
I would have been "too good to be true" to have correct dst rules in sdk time functions.
Without major changes (forget about TZ as integer, string needed) there is no decent implementation.
With only integer TZ=XY you still could be in an area with different offset (0, 1800, 3600) or with different start/end dates, therefor textual TZ is needed.
So the achievable optimum is reliably start time() on startup with TZoffset from configTime() to get UTC, have sntp_set_system_time() properly exposed (including cc.h is even worse than have declaration).
Either place it in time.c or have ESP.time(), ESP.configTime() and ESP.sntp_set_system_time()
Then it is up to the user to have correct TZdst handling via some TZ lib.
@Juppit unix-like "timestamps" are allways in UTC (ok, not really, no leapseconds) and have to be "translated" to local time. "DateTime" timestamps should never be used w/o postfix giving actual offset to UTC.
@igrr (@devyte ?)
What do you think of exposing sntp_set_system_time() globally (in #3808) along with configTime() in Arduino.h ?
or with a better arduinishCapsed name like setTime()/setEpoch() ?
How about overriding settimeofday to call sntp_set_system_time? That would
at least be posix-y and also compatible with the ESP32, where
settimeofday/gettimeofday work as one would expect.
On Mon, Nov 13, 2017 at 9:55 PM, david gauchard notifications@github.com
wrote:
@igrr https://github.com/igrr (@devyte https://github.com/devyte ?)
What do you think of exposing sntp_set_system_time() globally (in #3808
https://github.com/esp8266/Arduino/pull/3808) along with configTime()
in Arduino.h ?
or with a better arduinishCapsed name like setTime()/setEpoch() ?—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/esp8266/Arduino/issues/1679#issuecomment-343925688,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEJceoRUxdUGcDGW6L9FsPpc3sdsT-_0ks5s2EpugaJpZM4Hiela
.
@5chufti I put a library example esp8266/NTP-TZ-DST.
@igrr I did not try to fix gettimeofday @BrandonLWhite is working on it
I think implementing settimeofday as @igrr suggests sounds like the right approach. I try to prefer use of existing CRT or POSIX APIs vs. introducing new stuff like low-level lwip functions.
I completed the work for getting gettimeofday functional just now, here in a new squashed PR: #3830.
So what I'd recommend doing for a settimeofday implementation would be to extract my code
s_bootTime_us = now_s * 1000000ULL - micros64();
s_bootTimeSet = true;
into a new method, then call it for both ensureBootTimeIsSet as well as settimeofday.
I guess @igrr should take your PR first, then I'll adapt mine to yours.
@5chufti
I would have been "too good to be true" to have correct dst rules in sdk time functions.
I agree. I like to think of timezones, DST, localtime as a user interface display concern. Logic and event/data logging should stick to UTC. You can certainly convey what the timezone is for the device, so that other systems can be aware and format accordingly. As has been mentioned, you should use a textual ID such as an Olson ID. A simple UTC offset is generally not good enough.
So, I agree that timezone and DST support is not within the scope of the core lib. If someone really wants or needs to build localized times in their firmware that should be accomplished with libraries. That would have to pull in all of the tz database and logic and that's quite some baggage! That's why I'd avoid it in an MCU. Even if you are serving a web app straight from the ESP8266, you should use a JS client lib like moment.js to format time displays in a human format. That can even be done with regard to future changes to DST rules without having to update your device firmware!
Thanks, just my 2 cents on the matter.
Merged #3818, thanks!
One data point the topic of timezone/dst support. Newlib (C library which we use) knows how to handle timezones and DST. Application needs to supply the timezone description (which may include DST rule), and newlib will take it into account when e.g. localtime_r is called.
So an application can use it like this:
https://github.com/espressif/esp-idf/blob/master/examples/protocols/sntp/main/sntp_example_main.c#L76-L87
Obviously the application needs to store the list of timezone descriptions somewhere (possibly mapped to user-friendly names of regions or major cities in each timezone). That, indeed, should be application's responsibility. But there is a project which can help generating TZ strings from the timezone database: https://github.com/nayarsystems/posix_tz_db.
@BrandonLWhite
Hi, maybe I am wrong but isn't the whole "save bootTime" thing the wrong way around?
With your patch (in progress) we allways will have the "bootTime" for at least as long as int64 µs is worth (AFAIK more than a lifespan).
how about for time() just:
°) get real timestamp via sdk
°) if valid (t != 0) store diff to micros64() for later and return
tv_sec=timestamp
tv_usec=micros64() % 1000000
°) if not successful just return
tv_sec=(micros64() + diff)/1000000
tv_usec=micros64() % 1000000
this should return time from boot until ntp sync and +/- UTC after first sync?
reminder: we still need the correction of hardwired ntp servers from d-a-v patch.
edit: for use with rtc chip, sntp_set_system_time() would just update "diff".
@igrr
One data point the topic of timezone/dst support. Newlib (C library which we use) knows how to handle timezones and DST. Application needs to supply the timezone description (which may include DST rule), and newlib will take it into account when e.g. localtime_r is called.
Ah, yes of course, I completely forgot about that! So this is still not a fully capable local time implementation, but it is good enough for getting local time pretty much "right now" and light duty UTC-localtime conversions. The reason is that the TZ string that is used only supports a single DST forward/back rule. So therefore all calculations must assume that is the rule for all of time. While that is not correct for a great number of locales, it is acceptable for most applications.
You are right. This capability exists in the runtime and people can and should use it if it meets their needs.
@5chufti
2^64 us is about 584,554 years, so there is no danger there.
The time() CRT function only returns a 32-bit seconds-since-epoch value. So there is no sub-second component.
Regarding your example, I would be hesitant to mix values from different unrelated timing sources on each call. There is no observable relationship between micros64 (monotonic) and sntp_get_current_timestamp().
isn't the whole "save bootTime" thing the wrong way around?
It isn't ideal, but I think it's the best that can be achieved given what is available from the SDK. Since the SDK can only yield 32-bit seconds, it is limited in terms of precision and Y2038. It is good enough for use with the CRT time() function, but is not suitable for gettimeofday. So for gettimeofday we can use the SDK's 32-bit value as a reference point, and then achieve precision and longevity by combining it with a 64-bit monotonic clock.
This would be unnecessary if the SDK provided a 64-bit way to get the current real time, with either us or ms precision.
@BrandonLWhite
I took your micros64() implementation and the attached time.c
Ofcourse it would be also possible to do sync only once and subsequently only if configTime() was called again if you have doubts on monotonicity. But the rest should be as monotonic as your micros64().
I did test it for general function and it works on old and new lwip.
In #3835, there is still an issue with gettimeofday() if it is called for the first time before sntp-date is set by ntp. It works well after, or with settimeofday().
A solution would be to reset s_bootTimeSet when ntp is set (or adjusted?)
Can it be reset anytime with no harm ?
(note that this PR is now on my ntp branch if you try that way)
With the latest #3835 commit (not yet merged, or my ntp branch), the main bug is gettimeofday() which is always wrong if it is called first time before a ntp date is set.
A good way to fix this would be to be able to re-set s_bootTimeSet and al. everytime ntp receive a new timestamp. It is done though sntp_set_system_time() called from both settimeofday() and sntp from inside lwip through an lwip-api-#define.
sntp_set_system_time() is defined in lwip2, because it was defined inside lwip1.4. @igrr I think this code, which was introduced by espressif inside lwip - where it does not belong to - should be moved into cores/esp8266, but that would introduce a change in lwip1.4 too (we could move it anyway from lwip2 with attribute((weak)) and no change in lwip1.4)
In either case, sntp_set_system_time() can take one (epoch) or two (epoch, usec) arguments (it's a choice, it currently is one).
Do you think time.c can be changed to be able to be updated/reset/restarted through a call from sntp_set_system_time(epoch,usec) (ie. from external RTC or SNTP) ?
I very much agree about removing the time keeping functions from LwIP SNTP module.
In my mind, SNTP should just call settimeofday (or adjtime, if we can implement it) when new timestamp is received.
All timekeeping should be done in the core. All timezone management should be done in newlib. I think that having LwIP SNTP responsible for timekeeping is crazy, but apparently many of our customers already use these extra SNTP module functions, so they are not going away in the ESP8266 SDK.
But since we are moving towards using upstream LwIP in this project, i think we should nuke these additions and do things the right way.
already in progress, commit is coming :)
Complete commit log:
moved time functions from lwip2 to core
- sntp_set_system_time() is not anymore used by lwip2 nor callable, settimeofday() is the entry point
- ntp timestamps or settimeofday() now reset time functions, and give (sec,usec) instead of (sec)
- update esp8266/NTP-TZ-DST example accordingly
- new core files: coredecls.h and sntp-lwip2.c
- update lwip2 commit ref and lib accordingly
further work has to be done in settimeofday() to properly take tv_usec into account
I will let you fix tv->tv_usec in settimeofday(), I don't feel I'd be good at it :)
master's core has been updated
settimeofday() receiving but not honouring tv_usec (sntp or user calling) adjtime()@igrr thanks !
Hi,
I think a little more info is needed:
what is the result of "time();"?
a) an internal timer based value or b) a real ntp-server answer?
if a) how often/when is this internal timer synced with a ntp-server (if configured and active)?
can this be configured/forced (how)?
bg: I only need a longterm accuracy of appr. timestamp resolution (s), and I don't want to flood the ntp-server, so internal timer would suffice for a certain periode but eventual resync is necessary.
thanks for your patience
Without NTP, it starts from 1st january 1970 8am (it's EPOCH in Espressif's HQ). It can be forced with settimeofday().
Time is re-set / adjusted by sntp driver (edit: via NTP)
edit2: you can see time functions running with a new sketch example esp8266/NTP-TZ-DST
thank you,
I'm working my way through the "time&TZ" functions of newlib at the moment; looks good so far. If I'm using static IP and disable WiFi for energy efficiency there is no way to force "sync" during a WiFi enabled periode?
I did see the example but missed something more enduser oriented like the following on how to get a "real" time with "onboard" features.
#include <ESP8266WiFi.h>
#include <sys/time.h> // struct timeval
#include <time.h> // time() ctime()
#define ssid "yourSSID"
#define wpwd "yourPASSPHRASE"
#define RTC_TEST 1510592825 // = Monday 13 November 2017 17:07:05 UTC
// #define RTC_TEST 1499893466 // = Wed 12 July 2017 21:04:26 UTC
void setup() {
Serial.begin(74880);
WiFi.persistent(false);
WiFi.mode(WIFI_STA);
WiFi.begin(ssid, wpwd);
timeval tv = { RTC_TEST, 0 };
timezone tz = { 0 , 0 };
settimeofday(&tv, &tz);
configTime(0, 0, "pool.ntp.org");
setenv("TZ", "CET-1CEST,M3.5.0,M10.5.0/3", 3);
tzset();
// don't wait, observe time changing when ntp timestamp is received
}
void loop() {
time_t tnow = time(nullptr);
Serial.println(ctime(&tnow));
if (millis() > 30000 and millis() < 31000) {
WiFi.disconnect();
WiFi.mode(WIFI_OFF);
Serial.println("WiFi off");
}
delay(1000);
}
Do you want to avoid using an external RTC ?
Do you want to get a new NTP time stamp, and put Wifi off once received until next time you decide to resynchronize time ?
In you sketch above, use configTime to set TZ, DST and default NTP server.
sntp_request(NULL) is what is internally achieved every hour.
It can be called by the sketch.
Once NTP time stamp is received, settimeofday() is called by SNTP, then you can put Wifi off again.
There is curently no way to know when settimeofday() is called. It can be discussed later if it is needed, in the meantime you can wait a few seconds before going to sleep again and report if it fits your needs.
If I set the offset in configTime to 0,0 I know what I get: UTC.
If I set anything else the only thing I know is: its wrong 50% of the year (you cant get correct TZ rules from two plain offsets)
So I prefer to get plain UTC and have newlib take care of the TZ thing ...
And if it takes "several seconds" as I observe at the moment it is around 20!, this is not energy efficient. I speculate it is due to the 15s default of SNTP_UPDATE_DELAY that also applies to the first sync (why the "update" in the name?). It should be zero for start and some default after first successful sync.
... so no game for me, back to "manually" fetching sntp_timestamps.
SNTP_UPDATE_DELAY is the internal interval between two NTP requests in the sntp driver in a world where network is always on and available.
In your case, if you do your request by hand with sntp_request(NULL) once the network becomes available, you should get your answer right away. Did you try that ?
unfortunately: "sntp_request" was not declared in this scope
even if I declare it: undefined reference to `sntp_request'
extern "C" ?
what else ...
@5chufti I have updated lwip2 to expose sntp_request(NULL) which is static with a new function sntp_force_request().
I have patched arduino core to add a callback when settimeofday() is called.
I have updated the NTP_TZ_DST.ino example that makes use of them.
Please try and report back. You need to pull my branch goodies from scratch.
I did so because I believe in this use case :)
thank you very much, but it is not about me and my projects. I prefer a solution that will work in every upcoming release without special adaption; be it in the core or my code.
I understand this platform as part of IoT and therefor energy efficiency is relevant,
especially as in several fora I encounter requests for battery / solar powered gadgets.
thank you for your effort, your implementation and example seem to work.
BUT:
a) with all the headers to pull in and undocumented functions it is not very "arduinish" to use.
b) I think it should be emphasized in the example that there is _no automatic DST switchover_ in core functions. So maybe you would like to add few lines to setup() to show how easy real TZ setup can be achieved?
#else // ntp
configTime(0, 0, "pool.ntp.org");
settimeofday(0, nullptr);
setenv("TZ", "CET-1CEST,M3.5.0,M10.5.0/3", 3);
tzset();
WiFi.mode(WIFI_STA);
WiFi.begin(SSID, SSIDPWD);
// don't wait, observe time changing when ntp timestamp is received
#endif // ntp
I'm not belitteling your effort but I think the general arduino user is not concerned about monotonic system timestamps, they more likely are searching for local realtime. IMHO People concerned about monotonic µs timestamps should be using sdk ...
have system allways return UTC (be ntp compatible, all un*x systems I know have hw/system clock on UTC) and forget about tz and dst-offset in core functions, they are more confusing than helpfull.
It would be nice to have such an official example sketch with energy saving and NTP updates when coming up alive to keep sensor logs accurate.
Instead of adding your not-arduinish-too few lines, why wouldn't you write something like this function and propose it as a PR along with a sketch using it (a new one or the current esp8266/NTP-TZ-DST)
#define TZ_CHINA_MAINLAND "xxxyyy"
#define TZ_GERMANY "CET-1CEST,M3.5.0,M10.5.0/3"
#define TZ_...
...
void configTime (const char* TZ, const char* server1, const char* server2, const char* server3)
{
setenv("TZ", TZ, ...);
tzset()
...
}
@igrr @devyte what would you think of such an overload of configTime() ?
Hi d-a-v,
sorry if I just seem to be wasting your time.
I am not asking anybody to do something on my behalf, after 30+years in SW development I find my way around (except where problems hide in binary blobs - as is often the case with ESP).
I am just around to remind some of the devs that arduino is about "k.i.s.s." and I couldn't see it here.
Having system ignorant about TZ and DST (UTC only) is fine, so no overload is necessary, just remove the confusing remains also and give a decent example of how - even w/o additional libs - local time is achievable via ntp or rtc - easy as that. If thinking about overload, I would plead for an initial timestamp!
Alright, I've been keeping out of this discussion on purpose, mostly because I'm not using this time functionality, and because I currently don't have the details loaded into my brain of what the "standard" or "posixy" way of doing things are.
What I _am_ using is TimeLib, from which I derived my own slightly modified version, and it has been working without issues for months now.
Here are my thoughts from a top-level point of view, which completely disregard the current state of this implementation. To be specific, that means that I haven't looked at this implementation (apologies), and this is how _I_ would approach implementing the whole thing:
DST switches over on specific dates, and these dates are different for different countries. Not only that, but some countries change the official dates every so often (example in point: my own country, just ask any IT here how much they love the government). This seriously complicates trying to implement automatic changeover of DST to give correct time. As a result, most embedded devices don't implement this.
Implementing the time zone strings with #defines of const char strings will increase mem usage. I would suggest not showing that as an example. Or to anyone. The strings should always be in flash.
I consider the whole string and offset thing to be complex enough that I would implement a table of some kind that can be queried (what is the time offset for this string? what is the string for this time offset? what is the time offset for this index? etc). Personally, I would put the whole thing in a file on SPIFFS like a nano database.
Foremost, I am thinking of the memory reduction project, so I would like to reduce keeping any strings in memory where at all possible. If a time string is needed, then build it on the fly when requested, and return it, but don't keep any pieces in mem.
Do you want me to spend some time looking into the implementation, so that I can provide more detailed feedback? Maybe we can look into incorporating some of the TimeLib stuff here, making that lib superflous.
As we are in the "core", it should only provide 1a & 1b.
As you say, 1c - 4 are "next layer" but - fortunately - allready provided within newlib, so even more "arduinish" --> user don't have to find the "most compatible" TZlib, the core provides full standards compatible functions; so why hide it?
@5chufti our core can provide 1a and 1b in favor of arduino compatibility, but we can also provide 1c as our own lib, similar to our other ESP8266* libs.
Please don't confuse arduino compatibility with arduino style. We are striving for compatibility, so that apps based on arduino libs can be ported to the ESP easily. We are not striving for arduino style. In general terms, there are so many things wrong with the arduino usage design, that I sometimes seriously think about taking a trip to their HQ just to yell at their devs and slap them in the face.
Robin: "hey I just implemented this new global singleton..."
Batman: "NO!" Slap!
@d-a-v I'll try to take a look at the current time/ntp code, contrast against the arduino docs and against what I'm using, and provide feedback at that point. I'll try to come up with a way to get the best of all worlds. Please don't spend more time on this until then.
In the meantime, I could use your help elsewhere, to move towards 2.4 final!
I will not enter into details. What I proposed was quite simple and, I believe, coherent.
configTime(tz, dst, ntpsservers)tzset taking into acount tz, dstvoid configTime (const char* tz, const char* ntp_server)
{
setenv("TZ", tz, 1/*overwrite*/);
tzset();
configTime(0, 0, ntp_server);
}
with some defines which are not stored into ram/flash like the TZ_*country above, only one would be used in sketches (maybe they are already defined somewhere, I did not check). Maybe they would have to change from time to time because of specific politics per country, but they remain only defines. They can be living in their separate .h file, PRs can change them if needed, they are just harmless helpers.
We would then have one arduinish simple line per sketch that would configure in the most simple way accurate local time including DST for those who would like it, with no harm for others.
That's all what I suggested :)
(I was not aware of newlib and these powerful tzset settings before this thread, I'm glad we have it)
I need to know, for low powered sketches wanting to keep accurate NTP time, if the sntp_force_request() proposed above, along with a settimeofday() callback (example) can be useful to go back to low-power mode as soon as correct time is set and logs stored.
Hi,
thanks again for your interest. As you will see from following data, these functions are very helpfull for energy efficiency. The tests were performed with following sketch and either LwIP V2 or precompiled_gcc (1.4?)
#define VER 1 // old=0 new=1
#define PER 1 // persistent n=0 y=1
#include <ESP8266WiFi.h>
#include <time.h>
#if VER
#include <coredecls.h> // settimeofday_cb() / sntp_force_request()
#endif
// to get sntp
extern "C"
{
#include "sntp.h"
}
#define SSID "open"
#define WPWD ""
IPAddress ip(192, 168, 22, 5);
IPAddress gw(192, 168, 22, 123);
IPAddress nm(255, 255, 255, 0);
IPAddress mc(255, 255, 255, 255);
time_t t = 0;
void time_is_set (void)
{
time(&t);
Serial.println("got time (@" + String(millis()) + "ms) " + ctime(&t));
ESP.deepSleep(15000000);
delay(10);
}
void setup() {
Serial.begin(74880);
Serial.println("setup(): " + String(millis()));
#if VER
settimeofday_cb(time_is_set);
configTime(0, 0, "at.pool.ntp.org");
#else
sntp_stop();
sntp_setservername(0, "at.pool.ntp.org");
sntp_setservername(1, "de.pool.ntp.org");
sntp_set_timezone(0);
sntp_init();
#endif
WiFi.persistent(PER);
WiFi.mode(WIFI_STA);
WiFi.config(ip, gw, nm, gw, gw);
WiFi.begin(SSID, WPWD);
// WiFi.begin();
while (WiFi.status() != WL_CONNECTED) delay(1);
Serial.println("WiFi connected: " + String(millis()));
#if !PER
ESP.eraseConfig();
#endif
#if VER
sntp_force_request();
}
#else
while (t == 0) {
t = sntp_get_current_timestamp();
delay(1);
}
Serial.println("got time (@" + String(millis()) + "ms) " + ctime(&t));
Serial.println(ctime(&t));
ESP.deepSleep(15000000);
delay(10);
}
#endif
void loop() {
yield();
}
The results show that with LwIP V2 time until first valid timestamp was considerably longer than with old version.
#################### LwIP2, persisted WiFi conn. #############
setup(): 197
WiFi connected: 343
got time: 6414
setup(): 206
WiFi connected: 348
got time: 6441
setup(): 203
WiFi connected: 357
got time: 6449
setup(): 194
WiFi connected: 339
got time: 6365
#################### LwIP2, w/o persisted WiFi conn. ##############
setup(): 303
WiFi connected: 3344
got time: 7518
setup(): 307
WiFi connected: 3349
got time: 7369
setup(): 302
WiFi connected: 3351
got time: 7465
setup(): 305
WiFi connected: 3349
got time: 7491
#################### LwIP, persisted WiFi conn. ###############
setup(): 191
WiFi connected: 338
got time: 1123
setup(): 209
WiFi connected: 351
got time: 1055
setup(): 197
WiFi connected: 341
got time: 1118
setup(): 193
WiFi connected: 338
got time: 1137
#################### LwIP, w/o persisted WiFi conn. ###############
setup(): 307
WiFi connected: 3348
got time: 4568
setup(): 293
WiFi connected: 3334
got time: 4346
setup(): 306
WiFi connected: 3345
got time: 4463
setup(): 301
WiFi connected: 3377
got time: 4397
Using "sntp_force_request()" this is now considerably shortened, especially with persistent WiFi connection !!!
#################### LwIP2, force, persisted WiFi conn. #############
setup(): 206
WiFi connected: 351
got time (@397ms) Sat Nov 25 19:57:36 2017
setup(): 197
WiFi connected: 343
got time (@379ms) Sat Nov 25 19:57:51 2017
setup(): 199
WiFi connected: 347
got time (@369ms) Sat Nov 25 19:58:06 2017
setup(): 203
WiFi connected: 352
got time (@398ms) Sat Nov 25 19:58:21 2017
#################### LwIP2, force, w/o persisted WiFi conn. #############
setup(): 304
WiFi connected: 3348
got time (@3533ms) Sat Nov 25 19:53:08 2017
setup(): 306
WiFi connected: 3346
got time (@3537ms) Sat Nov 25 19:54:03 2017
setup(): 307
WiFi connected: 3345
got time (@3535ms) Sat Nov 25 19:54:21 2017
setup(): 314
WiFi connected: 3355
got time (@3538ms) Sat Nov 25 19:54:39 2017
so, again, thank you for your effort
Thanks for testing !
I'll try to have a look into lwip2's sntp init(boot time)/reinit(dhcp) to get a faster ntp update once wifi link is up.
For coherency, lwip2 should also be configured to allow 3 ntp servers like lwip1.4 (instead of one).
Do you think is is easy to take into account settimeofday()'s Usec and integrate it in system time ?
@d-a-v , yes for sure. I believe what should happen is to remove setting s_bootTimeSet in settimeofday (that can go back to non-extern I think), and call a new function in time.c for setting the s_bootTime_us. So something like this:
void setBootTime(uint64_t now_us)
{
s_bootTime_us = now_us - micros64();
s_bootTimeSet = true;
}
ensureBootTimeIsSet then becomes (to stay DRY)
static void ensureBootTimeIsSet()
{
....
if (now_s)
{
setBootTime(now_s * 1000000ULL);
}
}
}
then over in settimeofday
if (tv) /* after*/
{
sntp_set_system_time(tv->tv_sec);
// reset time subsystem
setBootTime(tv->tv_sec * 1000000ULL | tv->tv_usec);
}
Another thing I will throw out there, is that I think "boot time" should be renamed. It is really just an offset from micros64 that gets you the current REALTIME.
...
Another thing I will throw out there, is that I think "boot time" should be renamed. It is really just an offset from micros64 that gets you the current REALTIME.
...
:) my words ...
@BrandonLWhite @5chufti can you please check #4001 ?
as soon as I'm back from holidays...
Most helpful comment
thank you,
I'm working my way through the "time&TZ" functions of newlib at the moment; looks good so far. If I'm using static IP and disable WiFi for energy efficiency there is no way to force "sync" during a WiFi enabled periode?
I did see the example but missed something more enduser oriented like the following on how to get a "real" time with "onboard" features.