View previous topic :: View next topic | |||||||||||||
Author | Message | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
pbielen Guest |
|
||||||||||||
Back to top | |||||||||||||
DarrenM l33t Joined: 25 Apr 2002 |
|
||||||||||||
Back to top |
|
||||||||||||
pbielen Guest |
|
||||||||||||
Back to top | |||||||||||||
richyp n00b Joined: 24 Apr 2002 |
|
||||||||||||
Back to top |
|
||||||||||||
dufnutz Apprentice Joined: 01 May 2002 |
|
||||||||||||
Back to top |
|
||||||||||||
Guest |
|
||||||||||||
Back to top | |||||||||||||
R0B_IX Tux’s lil’ helper Joined: 15 Jun 2003 |
|
||||||||||||
Back to top |
|
||||||||||||
Regor Guru Joined: 06 Aug 2002 |
|
||||||||||||
Back to top |
|
||||||||||||
charlesnadeau Apprentice Joined: 19 May 2003 |
|
||||||||||||
Back to top |
|
||||||||||||
devsk Advocate Joined: 24 Oct 2003 |
|
||||||||||||
Back to top |
|
||||||||||||
Celtis l33t Joined: 05 Jul 2003 |
|
||||||||||||
Back to top |
|
||||||||||||
devsk Advocate Joined: 24 Oct 2003 |
|
||||||||||||
Back to top |
|
||||||||||||
Regor Guru Joined: 06 Aug 2002 |
|
||||||||||||
Back to top |
|
||||||||||||
devsk Advocate Joined: 24 Oct 2003 |
|
||||||||||||
Back to top |
|
||||||||||||
Celtis l33t Joined: 05 Jul 2003 |
|
||||||||||||
Back to top |
|
||||||||||||
toliman n00b Joined: 08 Aug 2002 |
|
||||||||||||
Back to top |
|
||||||||||||
zOOz n00b Joined: 24 Sep 2003 |
|
||||||||||||
Back to top |
|
||||||||||||
devsk Advocate Joined: 24 Oct 2003 |
|
||||||||||||
Back to top |
|
||||||||||||
nadamsieee Guru Joined: 30 May 2003 |
|
||||||||||||
Back to top |
|
||||||||||||
Regor Guru Joined: 06 Aug 2002 |
|
||||||||||||
Back to top |
|
||||||||||||
nadamsieee Guru Joined: 30 May 2003 |
|
||||||||||||
Back to top |
|
||||||||||||
Spiralis n00b Joined: 22 Jun 2003 |
|
||||||||||||
Back to top |
|
||||||||||||
TecHunter Tux’s lil’ helper Joined: 15 Feb 2003 |
|
||||||||||||
Back to top |
|
||||||||||||
moderately-anonymized n00b Joined: 08 Nov 2005 |
|
||||||||||||
Back to top |
|
||||||||||||
tgh Apprentice Joined: 05 Oct 2005 |
|
||||||||||||
Back to top |
|
||||||||||||
|
You cannot post new topics in this forum |
Version
Microsoft Windows [Version 10.0.19042.1348]
WSL Version
- WSL 2
- WSL 1
Kernel Version
5.10.102.1
Distro Version
Ubuntu 20.04.4 LTS
Other Software
No response
Repro Steps
When building a kernel I would often get multiple clock skew errors as has been discussed in prior bugs such as #4975 when performing a:
sudo make -j 8
During compile my Vmmem and COM Surrogate processes are taking up close to 100% of my CPU cycles.
This is on a i7-3820QM:
bengalih@MOBILEONE:/mnt/c/Users/bengalih$ getconf _NPROCESSORS_ONLN
8
As my system date/time appears normal when starting the build process, I theorized that the heavy hit to the CPU may have been contributing to a small clock drift during compile time. I tested this theory by running with:
sudo make -j 6
I performed at least 3 builds each using 6 and 8 respectively and without fail the clock skew error appeared when using the full 8 core, but never when I ran with 6.
(EDIT: On last build with «6» I did get one skew error when I also opened a zoom meeting on my device during build. This was an anomaly as on all other 6 builds I did not get any skew and on the 8 runs I would get multiple per run).
While this is not a «bug» per-se, many people seeing this issue at build time may be in a similar situation where they are using slightly older processors. However, it may be considered a bug that the kernel scheduler is not reserving cycles for keeping time in sync (as there have been reported other clock skew issues outside of just build).
I wanted to give this as a PSA for others having the issue, and maybe a more official note can be made somewhere, perhaps in the build instructions:
https://github.com/microsoft/WSL2-Linux-Kernel
Expected Behavior
clock can remain in sync even when running highly CPU intensive tasks.
Actual Behavior
clock appears to lose time when running highly CPU intensive tasks.
Diagnostic Logs
make[3]: warning: Clock skew detected. Your build may be incomplete.
1. Overview
While building an executable using make, we may sometimes get a “Clock skew detected” warning.
In this tutorial, we’ll see an example of this warning. Then, we’ll discuss how to solve it.
2. Examination of the Problem
We’ll examine the problem using an example.
2.1. Code Example
We’ll use the C program, hello_world.c, to analyze the problem:
#include <stdio.h>
int main()
{
printf("Hello Worldn");
return 0;
}
The program just prints Hello World and exits.
We’ll use the following Makefile to build the program:
.PHONY: all
all: hello_world
hello_world: hello_world.c
@gcc hello_world –o hello_world
.PHONY: clean
clean:
@rm –f hello_world
Now, let’s build the program using make:
$ pwd
/home/alice/hello_world_project
$ ls
hello_world.c Makefile
$ make
$ ls
hello_world hello_world.c Makefile
We built the executable, hello_world, successfully. Let’s run the program:
$ ./hello_world
Hello World
The program runs as expected.
2.2. Creation of the Problem
Now, we’ll try to obtain the “Clock skew detected” warning. However, let’s first check the modification time of hello_world:
$ ls –l hello_world
-rwxr-xr-x 1 alice alice 17536 Jan 17 08:02 hello_world
$ date
Tue Jan 17 08:02:43 +03 2023
We’ll change the modification time of hello_world using the touch command:
$ touch –t 202401010000 hello_world
$ ls –l hello_world
-rwxr-xr-x 1 alice alice 17536 Jan 1 2024 hello_world
The -t option of touch sets the modification time of a file to the specified time. We set the modification time of hello_world to a time in the future.
Let’s also update hello_world.c:
#include <stdio.h>
int main()
{
printf("Hello Worldn");
printf("Hello World againn");
return 0;
}
We just added the printf(“Hello World againn”) statement.
Now, let’s build the program once more using make:
$ make
make: Warning: 'hello_world' has modification 30123485 s in the future
make: Nothing to be done for 'all'
make: warning: Clock skew detected. Your build may be incomplete.
We got the “Clock skew detected” warning.
2.3. Analysis of the Problem
While performing an incremental build, make first checks the modification times of source files. If the modification time of a source file is more recent than the previously built executable using the source file, make rebuilds the executable.
However, this is the opposite in our case. The modification time of the previously built executable, hello_world, is more recent than the modification time of hello_world.c. This is the reason for getting the warning.
Let’s run hello_world once more:
$ ./hello_world
Hello World
As the output of running hello_world shows, we couldn’t even build the updated code – we don’t see Hello World again in the output.
Therefore, when we have the “Clock skew detected” warning, the source code may not even be built. It may also cause unnecessary builds.
3. Solution
Making a clean build solves the problem since the clean rule in the Makefile removes the executable using rm –f hello_world:
$ make clean
$ make
$ ls –l hello_world
-rwxr-xr-x 1 alice alice 17536 Jan 17 08:13 hello_world
We cleaned the executable using make clean. Then, we rebuilt it using make. The modification time of the executable, hello_world, was updated as expected.
Clean build of a big project that has hundreds of source files may take a long time. Therefore, manually removing only the executable that causes the warning might be useful for having an incremental build.
However, the main reason for getting the warning is the time difference between machines. For example, we might be working in an NFS (Network File System) mounted directory, and the clocks on the NFS server and the client may not be synchronized. We tried to simulate this situation by changing modification time of the built executable. Therefore, the real solution of the problem is the time synchronization of the machines using a protocol like NTP (Network Time Protocol).
4. Conclusion
In this article, we discussed the “Clock skew detected” warning that we may come across while building an executable with make.
First, we saw an example to understand the problem. The reason for the problem was the more recent modification time of the executable than the source file’s modification time.
Then, we discussed how to avoid the problem. We saw that making a clean build solves it. It’s also possible to have an incremental build by just removing the executable with the modification time in the future.
Finally, we learned that the real solution is synchronizing the system times of the machines using a protocol like NTP.
If you have a few years of experience in the Linux ecosystem, and you’re interested in sharing that experience with the community, have a look at our Contribution Guidelines.
-
#1
Hello, on my test Ceph cluster (4 nodes ) this morning i have this Warning
«Monitor clock skew detected»
root@n1:~# ceph health detail
HEALTH_WARN clock skew detected on mon.1, mon.2; Monitor clock skew detected
mon.1 addr 10.10.10.2:6789/0 clock skew 0.085488s > max 0.05s (latency 0.00120293s)
mon.2 addr 10.10.10.3:6789/0 clock skew 0.0755085s > max 0.05s (latency 0.00137496s)
I have installed ntp ntpdate but still have this error..
Any suggestion ?
Thanks
-
#2
Hi,
have you also disable systemd-timesyncd.service
and do you use the same server at the ntpd?
-
#3
apt-get install ntp ntpdate and reboot fix the problem.
Thanks
-
#4
Hi,
have you also disable systemd-timesyncd.service
and do you use the same server at the ntpd?
Hello Wolfgang,
we also have ceph clock skew.
currently we use for all nodes:
1- timedatectl set-ntp true
2- same servers in timesyncd.conf
I was reading up on systemd ntp and instead want to run ntp the old way.
so I figure these are the steps:
disable systemd timedatectl
Code:
timedatectl set-ntp false
install and configure real ntp
at all pve nodes
configure all by using the same /etc/default/ntpdate , 1ST entry to local ntp server
set up ntp server . we’ll use pfsense to start.
am I missing something in that set up?
-
#5
Hi Rob
use ntpd.
ntpdate set the time and this makes problems.
What I mean is ceph do not like time jumps. ntpd is a daemon and do not set the time hard, it makes correction with accelerate the time or decelerate the time.
-
#6
Hi Rob
use ntpd.
ntpdate set the time and this makes problems.
What I mean is ceph do not like time jumps. ntpd is a daemon and do not set the time hard, it makes correction with accelerate the time or decelerate the time.
OK — How do I use ntpd ? Is that just the systemd set up using :
-
#7
First you have to install ntpd
then edit the /etc/ntp.conf
set systemd-timesyncd on ntpd as you wrote
restart ntpd
-
#8
OK then this is what we’ll do to try to fix clock skew warning in ceph.
that should just work to deal with time settings on a node and not server time to network
adjust /etc/ntp.conf if you want to — change/add servers or be a ntp server for the local network
—
-
#10
We are still getting ‘Monitor clock skew detected’ after ntp changes.
quick question —
we have mon on same network as osd .
our osd systems are not mons
is it a good practice to use the same network for osd and mon?
-
#11
Yes the for small setups it is ok.
The main propose of separating this 2 networks is the network load.
So if you network has no capacity problems it is ok.
-
#13
Meet time sync problem with PVE 5.4 strange, that ntp time sync not enabled by default.
-
#14
Every time I find problems with time sync, I use chrony
-
#15
Every time I find problems with time sync, I use chrony
+1 for chrony.
(I think it could be great to have it by default for proxmox6 instead timesyncd)
-
#16
No-matter what, but something should be enable by default, to sync time
-
#17
No-matter what, but something should be enable by default, to sync time
![]()
systemd-timesyncd.service is enable by default.
but it’s not precise like a true ntp server. (it’s basicaly something like a ntpdate running at some interval, but ceph really need low drift)
-
#18
Hello Spirit ,
after install chrony ,
what adjustments if any do you make to /etc/chrony/chrony.conf ?
so far we just adjust the ‘server’ line to attempt to use something local to our timezone.
Last edited: Apr 21, 2019
-
#19
Hello Spirit ,
after install chrony ,
what adjustments if any do you make to /etc/chrony/chrony.conf ?
so far we just adjust the ‘server’ line to attempt to use something local to our timezone.
I keep default settings and only change server to our internal ntp server.
60 / 60 / 8 Регистрация: 15.10.2010 Сообщений: 356 |
|
1 |
|
02.08.2012, 13:14. Показов 5734. Ответов 7
Привет. Код \ Компиляция первого модуля (file.h) make: Warning: File '../proj/file.h' has modification time 3209 s in the future \ компиляция остальных модулей make: warning: Clock skew detected. Your build may be incomplete. Программа компилируется, работает. Однако ошибка — напрягает. Полазил по гуглам — везде рекомендуют выставить верное системное время. Оно — верное Код [pavel@localhost ~]$ date ]Чт. авг. 2 13:12:17 MSK 2012 С горя даже компутер перезагрузил. Ось — fedora 17 Спасибо.
__________________
0 |
Почетный модератор 11295 / 4266 / 437 Регистрация: 12.06.2008 Сообщений: 12,282 |
|
02.08.2012, 13:28 |
2 |
Заново пересохрани этот файл, что бы у него дата изменения обновилась. Хотя, если он пишет, что дата файла на 3209 секунд новее, то менее чем через час должно пройти.
1 |
60 / 60 / 8 Регистрация: 15.10.2010 Сообщений: 356 |
|
02.08.2012, 13:31 [ТС] |
3 |
Пересохранил… Не помогло. А с чем связана то ошибка такого рода?
0 |
2732 / 1428 / 89 Регистрация: 08.09.2011 Сообщений: 3,746 Записей в блоге: 1 |
|
02.08.2012, 13:33 |
4 |
очисти проект, так же в насройках креатора, можно выставить что бы он подтягивал изменения файлов которые сделаны вне его и посмтри что и как
1 |
136 / 140 / 7 Регистрация: 22.08.2011 Сообщений: 485 |
|
02.08.2012, 13:38 |
5 |
Может комп в домене (проверь обновление с тайм сервера есть или нет),
1 |
60 / 60 / 8 Регистрация: 15.10.2010 Сообщений: 356 |
|
02.08.2012, 13:41 [ТС] |
6 |
очисти проект Без толку) Это первое, что я сделал.
удали файл и пересоздай его скопировав содержимое прежнего Не первое, что я сделал, однако так же бесполезно)
можно выставить что бы он подтягивал изменения файлов которые сделаны вне его Искал минут 7. Пока не нашел. По-позже свежим взглядом еще гляну =)
проверь обновление с тайм сервера есть или нет Да, все ок. Совпадение — секунда в секунду.
(проверь свой часовой пояс) И с этим тоже все ок…
0 |
Почетный модератор 11295 / 4266 / 437 Регистрация: 12.06.2008 Сообщений: 12,282 |
|
02.08.2012, 13:43 |
7 |
Пересохранил… Не помогло Измени там хоть один символ… а то когда просто сохраняешь без изменений, он может не сохранить.
И уменьшается с каждым билдом Оно просто со временем уменьшается.
А с чем связана то ошибка такого рода? Похоже, что по какой-то причине дата создания файла указана неверно. Т.е. если сейчас 13:37, то у твоего файла указано время 14:20 (например). Вот он и говорит, что такого не может быть… что файл не мог быть создан в будущем. Поэтому он и предупреждает, что на компьютере какие-то проблемы с часами. Из-за чего это произошло изначально не знаю… видимо, почему-то часы прыгнули вперёд, а потом вернулись обратно…. и пока они спешили, ты сохранил файл. Подозреваю, что это произошло во время синхронизации времени с интернетовским сервером. Может, они там часы перевели на час вперёд. Добавлено через 1 минуту
1 |
60 / 60 / 8 Регистрация: 15.10.2010 Сообщений: 356 |
|
02.08.2012, 14:25 [ТС] |
8 |
Файл я создал с нуля… Добавлено через 41 секунду
А может, эта цифра при сохранении файла опять увеличивается до 3600 ? Да нет =) Добавлено через 39 минут
0 |
Nick
Случайный прохожий
-
#1
Привет всем, в рамках одного HA кластера виртуальная машина почему то не хочет мигрировать. Виртуалка выключена. При валидации выдает такую ошибку.
Too large clock skew was detected. Relative time skew 3081510405 between source and destination hosts is greater than five minutes.
Время что ли разное ? Как так может быть ?
Последнее редактирование: 17.12.2021