Docker events init error

docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)                                              
   Active: active (running) since Fri 2020-05-15 15:43:17 SAST; 2min 24s ago                                                         
     Docs: https://docs.docker.com                                                                                                   
 Main PID: 26981 (dockerd)                                                                                                           
    Tasks: 29                                                                                                                        
   Memory: 101.8M                                                                                                                    
      CPU: 3.806s                                                                                                                    
   CGroup: /system.slice/docker.service                                                                                              
           ├─26981 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock                                            
           └─27289 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.17.0.2 -container-port 80       

...
                                                                                                                                                                                          
May 15 15:43:17 skp-mint systemd[1]: Started Docker Application Container Engine.

«docker events» not showing service events #101

Expected behavior

docker events should show events regarding the creation of services.

Actual behavior

docker events doesn’t show events regarding the creation of services.

Steps to reproduce the behavior

I’m using a cluster of three manager nodes and use docker events on each of them. I then create a service like $ docker service create —name test nginx

I only get the following events:

Output of docker version :

Output of docker info :

Additional environment details (AWS, VirtualBox, physical, etc.)
Two of the nodes are Ubuntu 14.04, the third one is Ubuntu 16.04.. All of them are on an internal network.

The text was updated successfully, but these errors were encountered:

I’m not able to reproduce this. See the steps I used to try and reproduce:

On manager 1, start the docker events :

On manager 2, create, update, and remove a service:

Back on manager1, see the events that were received:

(note that container create/destroy events are not shown, because the tasks did not end up on the manager 1 node)

Are you running the docker events command on a manager? If the node you’re running the command on is not a manager, you’d only see container events, not swarm events.

Yes, docker events is running on a manager. However, I still don’t see those events when following your steps. As I can see the service when running docker service ls on the manager where docker events ran I guess this not caused by a communication error between the nodes.

Strange thing: On a second cluster (consisting of only Ubuntu 16.04) it just works. I then rebooted all of the nodes in the «buggy» cluster, after that it worked. Considering that rebooting «fixes» the other problem I sometimes have (see moby/moby#33430) I’m just accepting this as a little quirk and try to reboot those nodes regularly.

Same problem here.

After some time, our Swarm cluster (3x management nodes on 17.09) stopped generating «type=service» events. I have confirmed it by both docker events -f type=service and Python docker client. I haven’t tried yet if reboot will fix the problem.

Same issue here. For me, not even container events showed up.

I could not restart the nodes, as I’m using docker4aws, but slowly removing one node after the other and waiting for docker4aws to recreate them, the swarm propagated all service events properly in the end.

I have 3x managers running 17.12.1-ce. I wrote a microservice which pings slack with service updates, but it will go silent if the manager stops generating these events. I’ve just noticed the events stopped occurring on one of the managers, so I did a docker service update —force to move it to another manager, which seems to be working just fine. You can see the info for the managers below. Unfortunately, I can’t restart these nodes with zero downtime because live restore isn’t enabled. Maybe I’ll do it at some point later, perhaps enabling live restore.

Источник

docker system events

Get real time events from the server

Usage

Refer to the options section for an overview of available OPTIONS for this command.

Description

Use docker system events to get real-time events from the server. These events differ per Docker object type.

Object types

Containers

Docker containers report the following events:

• attach
• commit
• copy
• create
• destroy
• detach
• die
• exec_create
• exec_detach
• exec_die
• exec_start
• export
• health_status
• kill
• oom
• pause
• rename
• resize
• restart
• start
• stop
• top
• unpause
• update

Images

Docker images report the following events:

Plugins

Docker plugins report the following events:

Volumes

Docker volumes report the following events:

Networks

Docker networks report the following events:

Daemons

Docker daemons report the following events:

Limiting, filtering, and formatting the output

Limit events by time

The —since and —until parameters can be Unix timestamps, date formatted timestamps, or Go duration strings (e.g. 10m , 1h30m ) computed relative to the client machine’s time. If you do not provide the —since option, the command returns only new and/or live events. Supported formats for date formatted time stamps include RFC3339Nano, RFC3339, 2006-01-02T15:04:05 , 2006-01-02T15:04:05.999999999 , 2006-01-02Z07:00 , and 2006-01-02 . The local timezone on the client will be used if you do not provide either a Z or a +-00:00 timezone offset at the end of the timestamp. When providing Unix timestamps enter seconds[.nanoseconds], where seconds is the number of seconds that have elapsed since January 1, 1970 (midnight UTC/GMT), not counting leap seconds (aka Unix epoch or Unix time), and the optional .nanoseconds field is a fraction of a second no more than nine digits long.

Filtering (—filter)

The filtering flag ( -f or —filter ) format is of “key=value”. If you would like to use multiple filters, pass multiple flags (e.g., —filter «foo=bar» —filter «bif=baz» )

Using the same filter multiple times will be handled as a OR; for example —filter container=588a23dac085 —filter container=a8f7720b8c22 will display events for container 588a23dac085 OR container a8f7720b8c22

Using multiple filters will be handled as a AND; for example —filter container=588a23dac085 —filter event=start will display events for container container 588a23dac085 AND the event type is start

The currently supported filters are:

• container ( container= )
• daemon ( daemon= )
• event ( event= )
• image ( image= )
• label ( label= or label= = )
• network ( network= )
• plugin ( plugin= )
• type ( type= )
• volume ( volume= )

For example uses of this command, refer to the examples section below.

Options

Examples

Basic example

You’ll need two shells for this example.

Shell 1: Listening for events:

Shell 2: Start and Stop containers:

Shell 1: (Again .. now showing events):

To exit the docker system events command, use CTRL+C .

Filter events by time

You can filter the output by an absolute timestamp or relative time on the host machine, using the following different time syntaxes:

Filter events by criteria

The following commands show several different ways to filter the docker event output.

Format the output (—format)

If a format ( —format ) is specified, the given template will be executed instead of the default format. Go’s text/template package describes all the details of the format.

Источник

docker events

Get real time events from the server

Usage

Refer to the options section for an overview of available OPTIONS for this command.

Description

Use docker events to get real-time events from the server. These events differ per Docker object type. Different event types have different scopes. Local scoped events are only seen on the node they take place on, and swarm scoped events are seen on all managers.

Only the last 1000 log events are returned. You can use filters to further limit the number of events returned.

Object types

Containers

Docker containers report the following events:

• attach
• commit
• copy
• create
• destroy
• detach
• die
• exec_create
• exec_detach
• exec_die
• exec_start
• export
• health_status
• kill
• oom
• pause
• rename
• resize
• restart
• start
• stop
• top
• unpause
• update

Images

Docker images report the following events:

Plugins

Docker plugins report the following events:

Volumes

Docker volumes report the following events:

Networks

Docker networks report the following events:

Daemons

Docker daemons report the following events:

Services

Docker services report the following events:

Nodes

Docker nodes report the following events:

Secrets

Docker secrets report the following events:

Configs

Docker configs report the following events:

Limiting, filtering, and formatting the output

Limit events by time (—since, —until)

The —since and —until parameters can be Unix timestamps, date formatted timestamps, or Go duration strings (e.g. 10m , 1h30m ) computed relative to the client machine’s time. If you do not provide the —since option, the command returns only new and/or live events. Supported formats for date formatted time stamps include RFC3339Nano, RFC3339, 2006-01-02T15:04:05 , 2006-01-02T15:04:05.999999999 , 2006-01-02Z07:00 , and 2006-01-02 . The local timezone on the client will be used if you do not provide either a Z or a +-00:00 timezone offset at the end of the timestamp. When providing Unix timestamps enter seconds[.nanoseconds], where seconds is the number of seconds that have elapsed since January 1, 1970 (midnight UTC/GMT), not counting leap seconds (aka Unix epoch or Unix time), and the optional .nanoseconds field is a fraction of a second no more than nine digits long.

Only the last 1000 log events are returned. You can use filters to further limit the number of events returned.

Filtering (—filter)

The filtering flag ( -f or —filter ) format is of “key=value”. If you would like to use multiple filters, pass multiple flags (e.g., —filter «foo=bar» —filter «bif=baz» )

Using the same filter multiple times will be handled as a OR; for example —filter container=588a23dac085 —filter container=a8f7720b8c22 will display events for container 588a23dac085 OR container a8f7720b8c22

Using multiple filters will be handled as a AND; for example —filter container=588a23dac085 —filter event=start will display events for container container 588a23dac085 AND the event type is start

The currently supported filters are:

• config ( config= )
• container ( container= )
• daemon ( daemon= )
• event ( event= )
• image ( image= )
• label ( label= or label= = )
• network ( network= )
• node ( node= )
• plugin ( plugin= )
• scope ( scope= )
• secret ( secret= )
• service ( service= )
• type ( type= )
• volume ( volume= )

Format the output (—format)

If a format ( —format ) is specified, the given template will be executed instead of the default format. Go’s text/template package describes all the details of the format.

If a format is set to <> , the events are streamed as valid JSON Lines. For information about JSON Lines, please refer to https://jsonlines.org/.

For example uses of this command, refer to the examples section below.

Options

Examples

Basic example

You’ll need two shells for this example.

Shell 1: Listening for events:

Shell 2: Start and Stop containers:

Shell 1: (Again .. now showing events):

To exit the docker events command, use CTRL+C .

Filter events by time

You can filter the output by an absolute timestamp or relative time on the host machine, using the following different time syntaxes:

Filter events by criteria

The following commands show several different ways to filter the docker event output.

Источник

Docker OOM event handling inappropriate #670

Expected behavior

Not logging things blindly, be more cautious about outputing.

Blindly logging everything can pose a severe denial of service risk by exhausting system resources. Suggest aggregating similar log messages in some way to handle (especially) OOM event more appropriately.

Actual behavior

containerd continues to bubble up OOM events and dockerd blindly log such events.

For us this is a huge issue because the consequence has been dockerd generating a continuous rate of 10Mbps of log output observed on our systems and eventually exhausted all 80GB of system RAM per node + all disk spaces if not caught in time by us, which leads to a denial of service.

We limit our user’s resources but we have no control over what they run, but once OOM occurs by resource limits, dockerd is at fault for consuming the rest and majority of system resources by logging recurring OOM events.

Steps to reproduce the behavior

Make a container with resource limits (we did this through RKE Kubernetes) and continually allocate memory till OOM triggers.

An actual example we caught was setting a container memory limit at 4G and load a 5G csv file using read.csv() from R language. And it has caused dockerd to continually generate a massive amount of logs eventually jammed our system.

Output of docker version :

Client:
Version: 18.09.0
API version: 1.39
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:48:22 2018
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine — Community
Engine:
Version: 18.09.4
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: d14af54
Built: Wed Mar 27 18:04:46 2019
OS/Arch: linux/amd64
Experimental: false

Output of docker info :

Containers: 42
Running: 23
Paused: 0
Stopped: 19
Images: 61
Server Version: 18.09.4
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data Space Used: 112.2GB
Data Space Total: 765GB
Data Space Available: 652.9GB
Metadata Space Used: 70.76MB
Metadata Space Total: 8.049GB
Metadata Space Available: 7.978GB
Thin Pool Minimum Free Space: 76.5GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.149-RHEL7 (2018-07-20)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 5.0.7-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 18
Total Memory: 78.66GiB
Name: k-prod-cpu-6.dsa.lan
ID: S65O:V6T3:PQ2L:KKKW:PSC6:DTDJ:6ITH:KICE:EFWX:DQT3:NBSP:XSND
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: bridge-nf-call-ip6tables is disabled
WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.

Additional environment details (AWS, VirtualBox, physical, etc.)

The text was updated successfully, but these errors were encountered:

Источник

Docker Engine регистрирует событие каждый раз, когда демон выполняет какие-либо значительные действия.

Вы можете обратиться к журналу событий, чтобы определить, когда произошло то или иное действие, и отследить изменения объектов с течением времени.

В этой статье мы объясним, что фиксируется как события и как их просмотреть.

Затем мы покажем, как отслеживать события в режиме реального времени с помощью Docker CLI и REST API.

Docker Engine регистрирует событие каждый раз, когда демон выполняет значительные действия.

Вы можете обратиться к журналу событий, чтобы определить, когда произошло то или иное действие, и отследить изменения объектов с течением времени.

В этой статье мы объясним, что фиксируется как события и когда вы можете захотеть их просмотреть.

Затем мы покажем, как отслеживать события в режиме реального времени с помощью Docker CLI и REST API.

Что такое события Docker?

События Docker описывают действия, выполняемые демоном Docker.

Большинство взаимодействий с такими объектами, как контейнеры, образы, тома и сети, записывают событие, создавая журнал, который можно использовать для анализа прошлых изменений.

Существует множество различных видов событий, которые определяют конкретные изменения в вашей среде:

• Создание и удаление контейнеров
• Статусы проверки работоспособности контейнеров
• Команды, выполняемые внутри контейнеров с помощью docker exec
• Пуш и пул образов
• Создание, уничтожение, монтирование и размонтирование томов
• Включение и отключение плагинов демонов Docker.
• Полный список можно посмотреть в документации Docker.

Каждое записанное событие содержит метку времени и идентификатор затронутого объекта.

Вы можете использовать эту информацию для сбора истории изменений в вашей среде, независимо от того, наблюдали ли вы их первоначальные триггеры.

Сохраненные события также могут помочь в диагностике таких проблем, как неожиданные сбои контейнеров.

Просмотр лога позволяет определить точное время остановки контейнера, предоставляя дата поинт, который можно соотнести с другими журналами.

События могут установить, когда проверки контейнера начали давать сбои, сужая период, когда вам нужно проверить внешние службы, чтобы определить основную причину проблемы.

Передача событий Docker с помощью Docker CLI

Команда docker events CLI транслирует события от вашего демона Docker в окно терминала.

События будут появляться в реальном времени до тех пор, пока вы не завершите процесс нажатием комбинации клавиш Ctrl+C.

Выполнение команды без аргументов не покажет никаких результатов.

Отображается только новая активность, поэтому вывод остается пустым до тех пор, пока не произойдет событие.

Вы можете спровоцировать его, запустив новый контейнер в другой оболочке:

docker run --rm hello-world
   

Теперь в окне терминала, в котором запущена команда docker events, должно появиться несколько событий:

2022-05-31T15:20:00.267970018+01:00 image pull hello-world:latest (name=hello-world)
2022-05-31T15:20:00.347054862+01:00 container create 4a6c8d34a183363db5dbfdcc3cab4c82c4a341d719df56ec2e7f879ee8f02378 (image=hello-world, name=nifty_morse)
2022-05-31T15:20:00.347805277+01:00 container attach 4a6c8d34a183363db5dbfdcc3cab4c82c4a341d719df56ec2e7f879ee8f02378 (image=hello-world, name=nifty_morse)
2022-05-31T15:20:00.621070053+01:00 container start 4a6c8d34a183363db5dbfdcc3cab4c82c4a341d719df56ec2e7f879ee8f02378 (image=hello-world, name=nifty_morse)
...
   

Каждое событие отображается в отдельной строке.

Сначала отображается временная метка события, затем тип затронутого объекта (например, образ или контейнер), а затем действие, которое было выполнено (например, создание, присоединение и запуск).

Оставшаяся часть сообщения содержит полезные метаданные об объекте.

Приведенный выше пример показывает, что был извлечен образ hello-world:latest и на его основе создан контейнер.

Форматирование вывода

Необработанный список событий часто бывает громоздким.

Вы можете изменить формат вывода с помощью флага –format, который принимает строку шаблона Go:

docker events --format '{{ .Time }} {{ .Action }} {{ .Type}} {{ .ID }}'
   

1654006800 pull image hello-world:latest
1654006800 create container 4a6c8d34a183363db5dbfdcc3cab4c82c4a341d719df56ec2e7f879ee8f02378
1654006800 attach container 4a6c8d34a183363db5dbfdcc3cab4c82c4a341d719df56ec2e7f879ee8f02378
1654006800 start container 4a6c8d34a183363db5dbfdcc3cab4c82c4a341d719df56ec2e7f879ee8f02378
   

docker events --format '{{ json . }}' | jq
{
"status": "create",
"id": "4a6c8d34a183363db5dbfdcc3cab4c82c4a341d719df56ec2e7f879ee8f02378",
"from": "hello-world",
"Type": "container",
"Action": "create",
"Actor": {
"ID": "4a6c8d34a183363db5dbfdcc3cab4c82c4a341d719df56ec2e7f879ee8f02378",
"Attributes": {
"image": "hello-world",
"name": "nifty_morse"
}
},
"scope": "local",
"time": 1654006800,
"timeNano": 1654006800347054800
}
   

docker events 
--filter container=app-container
--filter container=api-container
--filter event=create
   

docker events --since 1h
docker events --since '2021-05-01T16:00:00'
   

curl http://127.0.0.1:2375/v1.41/events
{
"Type": "container",
"Action": "create",
"Actor": {
"ID": "4a6c8d34a183363db5dbfdcc3cab4c82c4a341d719df56ec2e7f879ee8f02378",
"Attributes": {
"image": "hello-world",
"name": "nifty_morse"
}
},
"scope": "local",
"time": 1654006800,
"timeNano": 1654006800347054800
   

curl http://127.0.0.1:2375/v1.41/events?since=1h&filters={'type':'container','action':'create'}
   

!/bin/bash
docker events --format '{{ .Time }} {{ .Action }} {{ .Type }} {{ .ID }}' | while read event
do
curl 
-X POST 
-H "Content-Type: application/json" 
-d '{"event": "$event"}' 
https://example.com/events
done
   

[Unit]
Description=Custom Docker Event Monitoring Service

[Service]
Type=forking
ExecStart=/usr/local/bin/docker-events.sh

[Install]
WantedBy=multi-user.target

$ sudo systemctl daemon-reload
$ sudo systemctl start docker-events
$ sudo systemctl enable docker-events

I was able to build a multiarch image successfully from an M1 Macbook which is arm64.
Here’s my docker file and trying to run from a raspberrypi aarch64/arm64 and I am getting this error when running the image: standard_init_linux.go:228: exec user process caused: exec format error

Editing the post with the python file as well:

FROM frolvlad/alpine-python3

RUN pip3 install docker
RUN mkdir /hoster
WORKDIR /hoster
ADD hoster.py /hoster/

CMD ["python3", "-u", "hoster.py"]




#!/usr/bin/python3
import docker
import argparse
import shutil
import signal
import time
import sys
import os

label_name = "hoster.domains"
enclosing_pattern = "#-----------Docker-Hoster-Domains----------n"
hosts_path = "/tmp/hosts"
hosts = {}

def signal_handler(signal, frame):
    global hosts
    hosts = {}
    update_hosts_file()
    sys.exit(0)

def main():
    # register the exit signals
    signal.signal(signal.SIGINT, signal_handler)
    signal.signal(signal.SIGTERM, signal_handler)

    args = parse_args()
    global hosts_path
    hosts_path = args.file

    dockerClient = docker.APIClient(base_url='unix://%s' % args.socket)
    events = dockerClient.events(decode=True)
    #get running containers
    for c in dockerClient.containers(quiet=True, all=False):
        container_id = c["Id"]
        container = get_container_data(dockerClient, container_id)
        hosts[container_id] = container

    update_hosts_file()

    #listen for events to keep the hosts file updated
    for e in events:
        if e["Type"]!="container": 
            continue
        
        status = e["status"]
        if status =="start":
            container_id = e["id"]
            container = get_container_data(dockerClient, container_id)
            hosts[container_id] = container
            update_hosts_file()

        if status=="stop" or status=="die" or status=="destroy":
            container_id = e["id"]
            if container_id in hosts:
                hosts.pop(container_id)
                update_hosts_file()


def get_container_data(dockerClient, container_id):
    #extract all the info with the docker api
    info = dockerClient.inspect_container(container_id)
    container_hostname = info["Config"]["Hostname"]
    container_name = info["Name"].strip("/")
    container_ip = info["NetworkSettings"]["IPAddress"]
    if info["Config"]["Domainname"]:
        container_hostname = container_hostname + "." + info["Config"]["Domainname"]
    
    result = []

    for values in info["NetworkSettings"]["Networks"].values():
        
        if not values["Aliases"]: 
            continue

        result.append({
                "ip": values["IPAddress"] , 
                "name": container_name,
                "domains": set(values["Aliases"] + [container_name, container_hostname])
            })

    if container_ip:
        result.append({"ip": container_ip, "name": container_name, "domains": [container_name, container_hostname ]})

    return result


def update_hosts_file():
    if len(hosts)==0:
        print("Removing all hosts before exit...")
    else:
        print("Updating hosts file with:")

    for id,addresses in hosts.items():
        for addr in addresses:
            print("ip: %s domains: %s" % (addr["ip"], addr["domains"]))

    #read all the lines of thge original file
    lines = []
    with open(hosts_path,"r+") as hosts_file:
        lines = hosts_file.readlines()

    #remove all the lines after the known pattern
    for i,line in enumerate(lines):
        if line==enclosing_pattern:
            lines = lines[:i]
            break;

    #remove all the trailing newlines on the line list
    if lines:
        while lines[-1].strip()=="": lines.pop()

    #append all the domain lines
    if len(hosts)>0:
        lines.append("nn"+enclosing_pattern)
        
        for id, addresses in hosts.items():
            for addr in addresses:
                lines.append("%s    %sn"%(addr["ip"],"   ".join(addr["domains"])))
        
        lines.append("#-----Do-not-add-hosts-after-this-line-----nn")

    #write it on the auxiliar file
    aux_file_path = hosts_path+".aux"
    with open(aux_file_path,"w") as aux_hosts:
        aux_hosts.writelines(lines)

    #replace etc/hosts with aux file, making it atomic
    shutil.move(aux_file_path, hosts_path)


def parse_args():
    parser = argparse.ArgumentParser(description='Synchronize running docker container IPs with host /etc/hosts file.')
    parser.add_argument('socket', type=str, nargs="?", default="tmp/docker.sock", help='The docker socket to listen for docker events.')
    parser.add_argument('file', type=str, nargs="?", default="/tmp/hosts", help='The /etc/hosts file to sync the containers with.')
    return parser.parse_args()

if __name__ == '__main__':
    main()