Runtime error main thread is not in main loop

import threading
import time
import PySimpleGUI as sg

"""
    DESIGN PATTERN - Multithreaded Long Tasks GUI using shared global variables

    Presents one method for running long-running operations in a PySimpleGUI environment.
    The PySimpleGUI code, and thus the underlying GUI framework, runs as the primary, main thread
    The "long work" is contained in the thread that is being started. Communicating is done (carefully) using global variables

    There are 2 ways "progress" is being reported to the user. 
    You can simulate the 2 different scenarios that happen with worker threads.
    1.  If a the amount of time is known ahead of time or the work can be broken down into countable units, then a progress bar is used.  
    2.  If a task is one long chunk of time that cannot be broken down into smaller units, then an animated GIF is shown that spins as
    long as the task is running.
"""


def long_operation_thread(seconds, window):
    """
    A worker thread that communicates with the GUI through a global message variable
    This thread can block for as long as it wants and the GUI will not be affected
    :param seconds: (int) How long to sleep, the ultimate blocking call
    """
    progress = 0
    print('Thread started - will sleep for {} seconds'.format(seconds))
    for i in range(int(seconds * 10)):
        time.sleep(.1)  # sleep for a while
        progress += 100 / (seconds * 10)
        window.write_event_value('-PROGRESS-', progress)

    window.write_event_value('-THREAD-', '*** The thread says.... "I am finished" ***')

class LongOperationThread(threading.Thread):
    def __init__(self, daemon=True, start=True):
        super(LongOperationThread, self).__init__()
        self.running = threading.Event()
        self.term = threading.Event()
        self.seconds = 0
        if start:
            self.start()
            
    def start(self):
        super(LongOperationThread, self).start()

    def run(self):
        while not self.term.is_set():
            if self.running.is_set():
                long_operation_thread(self.seconds, self.window)
                self.running.clear()

    def do(self, seconds, window):
        self.seconds = seconds
        self.window = window
        self.running.set()
        
    def stop(self):
        self.term.set()
        
    def is_running(self):
        return self.running.is_set()
    
def the_gui():
    """
    Starts and executes the GUI
    Reads data from a global variable and displays
    Returns when the user exits / closes the window
    """

    sg.theme('Light Brown 3')

    layout = [[sg.Text('Long task to perform example')],
              [sg.MLine(size=(80, 12), k='-ML-', reroute_stdout=True,write_only=True, autoscroll=True, auto_refresh=True)],
              [sg.Text('Number of seconds your task will take'),
               sg.Input(key='-SECONDS-', focus=True, size=(5, 1)),
               sg.Button('Do Long Task', bind_return_key=True),
               sg.CBox('ONE chunk, cannot break apart', key='-ONE CHUNK-')],
              [sg.Text('Work progress'), sg.ProgressBar(100, size=(20, 20), orientation='h', key='-PROG-')],
              [sg.Button('Click Me'), sg.Button('Exit')], ]

    window = sg.Window('Multithreaded Demonstration Window', layout, finalize=True)

    timeout = thread = None
    thread = LongOperationThread()
    # --------------------- EVENT LOOP ---------------------
    while True:
        event, values = window.read(timeout=timeout)
        # print(event, values)
        if event in (sg.WIN_CLOSED, 'Exit'):
            thread.stop()
            thread.join()
            break
        elif event.startswith('Do') and not thread.is_running():
            print('Thread Starting! Long work....sending value of {} seconds'.format(float(values['-SECONDS-'])))
            timeout = 100 if values['-ONE CHUNK-'] else None
            thread.do(float(values['-SECONDS-']),window)
            if values['-ONE CHUNK-']:
                sg.popup_animated(sg.DEFAULT_BASE64_LOADING_GIF, background_color='white', transparent_color='white', time_between_frames=100)
        elif event == 'Click Me':
            print('Your GUI is alive and well')
        elif event == '-PROGRESS-':
            if not values['-ONE CHUNK-']:
                window['-PROG-'].update_bar(values[event], 100)
        elif event == '-THREAD-':            # Thread has completed
            print('Thread job done')
            sg.popup_animated(None)                     # stop animination in case one is running
            message, progress, timeout = '', 0, None     # reset variables for next run
            window['-PROG-'].update_bar(0,0)            # clear the progress bar
        if values['-ONE CHUNK-'] and thread is not None:
            sg.popup_animated(sg.DEFAULT_BASE64_LOADING_GIF, background_color='white', transparent_color='white', time_between_frames=100)
    window.close()


if __name__ == '__main__':
    the_gui()
    print('Exiting Program')

This is a known problem. I’m trying to find the Issue that I opened / announcement I made regarding the problem.

The search is on for a new way to signal from a thread. I could swear I opened an Issue that explained the problem and that the only safe thing to do is to not make any PySimpleGUI calls from a thread until I can solve the problem with write_event_value.

It’s a high priority item.

Thanks a lot for the support and your project as a whole.
I intended that #4009 was the last activity on the subject and therefore I hoped that my test case could help.
If I understand correctly, the use of self.thread_strvar.set(‘new item’) is a way to propagate the queue event downto main loop.
Why not replacing it with a real thread safe event?
I mean,

self.thread_queue = queue.Queue()
self.thread_event = threading.Event()

Then I would use self.thread_event.set() within write_event_value()
and methods is_set() and clear() somewhere in the main loop.
BR

OK…. here’s the most complete picture of the situation I’m able to provide at the moment….

The «most threadafe» way of operating is to make zero calls into PySimpleGUI (and thus tkinter) from a thread, and to use a signalling mechanism similar to your suggestion of a queue. This is what you’ll find was the original way threads be handled with PySimpleGUI.

This comment from Demo_Multithreaded_Long_Tasks:

"""
    DESIGN PATTERN - Multithreaded Long Tasks GUI
    Presents one method for running long-running operations in a PySimpleGUI environment.
    The PySimpleGUI code, and thus the underlying GUI framework, runs as the primary, main thread
    The "long work" is contained in the thread that is being started.

    July 2020 - Note that this program has been updated to use the new Window.write_event_value method.
    This method has not yet been ported to the other PySimpleGUI ports and is thus limited to the tkinter ports for now.
    
    Internally to PySimpleGUI, a queue.Queue is used by the threads to communicate with main GUI code
    The PySimpleGUI code is structured just like a typical PySimpleGUI program.  A layout defined,
        a Window is created, and an event loop is executed.


    This design pattern works for all of the flavors of PySimpleGUI including the Web and also repl.it
    You'll find a repl.it version here: https://repl.it/@PySimpleGUI/Async-With-Queue-Communicationspy
"""

I seem to have retained the older mechanism in the Demo Programs in the Demo_Multithreaded_popup.py

Here’s the code from that Demo:

import threading
import time
import PySimpleGUI as sg
import queue

"""
    Threading Demo - "Call popup from a thread"

    Can be extended to call any PySimpleGUI function by passing the function through the queue


    Safest approach to threading is to use a Queue object to communicate
    between threads and maintrhead.

    The thread calls popup, a LOCAL function that should be called with the same
    parameters that would be used to call opiup when called directly

    The parameters passed to the local popup are passed through a queue to the main thread.
    When a messages is received from the queue, sg.popup is called using the parms passed
    through the queue
    
    Copyright 2021 PySimpleGUI.org
"""

mainthread_queue:queue.Queue = None

def popup(*args, **kwargs):
    if mainthread_queue:
        mainthread_queue.put((args, kwargs))

def the_thread(count):
    """
    The thread that communicates with the application through the window's events.

    Once a second wakes and sends a new event and associated value to the window
    """
    i = 0
    while True:
        time.sleep(2)
        popup(f'Hello, this is the thread #{count}', 'My counter value', i, text_color='white', background_color='red', non_blocking=True, keep_on_top=True, location=(1000-200*count, 400))
        i += 1


def process_popup():
    try:
        queued_value = mainthread_queue.get_nowait()
        sg.popup_auto_close(*queued_value[0], **queued_value[1])
    except queue.Empty:     # get_nowait() will get exception when Queue is empty
        pass


def main():
    """
    The demo will display in the multiline info about the event and values dictionary as it is being
    returned from window.read()
    Every time "Start" is clicked a new thread is started
    Try clicking "Dummy" to see that the window is active while the thread stuff is happening in the background
    """
    global mainthread_queue

    mainthread_queue = queue.Queue()

    layout = [  [sg.Text('Output Area - cprint's route to here', font='Any 15')],
                [sg.Multiline(size=(65,20), key='-ML-', autoscroll=True, reroute_stdout=True, write_only=True, reroute_cprint=True)],
                [sg.T('Input so you can see data in your dictionary')],
                [sg.Input(key='-IN-', size=(30,1))],
                [sg.B('Start A Thread'), sg.B('Dummy'), sg.Button('Exit')]  ]

    window = sg.Window('Window Title', layout, finalize=True, keep_on_top=True)
    count = 0
    while True:             # Event Loop
        event, values = window.read(timeout=500)
        sg.cprint(event, values) if event != sg.TIMEOUT_EVENT else None
        if event == sg.WIN_CLOSED or event == 'Exit':
            break
        process_popup()
        if event.startswith('Start'):
            threading.Thread(target=the_thread, args=(count,), daemon=True).start()
            count += 1
    window.close()

if __name__ == '__main__':
    main()

Getting back to your direct question

Why not replacing it with a real thread safe event?

The answer is — efficiency

Without a way of signaling an event to tkinter, the user is left with this very safe mechanism, but it comes at a price. That price is the time it takes to poll the queue for new items from the thread.

The purpose of the write_event_value method was to facilitate not only the queue portion but most importantly, the ability to interrupt the tkinter mainloop. Without this ability to signal tkinter, the user cannot perform a normal read:

event, values = window.read()

Once it appeared that the older form of threaded communication wasn’t needed, and that the greatly simplified since call write_event_value could be used, there was a large effort expended to convert all of the demos and documentation to use this new technique.

It worked well…… for a long time…… until it didn’t anymore which is where we are today.

So the search is on for something that can be called from a thread that will allow me to signal tkinter and thus break out of the tkinter event loop.

Thanks for your time explaining me the whole picture. I think I got your point.
I also found the discussion in #3580 with your explanations that further helped my understanding.
I therefore spent some time implementing changes both to my code and also to PySimpleGUI.py.
Finally I was able to understand the root cause and to solve.
I share hereafter my results that maybe could be of intererest to someone.

Initially I was looking for issues related to unsafe writing to global data or Global Collector side-effects.
Therefore my first approach was to change the method of signaling tkinter and invoking the callback.
This didn’t yield the solution, but the findings maybe are interesting.
I tested various implementations for write_event_value().
I compared:

• original code using thread_strvar.set() plus thread_strvar.trace() callback on writing
• modified code using thread_strvar.get() plus thread_strvar.trace() callback on reading
• modified code using Tkroot.event_generate(‘<<SG>>’) plus TKroot.bind() callback to virtual event

I measured the execution time for 1000 loop calls from within a thread worker.
All worked the same and all lasted around 53.4 seconds on my machine.
Then I evaluated, without success, the usage of threading.Lock() and traced the start/stop of GC.
Finally I suspected that the issue could be related to an expired timeout from within TCL/TK.
I localized the timeout within the source _tinker.c
Tkapp_Call() calls WaitForMainloop() where there is a timeout in case mainloop is not executed within 1 second.
That is the origin of message «RuntimeError: main thread is not in main loop».
Infact my suspect comes from the fact that it is not said «main loop is not in main thread», but the converse.
Tkinter perfectly supports Tk calls from another thread by marshalling into the thread where the TCL interpreter executes.
The issue here is that mainloop is randomly not executed in time or not enough for consuming all the events launched by threads.

Coming to the solution, PySimpleGUI doesn’t require changes (anyway, about that see more below).

The modified Demo_Multithreaded_Long_Task_Simple.py (the Test Case attached initially),
triggered the issue on app closing because the worker thread still generates events for seconds before stopping,
while mainloop was not called to process them because the execution was blocked by thread.join().
The fix requires the following changes:

def long_operation_thread(seconds, window, term):
    """
    A worker thread that communicates with the GUI through a global message variable
    This thread can block for as long as it wants and the GUI will not be affected
    :param seconds: (int) How long to sleep, the ultimate blocking call
    """
    progress = 0
    print('Thread started - will sleep for {} seconds'.format(seconds))
    for i in range(int(seconds * 10)):
        time.sleep(.1)  # sleep for a while
        if term.is_set(): break
        progress += 100 / (seconds * 10)
        window.write_event_value('-PROGRESS-', progress)
    window.write_event_value('-THREAD-', '*** The thread says.... "I am finished" ***')
    
    [...]
    
    def run(self):
        while not self.term.is_set():
            if self.running.is_set():
                long_operation_thread(self.seconds, self.window, self.term)
                self.running.clear()
    [...]
    
    # --------------------- EVENT LOOP ---------------------
    while True:
        event, values = window.read(timeout=timeout)
        # print(event, values)
        if event in (sg.WIN_CLOSED, 'Exit'):
            thread.stop()
            window.read(timeout=1000)
            thread.join()
            break
  

Instead the other case of my simple app using threads (that I didn’t shared because too long) triggered the issue randomly during execution.
It was more difficult to understand the reason, although with the same root cause.
The GUI was a simple event loop:

    while True:
        event, values = _window.read(timeout=1)
        if event == '-LOG-':
            _window['ML-Log'].update(f'{values[EVENT_LOG]}n', append=True)
        [...]

The event ‘-LOG-‘ is generated from a callback thread using write_event_value().
The application generally works, but randomly the execution stalled for 1 second and then only the thread crashed with exception «RuntimeError: main thread is not in main loop».
Everything is finally fixed by using _window.read(timeout=10) instead of _window.read(timeout=1).

That is interesting to me… with just 1ms timeout it seems that the execution of mainloop is not always guaranteed.
Randomly for some event «in-flight», launched by the thread, the resulting Tkapp_Call() remained stalled within Tkinter because mainloop is not executed, until the timeout of 1 second expires.
It appears that a 1ms timeout for window.read() is critic because of some aspect related to thread scheduling.
It could be caused by something specific to Windows, Python GIL or TCL/TK.
Here I stop because I reached the limit of my knowledge and understanding.
I hope that what above could be of some usefulness.

My experience is that things that rely on timing….. work….. until they don’t.

This problem, and the many solutions that have been tried, has been a classic example of this. Until I can get a guaranteed way to make a call into tkinter from a thread that will not generate an error by tkinter, there is no safe technique that can be devised. I don’t yet have that call. I thought that the set call was, but clearly, it has not turned out to be this.

There are 2 ways to solve threading communications:

1. Polling
2. Waiting for an «interrupt»

What I’ve been trying to do with the write_event_values is to make the communications more «interrupt driven» (an embedded systems way of viewing this). Polling is inefficient. Being interrupt driven is the more efficient choice in most circumstances.

Thank you @daemon2021 for spending so much timing working on this. I’ll go through your analysis when I’ve got time that I can properly devote the attention it deserves.

DOH!

OK, I thought I got it nailed! LOL.,… I’m getting closer at least….

This is called a jinx right here. I know better than to claim victory quite so quickly. It’s OK, getting closer and understanding things a bit better.

Still working hard on this!!

MUCH better understanding, but when put under a massive load test, still have a few issues. Quite determined to get this solved! I do NOT want to have to use polling with threads unless absolutely necessary, and I continue to believe there is a path forward. SOON!

Thanks a lot for the update, I look forward for testing your changes once you are confident!

Your comment restarted my thinking… (you know, sometimes you really want «the bug» to go out of your head!)
I’ve searched a bit more around my feelings on the issue.
I’ve clarified myself that CPython has preemptive multitasking:
If a thread runs uninterrupted for 1000 bytecode instructions in Python 2, or runs 15 milliseconds in Python 3, then it gives up the GIL and another thread may run.
This means that swapping execution between threads can happen always, not just when the thread suspends as in cooperative multitasking (wait, sleep, I/O).
I see a critic situation between execution of mainloop() and window.write_event_value().
For example, in tk it is stated that execution of tk.update() must not be called from within an event callback.
And tk.update() is the core part of what mainloop() does.

Because of multi-threading preeemption, it could be that root.mainloop() is suspended while it is already processing the pending events and then thread_strvar.set() is executed. Maybe this could be the reason for the stall and consequent timeout.
I feel the need of identifying the critic section and to acquire/release a threading.Lock() around it.

I have the feeling that the culprit could be the following code used for _read() and read_all_windows()

            # normal read blocking code....
            if timeout != None:
                self.TimerCancelled = False
                self.TKAfterID = self.TKroot.after(timeout, self._TimeoutAlarmCallback)
            self.CurrentlyRunningMainloop = True
            Window._window_running_mainloop = self
            Window._root_running_mainloop.mainloop()

Maybe that portion needs addition of threading.Lock() for making it atomic as critic section (no preentive scheduling of other threads in-between).
Or else try replacing root.mainloop() with periodic running of:

thread_lock.acquire()
Window._root_running_mainloop.update() 
thread_lock.release()

I mean, moving the blocking part out of Tk and better in control of PySimpleGUI (i.e. with a loop and a timer).

Nope!… I tried with thread locking at the same time around both window.read(timeout=1) and around window.write_event_value, but no way…

But I have a different hypothesis now.
Suppose you have a sequence of two (or more) calls of window.write_event_value that are very close to each other in time (as it happens under high load).
With timeout=1 I suspect there is time for mainloop to execute only the first write_event_value.
Also it could be that the related callback will be executed later, on next time window.read() is called.
When the second call of write_event_value is executed, it finds that mainloop has been already exited by the first call.
Therefore the inner Tkapp_Call() will let expire the «one second timeout» of WaitForMainloop().
Could it be?

I’ll try and get my changes posted today. There are too many balls in the air but want to get this into your hands to use. I can’t explain it all here and now. I would rather get it done and over to you.

Can you email me? Contact info is with the account info.

A HUGE THANK YOU owed to @daemon2021 !

######## ##     ##    ###    ##    ## ##    ##    ##    ##  #######  ##     ## #### #### #### 
   ##    ##     ##   ## ##   ###   ## ##   ##      ##  ##  ##     ## ##     ## #### #### #### 
   ##    ##     ##  ##   ##  ####  ## ##  ##        ####   ##     ## ##     ## #### #### #### 
   ##    ######### ##     ## ## ## ## #####          ##    ##     ## ##     ##  ##   ##   ##  
   ##    ##     ## ######### ##  #### ##  ##         ##    ##     ## ##     ##                
   ##    ##     ## ##     ## ##   ### ##   ##        ##    ##     ## ##     ## #### #### #### 
   ##    ##     ## ##     ## ##    ## ##    ##       ##     #######   #######  #### #### ####

Thank you for taking on unraveling all this. I’m stunned…. I appreciate you contacting me via email so that we could continue to work this. The solution ended up being 1 additional line of code, but it might as well been 1,000 given the amount of research and testing done by @daemon2021

I have tested the change on 3.6, 3.7, 3.8, 3.9 as well as testing on Linux and I never got a single instance of the dreaded error message.

4.38.0.11 has the fix.

I will release this to PyPI over the weekend!

Now we get to celebrate @daemon2021 and close the issue…

Posted to PyPI in version 4.39.1

Thank you for all the acknowledgement!
I just left to you the «dirty» work of validating it… 😁
This morning I synchronized my code to 4.39.1.
I just wanted to highlight another interesting improvement.
I timed again the execution time for 1000 loop calls from within a thread worker.
What before the change tooks around 53.4 seconds on my machine, now it takes 6.6 seconds! 😲
Not bad a 8x speedup!… Before the last change a lot of time was lost within tkinter’s windings.

Got the error randomly today though was using only write events same code works fine but got this error 2day will increasing timeout value from 1to 10 help plz suggest

It will be better to post a new issue for your case, more detail information required for your case.

suggest

As Jason said, open an issue. Include details.
Have you tried running garbage collect anytime something is destroyed?

When you open the issue, but sure to include all information printed with the error. It’s important to know which part of tkinter is complaining.

@PySimpleGUI You are late, new issued already opened and closed yesterday.
Issue #5976

@PySimpleGUI You are late

Being in a state of «Late» and «Confused» is the norm.

I don’t know how I missed it in the list of closed items…. I’m sorry… thank you for both being a wizard and making problems disappear and letting me know.

Есть код(приводить нет смысла) с использованием Tkinter который прекрасно работает на виндовс и мак, но при таком импортирование из __init__.py:

При любом вызове label.configure(text=’1111′) в потоке получаю: RuntimeError: main thread is not in main loop

Из-за чего происходит и как лечить? В принципе понятно что дело в потоке, так как Tkinter с ним не работает, но почему при вызове напрямую init.py работает, а через __init__.py нет?

Нашел вот такую вещь http://tkinter.unpythonic.net/wiki/mtTkinter
Стоит ли с ней возиться? или забить на все Tkinter, ttk, и все же перейти как и думал месяц назад на PyQt4..

Спасибо)

4kpt_III
__init__.py это зарегистрированное имя. Я называю несущий файл в этом случае system.py или main.py.__init__.py имеет смысл, когда Вы работаете с пакетом. Но тогда указывается название пакета, а __init__.py просто лежит в нем. Кроме того в __init__.py есть возможность конфигурировать загрузку пакета и еще немножко няшностей

Ну не обязательно __init__.py можно что нибуть другое, главное чтоб импортируемый import модуль сработал так как будто он главный(первый) стартовал… В любом другом случаи выводит ошибки: RuntimeError: main thread is not in main loop

Что за няшности если не секрет?

P.s. уже 4kpt_III, а со старыми что?)

Отредактировано 4kpt_III (Дек. 22, 2014 14:55:32)

4kpt_III
Я не совсем понимаю Вашу ситуацию. Я делаю так. Создаю файл, который запускает базовый виджет (чаще всего это root с верхним меню) . Все остальные виджеты импортируются по необходимости при выборе нужного пункта меню. Строятся они, чаще всего либо на frame, либо на toplevel. В качестве аргумента я им передаю или весь root или его часть под меню (зависит от того, каким меню я пользовался — если встроенным — то весь root, если своим — то внутренний frame). Они уже на нем и строятся (используют или root или frame в качестве подложки). Классу, построенному на toplevel я не передаю ничего Вот про этот system.py я и писал. В любом моем проекте GUI является основным управляющим элементом. Все остальное строится уже на нем: вызов БД, создание или изменение файлов, подключение к сервисам или к сайтам и т.п.

Хм а интересная структура) возьму на заметку) у меня почти похоже.. ток вместо импорта классы…

Попробую еще раз объяснить.. Мне по сути просто нужно(для примера): файл start.py в нем import dops.py, и больше ничего. Это мне нужно для того что бы скрыть исходники хоть чуток(специфика мак).
Но почему то при таком примере ткинтер перестает обрабатывать переданные переменные в любом потоке..

Нужно что бы при запуске start.py я не почувствовал разницу со стартом dops.py..

P.s. ясненько) тогда добавлю красоты в виде репы) а то 0… не смотриться

4kpt_III
Многопоточное приложение?

Да

4kpt_III
Ну тогда сложно. Многопоточность и tkinter это вообще отдельная работа. Простое импортирование работает. Только что проверил. Как с threading я даже не знаю… Нужно именно копаться. Честно говоря я не сталкивался. Многопоточность дело нужно, но вот tkinter с ним прямо дружить не очень любит.

С threading у меня так(это в dops.py):

from Tkinter import Tk, Text
from threading import Thread
root = Tk()
txt = Text(root, text='1111')
txt.pack()
def test(txt):
    txt.insert(INSERT, '22222')
    txt.configure(state='disabled') #RuntimeError: main thread is not in main loop
Thread(target=test, args=(txt, )).start()
root.mainloop()

http://tkinter.unpythonic.net/wiki/mtTkinter — вот это должно заставить Tkinter работать с многопоточностью.. Так ли оно или еще больше проблем будит с ним?

Отредактировано 4kpt_III (Дек. 22, 2014 15:39:44)