Why are my Python multiprocessing.Pool subprocess logs going to my log file but not the stream (command window)?

Question

Python version 3.13.7 on Windows 11 Enterprise

I am running a FileHandler and StreamHandler using a QueueHandler for my pyton script. The main process logs go to both the stream and file, but the logs for the two subprocesses using mutiprocessing.Pool only go to the file.

I placed a third pass that runs the subprocess outside of the multiprocess to make sure it wasn't a coding error in my subprocess, and it functions as expected.

I put a print command into the subprocess to see if it was the StreamHandler or QueueHandler malfunctioning. The print does not function during multiprocessing, but will display for the third pass outside of the multiprocess.

I know I am capturing everything in the log file (except the print commands), but I am running a multi-step subprocess with large data sets using this setup and would like to see where the process is without having to constantly open the log file. All the documentation and tutorials I have been able to find point me to this setup.

Everything works correctly when writing to the log file, just not the stream output. How do I make the subprocess logging commands go to the the stream (command window) in addition to the log file while using multiprocessing.Pool?

stream output:

INFO     | 2026-06-29 09:26:41 | Running: test.py
INFO     | 2026-06-29 09:26:41 | Start processing
INFO     | 2026-06-29 09:26:46 | Starting test3
test3 yo we here!
INFO     | 2026-06-29 09:26:46 | Finished test3 
INFO     | 2026-06-29 09:26:46 | End processing

file output:

2026-06-29 09:26:41 | INFO     | Main           | 100  | Running: test.py
2026-06-29 09:26:41 | INFO     | Main           | 108  | Start processing
2026-06-29 09:26:41 | INFO     | workerProcess  | 79   | Starting test1
2026-06-29 09:26:41 | INFO     | workerProcess  | 83   | Finished test1 
2026-06-29 09:26:41 | INFO     | workerProcess  | 79   | Starting test2
2026-06-29 09:26:41 | INFO     | workerProcess  | 83   | Finished test2 
2026-06-29 09:26:46 | INFO     | workerProcess  | 79   | Starting test3
2026-06-29 09:26:46 | INFO     | workerProcess  | 83   | Finished test3 
2026-06-29 09:26:46 | INFO     | Main           | 118  | End processing

Test.py

import logging
import multiprocessing
from logging.handlers import QueueHandler
import sys
import time

import myLogger
myLogger.Main(sys.argv[1], "TEST")


def logging_process(queue):
    logger = logging.getLogger('app')
    while True:
        message = queue.get()
        if message is None:
            break
        
def workerProcess(Args):
    queue, outputDir, rFC = Args
    
    # Set logging queue
    logger = logging.getLogger('app')
    logger.addHandler(QueueHandler(queue))

    logger.info("Starting {}".format(rFC))
    print("{} yo we here!".format(rFC))

    # Export to final geodatabase
    logger.info("Finished {} ".format(rFC))


def Main(outputDir):
    # Start logging fuction
    with multiprocessing.Manager() as manager:
        queue = manager.Queue()
        
        logger = logging.getLogger('app')
        logger.addHandler(QueueHandler(queue))
        logger_proc = multiprocessing.Process(target=logging_process, args =(queue,))
        logger_proc.start()
        
        try:
            logger.info("Running: test.py")

            # Determine number of processing pools
            processes = []
            for rFC in ('test1', 'test2'):
                processes.append((queue, outputDir, rFC))
            
            # Start threads
            logger.info("Start processing")
            with multiprocessing.Pool(4) as pool:
                pool.map(workerProcess, processes)

                pool.close()
                pool.join
            
            time.sleep(5)
            workerProcess((queue, outputDir, 'test3'))
            # Declare running time
            logger.info("End processing")

        except Exception as e:
            logger.exception(e)
            logger.error("error", stack_info=True)
            raise

        # End logging function
        queue.put(None)
        logger_proc.join()  # Always join started processes


if __name__ == "__main__":
    # Running Main Program
    Main(sys.argv[1])

myLogger.py

import logging
import os
import sys

def handle_exception(exc_type, exc_value, exc_traceback):
    if issubclass(exc_type, KeyboardInterrupt):
        sys.__excepthook__(exc_type, exc_value, exc_traceback)
        return

    logger.critical("Uncaught exception", exc_info=(exc_type, exc_value, exc_traceback))

def Main(log_path, log_name):
    # Create and set log path
    if not os.path.exists("{}".format(log_path)):
        os.mkdir("{}".format(log_path))

    status_log_target = os.path.join(log_path, "{}_status.log".format(log_name))

    # Set up the logger format
    logger = logging.getLogger('app')
    logger.setLevel(logging.DEBUG)

    sh = logging.StreamHandler(sys.stdout)
    sh.setLevel(logging.INFO)
    shortFMT = logging.Formatter(fmt="%(levelname)-8s | %(asctime)s | %(message)s", datefmt="%Y-%m-%d %H:%M:%S", style='%')
    sh.setFormatter(shortFMT)
    
    fh = logging.FileHandler(status_log_target)
    fh.setLevel(logging.DEBUG)
    longFMT= logging.Formatter(fmt="%(asctime)s | %(levelname)-8s | %(funcName)-15s| %(lineno)-4d | %(message)s", datefmt="%Y-%m-%d %H:%M:%S", style='%')
    fh.setFormatter(longFMT)

    
    logger.addHandler(fh)
    logger.addHandler(sh)

    # Handle exceptions
    sys.excepthook = handle_exception


if __name__ == "__main__":
    Main(sys.argv[1], sys.argv[2])

user2357112 · Accepted Answer · 2026-06-29 11:56:53Z

3

You set up a QueueHandler, but for that to make sense, you need another process listening on the queue with a QueueListener. The QueueListener reads log records from the queue and passes them to handlers to handle them. As things stand, your logging_process just ignores all messages.

The logging output you see has nothing to do with the QueueHandler. When you do

myLogger.Main(sys.argv[1], "TEST")

outside of an if __name__ == "__main__" guard, this call sets up a StreamHandler writing to stdout and a FileHandler writing to the file. Because this is outside of any if __name__ == "__main__" guard, it happens for the worker processes too. That means the worker processes are directly logging to stdout, and directly logging to the file, instead of handling that through the queue.

All the processes share the same log file, so you see everything they log to the file. But the processes don't all share the same stdout. Only the master process's stdout is connected to your terminal, so that's the only stdout you see. Anything the workers write to their own stdout doesn't show up on your screen.

answered Jun 29 at 11:56

user2357112

287k32 gold badges494 silver badges577 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

LyseBeke Jun 30 at 11:24

I am using a listener process (logging_process) in place of a QueueListener. Based on the example from Python Logging Cookbook Logging to a single file from multiple processes. Then Multiprocessing Pool Logging From Worker Processes helped with the set up of the multiprocessing.Manager, that is required for using multiprocessing.Pool.

user2357112 Jun 30 at 11:36

You've got an ad-hoc DIY listener process, but your listener process just ignores messages. The listener process in the cookbook actually has log handling logic in it.

LyseBeke Jul 1 at 7:26

I get the same result when running the cookbook example. The worker process' print statements do not show up on the screen. You stated earlier that the worker processes write to their own stdout, is there a way to merge the stdouts to my main terminal? I thought that was what the listener process was doing. Or do I need to approach the terminal display in a different manner?

user2357112 Jul 2 at 7:53

Normally they would already share the same stdout, but I suspect you're running this code in an environment where the master process does some automatic setup that changes its stdout. That, or it's a Windows thing I don't remember.

user2357112 Jul 2 at 7:56

But even when they do share a stdout, concurrent unsynchronized access to stdout can produce garbled output. (Same with concurrent unsynchronized access to the log file, which is one of the reasons to use a QueueHandler instead of having every process try to log things directly.) The print statements in the cookbook example aren't something you should do in a real program.

user2357112 Jul 2 at 8:00

A QueueListener sets up a background thread to do its work, so you can run it in the master process if you want without blocking execution. If you run the QueueListener in the main process, anything it writes to stdout should show up normally. You can do the same with a DIY listener that you launch in a background thread manually, if you want.

Collectives™ on Stack Overflow

Why are my Python multiprocessing.Pool subprocess logs going to my log file but not the stream (command window)?

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related