Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using ProcessorFormatter with RotatingFileHandler misformats the events #119

Open
if-fi opened this issue May 14, 2017 · 0 comments
Open

Comments

@if-fi
Copy link

if-fi commented May 14, 2017

I have the following configuration:

structlog.configure(
    processors=[
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.stdlib.PositionalArgumentsFormatter(),
        structlog.processors.UnicodeEncoder(encoding='utf-8', errors='backslashreplace'),
        structlog.processors.TimeStamper(fmt='iso'),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
    ],
    context_class=structlog.threadlocal.wrap_dict(dict),
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True,
)

pre_chain = [
    # Add the log level and a timestamp to the event_dict if the log entry
    # is not from structlog.
    structlog.stdlib.add_log_level,
    structlog.processors.TimeStamper(fmt='iso'),
]

LOGGING_CONFIG = {
    'version': 1,
    'disable_existing_loggers': False,

    'formatters': {        
        'plain_struct': {
            "()": structlog.stdlib.ProcessorFormatter,
            "foreign_pre_chain": pre_chain,
            "processor": structlog.dev.ConsoleRenderer(colors=False, pad_event=0),
        },
        'json_struct': {
            "()": structlog.stdlib.ProcessorFormatter,
            "foreign_pre_chain": pre_chain,
            "processor": structlog.processors.JSONRenderer(sort_keys=True),
        }
     },

    'filters': {
    },

    'handlers': {
        # Handler to log all the Website generic information
        'log_generic': {
              'class': 'logging.handlers.RotatingFileHandler',
              'level': 'DEBUG',
              'formatter': 'plain_struct',
              'filters': [],
              'maxBytes': 1024*1024,
              'filename': os.path.join(app.config.get('LOG_LOCATION'), 'app_debug.log')
        },

        # Handler to log all the Website generic information in json format
        'log_json': {
            'class': 'logging.handlers.RotatingFileHandler',
            'level': 'DEBUG',
            'formatter': 'json_struct',
            'filters': [],
            'maxBytes': 1024 * 1024,
            'filename': os.path.join(app.config.get('LOG_LOCATION'), 'app_json.log')
        },
    },

    'loggers': {
        # logs website errors and debug logs
        'fbu': {
            'handlers': ['log_json', 'log_generic'],
            'level': 'DEBUG',
            'propagate': False
        },
    }
}

Due to some specifics in the RotatingFileHandler internal implementation, the log messages go through formatting twice, and I end up with those log messages in both files:

app_debug.log:

2017-05-14T21:03:26.563866Z [info     ] 2017-05-14T21:03:26.563745Z [info     ] {"event": "{\"event\": \"I am here\", \"level\": \"info\", \"logger\": \"fbu.process\", \"timestamp\": \"2017-05-14T21:03:26.562886Z\"}", "level": "info", "timestamp": "2017-05-14T21:03:26.563333Z"}

app_json.log:

{"event": "{\"event\": \"I am here\", \"level\": \"info\", \"logger\": \"fbu.process\", \"timestamp\": \"2017-05-14T21:03:26.562886Z\"}", "level": "info", "timestamp": "2017-05-14T21:03:26.563333Z"}
if-fi pushed a commit to if-fi/structlog that referenced this issue May 14, 2017
…ynek#119)

Due to the nature of how RodatingFileHandler is implemented, the log recors are formatted twice.
Once in `def shouldRollover(self, record):` in `class RotatingFileHandler` and second time when `emit` is actually called after that
https://github.com/python/cpython/blob/master/Lib/logging/handlers.py#L71

That results in `ProcessorFormatter` formatting the same message twice, but the second time, it is not aware if the message itslef is a dict, as it arrives as string.

Here we are handling exactly this case - we check if the string record message that arrives, has json structure, and if it does, we treat it, as if we would treat a dict
if-fi pushed a commit to if-fi/structlog that referenced this issue May 14, 2017
…ynek#119)

Due to the nature of how RodatingFileHandler is implemented, the log recors are formatted twice.
Once in `def shouldRollover(self, record):` in `class RotatingFileHandler` and second time when `emit` is actually called after that
https://github.com/python/cpython/blob/master/Lib/logging/handlers.py#L71

That results in `ProcessorFormatter` formatting the same message twice, but the second time, it is not aware if the message itslef is a dict, as it arrives as string.

Here we are handling exactly this case - we check if the string record message that arrives, has json structure, and if it does, we treat it, as if we would treat a dict
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant