Logging Best Practices

This document outlines simple logging practices for a small development team. No over-engineering, just log what you need.

Current Implementation Overview

Our logging system uses structlog with minimal abstractions:

• Structured JSON logging for production environments
• Correlation ID tracking for request tracing (when needed)
• Basic performance monitoring (when you actually need it)
• Sentry integration for error tracking

Core Principles

1. Keep It Simple

Purpose: Log what you need to debug issues. Don't over-engineer for problems you don't have.

Key Goals:

• Enable fast debugging
• Track important business events
• Monitor critical performance issues
• Don't create abstractions for the sake of abstractions

2. Use Direct Logging

Just use the logger directly with key-value pairs:

from ghost.logs import log

# Simple and direct - just log what happened
log.info("Contact created", user_id=user.id, contact_id=contact.id, table_id="contacts")

# Database operations - just log directly
log.info("Database query completed",
         operation="insert",
         table="contacts",
         user_id=user.id,
         duration_ms=45.2)

# API calls - no special methods needed
log.info("External API call",
         service="stripe",
         endpoint="customers.create",
         status_code=200,
         user_id=user.id)

3. Include Context When It Matters

Add correlation IDs only when you actually need request tracing:

from ghost.logs import with_correlation_id

# Only use correlation IDs for complex operations that span multiple services
corr_logger = with_correlation_id()
corr_logger.info("Starting complex operation", operation="bulk_import")

Available Methods

Basic Logging Methods

from ghost.logs import log

# Standard log levels - use these directly
log.debug("Detailed info for debugging", variable=value)
log.info("General information", event="user_action", user_id=123)
log.warn("Something unexpected but recoverable", issue="rate_limit_approaching")
log.error("An error occurred", error=str(exception), user_id=123)
log.critical("System is in critical state", system="database")

Performance Tracking (When You Actually Need It)

Only use performance tracking for operations you're actually monitoring:

from ghost.logs import log

# Only for operations where performance matters
with log.performance_context("csv_import", file_size=file_size) as perf_log:
    contacts = process_csv_file(file)
    perf_log.info("CSV processing completed", contacts_processed=len(contacts))
# Automatically logs duration

Error Handling

For errors that need special handling:

from ghost.logs import log

try:
    # some operation
    pass
except Exception as e:
    # Usually just log and re-raise naturally
    log.error("Operation failed", error=str(e), user_id=user.id, operation="contact_import")
    raise

# Only use error_and_raise for special cases where you need Sentry integration
try:
    # some operation
    pass
except Exception as e:
    log.error_and_raise(e, propagate_to_sentry=True, user_id=user.id, operation="critical_operation")

Log Levels and When to Use Them

DEBUG

• Detailed diagnostic information
• Only in development/staging
• Variable values and state changes

log.debug("Processing CSV row", row_number=row_num, email=email)

INFO

• General application flow
• Business events
• Successful operations
• This is what you'll use 90% of the time

log.info("User logged in", user_id=user.id)
log.info("Database operation completed", operation="insert", table="contacts", user_id=user.id)
log.info("Email sent", campaign_id=campaign.id, contact_id=contact.id)

WARNING

• Recoverable errors
• Performance issues
• Unexpected but handled conditions

log.warn("Slow database query", operation="select", table="contacts", duration_ms=1500)
log.warn("Rate limit approaching", user_id=user.id, current_usage=95)

ERROR

• Application errors
• Failed operations
• Exceptions that are handled

log.error("Failed to send email", error=str(e), contact_id=contact.id, campaign_id=campaign.id)
log.error("Database connection failed", error=str(e), retry_count=3)

CRITICAL

• System failures
• Security breaches
• Data corruption

log.critical("Database is down", error=str(e))
log.critical("Security breach detected", user_id=user.id, ip_address=request.ip)

Common Patterns

Database Operations

# Simple and direct
log.info("Database query", operation="select", table="contacts", user_id=user.id, result_count=150)
log.info("Database update", operation="update", table="contacts", record_id=contact.id, user_id=user.id)

API Requests

# Just log what you need
log.info("API request completed", method="POST", endpoint="/api/contacts/", status_code=201, user_id=user.id)

Business Events

# No special methods needed
log.info("Contact created", contact_id=contact.id, user_id=user.id, source="csv_import")
log.info("Campaign started", campaign_id=campaign.id, contact_count=500)

Security Events

# Just use warning or error levels
log.warn("Failed login attempt", user_email=email, ip_address=request.ip)
log.error("Suspicious activity detected", user_id=user.id, activity="multiple_failed_logins")

What NOT to Do

Don't Over-Abstract

# ❌ DON'T create special methods for everything
log.log_database_operation(operation="select", table="contacts")

# ✅ DO just log directly
log.info("Database query completed", operation="select", table="contacts", user_id=user.id)

Don't Create Unnecessary Context Bindings

# ❌ DON'T create complex context managers for simple operations
user_logger = with_user_context(user_id=user.id, account_id=account.id)
business_logger = user_logger.bind_business_context(campaign_id=campaign.id)

# ✅ DO just include the context you need
log.info("Campaign started", user_id=user.id, account_id=account.id, campaign_id=campaign.id)

Don't Log Everything

# ❌ DON'T log every single operation
log.debug("Entering function", function="process_contact")
log.debug("Validating email", email=contact.email)
log.debug("Email valid", email=contact.email)
log.debug("Exiting function", function="process_contact")

# ✅ DO log meaningful events
log.info("Contact processed", contact_id=contact.id, user_id=user.id, validation_status="valid")

Security Considerations

Never Log Sensitive Data

# ❌ NEVER log passwords, tokens, or sensitive data
log.info("User login", password=password, api_key=api_key)

# ✅ Log safe identifiers only
log.info("User login", user_id=user.id, login_method="password")

Log Security Events

# Log security-relevant events
log.warn("Failed login", user_email=email, ip_address=request.ip, attempt_count=3)
log.error("Unauthorized access attempt", endpoint="/admin/", ip_address=request.ip)

Performance Considerations

Don't Log in Hot Paths

# ❌ DON'T log in tight loops
for contact in contacts:  # 10,000 contacts
    log.debug("Processing contact", contact_id=contact.id)  # 10,000 log entries!

# ✅ DO log batches or summaries
log.info("Processing contacts", batch_size=len(contacts))
# Process all contacts
log.info("Contacts processed", processed_count=len(contacts), failed_count=failed)

Use Appropriate Log Levels

• DEBUG: Only in development
• INFO: Production events you care about
• WARNING/ERROR: Issues that need attention

Example Usage

Here's how you should typically use logging in your code:

from ghost.logs import log

def import_contacts(user_id, csv_file):
    """Import contacts from CSV file"""

    log.info("Contact import started", user_id=user_id, filename=csv_file.name)

    try:
        contacts = parse_csv(csv_file)
        log.info("CSV parsed", user_id=user_id, contact_count=len(contacts))

        # Only use performance tracking for slow operations
        with log.performance_context("database_bulk_insert") as perf_log:
            saved_contacts = save_contacts_to_db(contacts)
            perf_log.info("Contacts saved",
                         user_id=user_id,
                         saved_count=len(saved_contacts),
                         table="contacts")

        log.info("Contact import completed",
                user_id=user_id,
                total_contacts=len(contacts),
                successful=len(saved_contacts))

        return saved_contacts

    except Exception as e:
        log.error("Contact import failed",
                 user_id=user_id,
                 error=str(e),
                 filename=csv_file.name)
        raise

Remember: Keep it simple. Log what you need. Don't over-engineer.