Distributed Tracing & Backend Performance

In the previous module, we fixed the frontend by adding a skeleton screen to the sale page. Users now see content immediately instead of a blank screen. But the skeleton loads for 2-3 seconds before products appear. The problem isn’t the frontend anymore - it’s the backend API. Let’s use distributed tracing to find out why.

Learning Objectives

By the end of this module, you will:

• Use distributed tracing to connect frontend performance to backend bottlenecks
• Simulate traffic to trigger Sentry’s automatic N+1 query detection
• Understand what N+1 queries are and how they impact performance
• Fix N+1 problems with SQL JOINs and database optimization
• Measure improvements with Sentry’s performance monitoring

Understanding Distributed Tracing

Distributed tracing lets you follow a request from the user’s browser, through your frontend, to your backend API, and down to the database. This is crucial for understanding end-to-end performance.

The configuration you added in Module 1 (tracePropagationTargets in frontend) automatically connects frontend and backend traces.

1. Verify distributed tracing is working

   Navigate to a product page in your app (http://localhost:4173)

2. Find a pageload transaction

   In Sentry Explore → Traces, find the pageload transaction. Search for span.op is pageload and click on any of the results.

   

3. View the full distributed trace

   Click “View Full Trace” to see both frontend and backend spans connected in a single waterfall view

   

What You Should See

A waterfall view showing your React pageload, the API request, database queries, and total response time. This is your performance debugging superpower - you can see exactly where time is spent across your entire stack.

The Problem: Sale API is Slow

We added a skeleton screen, which improved perceived performance (FCP is better). But the actual API response time is still terrible:

• Sale API response time: 2-3 seconds
• Impact: Users wait 2-3 seconds staring at skeletons before seeing products
• Root cause: Let’s use traffic simulation to reveal it (Sentry will detect it automatically!)

Frontend Fix ≠ Backend Fix

Skeleton screens improve perceived performance by showing content immediately. But they don’t fix actual performance. If your API is slow, users still wait - they just see a nicer loading state. Let’s fix the real problem.

Running the Traffic Simulation

Under normal load (< 10 users), the sale API seems fine. But during holiday traffic, it becomes unusably slow. Let’s simulate Black Friday traffic to reveal the bottleneck.

1. Run the traffic simulation script

   From the root of your unborked repository:

   pnpm generate:traffic

   This simulates 100 concurrent users hitting the sale endpoint repeatedly.

   

2. Watch Sentry detect the problem automatically

   • Open your backend project in Sentry
   • Navigate to Issues
   • Within 1-2 minutes, you’ll see a new performance issue appear:
     • Type: N+1 Query
     • Status: New
     • Transaction: GET /api/sale
     • Shows the repeating SQL query in the issue title

   

3. Click into the N+1 issue

   • Sentry automatically detected the problem!
   • Scroll down to Span Evidence to see:
     • Transaction: GET /api/sale
     • Parent Span: request_handler.express - /api/sale
     • Preceding Span: Initial query (e.g., SELECT FROM sale_categories)
     • Repeating Spans (18+): The N+1 query pattern

       SELECT id, product_id, sale_price, created_at
       FROM sale_prices
       WHERE sale_prices.product_id = $1
       LIMIT $2

   Automatic Detection

   Sentry automatically detects N+1 queries when it sees:

   • 5+ sequential database queries with similar descriptions
   • Total duration > 50ms
   • A source span (preceding span) followed by repeating query patterns

   No manual instrumentation required!

Understanding the N+1 Issue with Distributed Tracing

Sentry automatically flagged the N+1 problem. Now let’s use distributed tracing to see the actual query execution and understand the performance impact.

1. View the full trace

   • From the N+1 issue page, scroll down to Span Evidence
   • Click “View Full Trace” button (top right)
   • This opens the complete trace waterfall view

2. Examine the trace waterfall

   • You’ll see the classic “comb” pattern - many sequential database queries
   • Look for repeating spans with similar SQL queries
   • The Repeating Spans section shows you exactly which query is repeated (18+ times in this case)

   

3. Click on individual spans (optional)

   • Click any repeating database span to see:
     • Full SQL query with parameters
     • Query duration
     • Database connection details
   • This helps you understand exactly what’s being queried repeatedly

What is the N+1 Query Problem?

The N+1 problem occurs when you fetch N records, then make 1 additional query for each record to fetch related data.

In our sale API:

Looking at the code, here’s what’s happening:

sale.ts

// Fetch all products (multiple queries to get products, metadata, categories)
const allProducts = await tx.select().from(products);

// ⚠️ THEN, for each product, query sale prices individually:
for (const product of allProducts) {
  const salePrice = await tx
    .select()
    .from(salePrices)
    .where(eq(salePrices.productId, product.id))
    .limit(1);
  // ... build sale product
}

The problem:

1. Initial queries to fetch products, metadata, categories: ~100ms
2. For each of 50 products, query salePrices: ~40ms × 50 = 2000ms ⚠️

Total time: ~2100ms just from unnecessary queries! Total queries: 50+ database roundtrips instead of 1!

N+1 Query Anti-Pattern

The N+1 problem occurs when you fetch N records, then make 1 additional query for each record to fetch related data. In our case, we’re looping through products and querying salePrices once per product. Under load, this destroys performance.

Fixing the N+1 Problem with SQL JOINs

Now that we’ve identified the problem, let’s fix it by using SQL JOINs to fetch all data in a single query.

1. Open the sale route

   Navigate to apps/api/src/routes/sale.ts

2. Rewrite with JOINs instead of loops

   Replace the entire route with this optimized version:

   sale.ts

   import express, { Request, Response } from 'express';
   import * as Sentry from '@sentry/node';
   import { db } from '../db';
   import {
     products,
     salePrices,
     productMetadata,
     saleCategories,
   } from '../db/schema';
   import { eq } from 'drizzle-orm';
   import { SaleProduct } from '../types';
   
   const router = express.Router();
   
   router.get('/', async (_req: Request, res: Response) => {
     // Wrap the entire route handler in a Sentry span
     return await Sentry.startSpan(
       {
         name: 'GET /api/sale',
         op: 'http.server',
       },
       async (span) => {
         try {
           console.log('Fetching sale products with optimized query');
   
           // Track the database query with a child span
           const saleProductsRaw = await Sentry.startSpan(
             {
               name: 'db.query.sale_products_optimized',
               op: 'db.query',
             },
             async () => {
               // ✅ OPTIMIZED: Single query with JOINs - no N+1!
               return await db
                 .select({
                   // Product fields
                   id: products.id,
                   name: products.name,
                   description: products.description,
                   image: products.image,
                   price: products.price,
                   // Sale price from JOIN
                   salePrice: salePrices.salePrice,
                   // Metadata from JOIN
                   discount: productMetadata.discount,
                   saleCategory: productMetadata.saleCategory,
                   featured: productMetadata.featured,
                   priority: productMetadata.priority,
                   // Category description from JOIN
                   categoryDescription: saleCategories.description,
                 })
                 .from(products)
                 .innerJoin(salePrices, eq(products.id, salePrices.productId))
                 .leftJoin(
                   productMetadata,
                   eq(products.id, productMetadata.productId)
                 )
                 .leftJoin(
                   saleCategories,
                   eq(productMetadata.saleCategory, saleCategories.name)
                 );
             }
           );
   
           // Transform to SaleProduct type
           const result: SaleProduct[] = saleProductsRaw.map((row) => ({
             ...row,
             category: 'Sale', // All products in this endpoint are sale items
             originalPrice: row.price,
             discount: row.discount || null,
             saleCategory: row.saleCategory || null,
             featured: row.featured || false,
             priority: row.priority || 0,
             categoryDescription: row.categoryDescription || null,
           }));
   
           // Sort by priority
           result.sort((a, b) => (b.priority || 0) - (a.priority || 0));
   
           // Add span attributes to track query results
           span.setAttributes({
             'products.count': result.length,
             'db.queries.total': 1, // Down from 50+!
           });
   
           console.log(
             `Successfully fetched ${result.length} sale products with 1 query`
           );
   
           res.json(result);
         } catch (err: any) {
           Sentry.captureException(err);
           console.error(
             'Error fetching sale products:',
             err.message,
             err.stack
           );
           res.status(500).json({ error: 'Failed to fetch sale products' });
         }
       }
     );
   });
   
   router.get('/shop', async (_req: Request, res: Response) => {
     try {
       console.log('Fetching shop products');
       const allProducts = await db.select().from(products);
       console.log(`Successfully fetched ${allProducts.length} shop products`);
       res.json(allProducts);
     } catch (err: any) {
       Sentry.captureException(err);
       console.error('Error fetching shop products:', err.message, err.stack);
       res.status(500).json({ error: 'Failed to fetch shop products' });
     }
   });
   
   export default router;

   Key changes:

   • Before: Looped through products, querying salePrices once per product (N+1)
   • After: Single query with innerJoin and leftJoin to fetch everything at once
   • Sentry instrumentation: Added custom spans to track query performance
     • Parent span: GET /api/sale wraps the entire route
     • Child span: db.query.sale_products_optimized tracks the database query
     • Span attributes: products.count and db.queries.total for monitoring
   • Result: 1 query instead of 50+ queries! ✅

3. Test the optimization

   • Restart the backend: pnpm dev
   • Run the traffic simulation again: pnpm traffic
   • Watch Sentry for new transactions

4. Verify the fix in Sentry

   • Go to Issues → Find your N+1 issue

   • The issue should start auto-resolving as new traces come in without the N+1 pattern ✅

   • Check Performance → GET /api/sale

   • Click into a new transaction and expand the waterfall

   • You’ll now see clean, organized spans:

     • GET /api/sale (parent span) - ~150ms total
       • db.query.sale_products_optimized (child span) - ~80-100ms
     • No more 50+ repeating queries! ✅

   • The custom span attributes show:

     • products.count: Number of products returned
     • db.queries.total: 1 (was 50+)

   

Why Custom Spans?

Custom Sentry spans let you:

• Track specific operations: See exactly how long your database query takes
• Add context with attributes: Track query counts, row counts, or optimization flags
• Compare before/after: Easily see performance improvements in Sentry’s UI
• Debug regressions: If performance degrades, you’ll see which span slowed down

Measuring the Improvement

Let’s verify the optimization worked using Sentry’s performance data.

1. Compare API response times

   • Go to Insights → Backend
   • Look at the p95 duration over time
   • You should see a dramatic drop after deploying the fix:
     • Before: 2000-3000ms
     • After: 100-200ms ✅
     • Improvement: 93% faster!

   

2. Verify the N+1 is gone

   • Click into an optimized transaction
   • Expand the waterfall view
   • Before: 50+ sequential database queries (comb pattern)
   • After: Single query with JOINs ✅
   • Reduction: 99% fewer queries!

3. Check frontend improvement

   • Navigate to your frontend project in Sentry
   • Find /sale page transactions
   • The skeleton screen now disappears in < 300ms instead of 2500ms
   • Users see products almost instantly! ✅

What About Database Indexes?

Even with optimized JOINs, queries can be further improved with proper database indexes.

When to add indexes:

• Columns used in WHERE clauses (e.g., products.on_sale)
• Foreign key columns used in JOINs (e.g., discounts.product_id, inventory.product_id)
• Columns used in ORDER BY or frequently searched

Performance impact:

• Indexed queries can be 2-3x faster than unindexed queries
• Critical for large tables (100K+ rows)
• Trade-off: Slightly slower writes, much faster reads

For this workshop, the JOIN optimization is the critical fix. In production, you’d add indexes based on your actual query patterns and table sizes.

Database Performance Best Practices

• Use JOINs instead of separate queries to avoid N+1 problems
• Add indexes on frequently filtered and joined columns
• Select only needed columns - avoid SELECT *
• Add LIMIT clauses to prevent accidentally fetching millions of rows
• Monitor slow queries in Sentry to identify optimization opportunities

Performance Comparison

Here’s the complete before/after:

Metric	Before (N+1)	After (JOINs)	Improvement
API Response Time (p95)	2500ms	150ms	94% faster
Database Queries	50+ queries	1 query	99% reduction
Frontend Skeleton Duration	2500ms	150ms	94% faster
User Experience	Terrible 😞	Fast ✅	🎉

Key Takeaways

• Frontend fixes (skeleton screens) improve perceived performance, not actual performance
• Distributed tracing reveals the complete request flow from frontend to database
• Traffic simulation reveals performance problems that don’t appear under light load
• Sentry automatically detects N+1 queries - no manual instrumentation required!
• N+1 queries appear as a “comb” pattern in trace waterfalls - look for repeating database spans
• SQL JOINs can reduce 100+ queries to a single query (99% reduction)
• Database indexes provide additional performance gains (2-3x faster queries)
• Monitor query performance in Sentry to identify optimization opportunities

What’s Next?

We’ve instrumented Sentry, fixed Web Vitals issues, and optimized backend performance. In the next module, we’ll use Session Replay and Logs to debug complex user experience issues and understand exactly what users are experiencing.

Production Checklist

Before deploying these changes:

• N+1 queries identified and fixed with JOINs
• Performance improvements verified in Sentry (> 90% faster)
• Traffic simulation run successfully with no errors
• Distributed tracing shows single optimized query instead of 100+
• Frontend skeleton screen duration improved significantly
• Consider adding database indexes for further optimization

---

Next up: We’ll use Session Replay and Logs to watch real user sessions and debug issues we can’t see in performance data alone.