quality_app/DEPLOYMENT_QUICK_REFERENCE.md

# Quick Reference - Connection Pooling & Logging

## ✅ What Was Fixed

**Problem:** Database timeout after 20-30 minutes on fgscan page  
**Solution:** DBUtils connection pooling + comprehensive logging  
**Result:** Max 20 connections, proper resource cleanup, full operation visibility

---

## 📊 Configuration Summary

### Connection Pool
```
Maximum Connections: 20
Minimum Cached: 3
Maximum Cached: 10
Max Shared: 5
Blocking: True
Health Check: On-demand ping
```

### Log Files
```
/srv/quality_app/py_app/logs/
├── application_YYYYMMDD.log    - All DEBUG+ events
├── errors_YYYYMMDD.log         - ERROR+ events only
├── database_YYYYMMDD.log       - DB operations
├── routes_YYYYMMDD.log         - HTTP routes + login attempts
└── settings_YYYYMMDD.log       - Permission checks
```

### Docker Configuration
```
Data Root: /srv/docker
Old Root: /var/lib/docker (was 48% full)
Available Space: 209GB in /srv
```

---

## 🔍 How to Monitor

### View Live Logs
```bash
# Application logs
tail -f /srv/quality_app/py_app/logs/application_*.log

# Error logs
tail -f /srv/quality_app/py_app/logs/errors_*.log

# Database operations
tail -f /srv/quality_app/py_app/logs/database_*.log

# Container logs
docker logs -f quality-app
```

### Check Container Status
```bash
# List containers
docker ps

# Check Docker info
docker info | grep "Docker Root Dir"

# Check resource usage
docker stats quality-app

# Inspect app container
docker inspect quality-app
```

### Verify Connection Pool
Look for these log patterns:
```
✅ Log message shows: "Database connection pool initialized successfully (max 20 connections)"
✅ Every database operation shows: "Acquiring database connection from pool"
✅ After operation: "Database connection closed"
✅ No "pool initialization failed" errors
```

---

## 🧪 Testing the Fix

### Test 1: Login with Logging
```bash
curl -X POST http://localhost:8781/ -d "username=superadmin&password=superadmin123"
# Check routes_YYYYMMDD.log for login attempt entry
```

### Test 2: Extended Session (User Testing)
1. Login to application
2. Navigate to fgscan page
3. Submit data multiple times over 30+ minutes
4. Verify:
   - No timeout errors
   - Data saves correctly
   - Application remains responsive
   - No connection errors in logs

### Test 3: Monitor Logs
```bash
# In terminal 1 - watch logs
tail -f /srv/quality_app/py_app/logs/application_*.log

# In terminal 2 - generate traffic
for i in {1..10}; do curl -s http://localhost:8781/ > /dev/null; sleep 5; done

# Verify: Should see multiple connection acquire/release cycles
```

---

## 🚨 Troubleshooting

### No logs being written
**Check:**
- `ls -la /srv/quality_app/py_app/logs/` - files exist?
- `docker exec quality-app ls -la /app/logs/` - inside container?
- `docker logs quality-app` - any permission errors?

### Connection pool errors
**Check logs for:**
- `charset' is an invalid keyword argument` → Fixed in db_pool.py line 84
- `Failed to get connection from pool` → Database unreachable
- `pool initialization failed` → Config file issue

### Docker disk space errors
**Check:**
```bash
df -h /srv           # Should have 209GB available
df -h /             # Should no longer be 48% full
docker system df    # Show Docker space usage
```

### Application not starting
**Check:**
```bash
docker logs quality-app            # Full startup output
docker inspect quality-app         # Container health
docker compose ps                  # Service status
```

---

## 📈 Expected Behavior After Fix

### Before Pooling
- Random timeout errors after 20-30 minutes
- New database connection per operation
- Unlimited connections accumulating
- MariaDB max_connections (150) reached
- Page becomes unresponsive
- Data save failures

### After Pooling
- Stable performance indefinitely
- Connection reuse from pool
- Max 20 connections always
- No connection exhaustion
- Page remains responsive
- Data saves reliably
- Full operational logging

---

## 🔧 Key Files Modified

| File | Change | Impact |
|------|--------|--------|
| app/db_pool.py | NEW - Connection pool | Eliminates connection exhaustion |
| app/logging_config.py | NEW - Logging setup | Full operation visibility |
| app/routes.py | Added logging + context mgr | Route-level operation tracking |
| app/settings.py | Added logging + context mgr | Permission check logging |
| app/__init__.py | Init logging first | Proper initialization order |
| requirements.txt | Added DBUtils==3.1.2 | Connection pooling library |
| /etc/docker/daemon.json | NEW - data-root=/srv/docker | 209GB available disk space |

---

## 📞 Contact Points for Issues

1. **Application Logs:** `/srv/quality_app/py_app/logs/application_*.log`
2. **Error Logs:** `/srv/quality_app/py_app/logs/errors_*.log`  
3. **Docker Status:** `docker ps`, `docker stats`
4. **Container Logs:** `docker logs quality-app`

---

## ✨ Success Indicators

After deploying, you should see:

✅ Application responds consistently (no timeouts)  
✅ Logs show "Successfully obtained connection from pool"  
✅ Docker root is at /srv/docker  
✅ /srv/docker has 209GB available  
✅ No connection exhaustion errors  
✅ Logs show complete operation lifecycle  

---

**Deployed:** January 22, 2026  
**Status:** ✅ Production Ready