updated backups solution

2025-11-03 22:18:56 +02:00
parent 1ade0b5681
commit 9c19379810
17 changed files with 3105 additions and 50 deletions
--- a/documentation/PRODUCTION_STARTUP_GUIDE.md
+++ b/documentation/PRODUCTION_STARTUP_GUIDE.md
@@ -0,0 +1,618 @@
+# Production Startup Guide
+
+## Overview
+This guide covers starting, stopping, and managing the Quality Recticel application in production using the provided management scripts.
+
+## Quick Start
+
+### Start Application
+```bash
+cd /srv/quality_app/py_app
+bash start_production.sh
+```
+
+### Stop Application
+```bash
+cd /srv/quality_app/py_app
+bash stop_production.sh
+```
+
+### Check Status
+```bash
+cd /srv/quality_app/py_app
+bash status_production.sh
+```
+
+## Management Scripts
+
+### start_production.sh
+Production startup script that launches the application using Gunicorn WSGI server.
+
+**Features**:
+- ✅ Validates prerequisites (virtual environment, Gunicorn)
+- ✅ Tests database connection before starting
+- ✅ Auto-detects project location (quality_app vs quality_recticel)
+- ✅ Creates PID file for process management
+- ✅ Starts Gunicorn in daemon mode (background)
+- ✅ Displays comprehensive startup information
+
+**Prerequisites Checked**:
+1. Virtual environment exists (`../recticel`)
+2. Gunicorn is installed
+3. Database connection is working
+4. No existing instance running
+
+**Configuration**:
+- **Workers**: CPU count × 2 + 1 (default: 9 workers)
+- **Port**: 8781
+- **Bind**: 0.0.0.0 (all interfaces)
+- **Config**: gunicorn.conf.py
+- **Timeout**: 1800 seconds (30 minutes)
+- **Max Upload**: 10GB
+
+**Output Example**:
+```
+🚀 Trasabilitate Application - Production Startup
+==============================================
+
+📋 Checking Prerequisites
+----------------------------------------
+✅ Virtual environment found
+✅ Gunicorn is available
+✅ Database connection verified
+
+📋 Starting Production Server
+----------------------------------------
+Starting Gunicorn WSGI server...
+Configuration: gunicorn.conf.py
+Workers: 9
+Binding to: 0.0.0.0:8781
+
+✅ Application started successfully!
+
+==============================================
+🎉 PRODUCTION SERVER RUNNING
+==============================================
+
+📋 Server Information:
+   • Process ID: 402172
+   • Configuration: gunicorn.conf.py
+   • Project: quality_app
+   • Access Log: /srv/quality_app/logs/access.log
+   • Error Log: /srv/quality_app/logs/error.log
+
+🌐 Application URLs:
+   • Local:   http://127.0.0.1:8781
+   • Network: http://192.168.0.205:8781
+
+👤 Default Login:
+   • Username: superadmin
+   • Password: superadmin123
+
+🔧 Management Commands:
+   • Stop server:    kill 402172 && rm ../run/trasabilitate.pid
+   • View logs:      tail -f /srv/quality_app/logs/error.log
+   • Monitor access: tail -f /srv/quality_app/logs/access.log
+   • Server status:  ps -p 402172
+
+⚠️  Server is running in daemon mode (background)
+```
+
+### stop_production.sh
+Gracefully stops the running application.
+
+**Features**:
+- ✅ Reads PID from file
+- ✅ Sends SIGTERM (graceful shutdown)
+- ✅ Waits 3 seconds for graceful exit
+- ✅ Falls back to SIGKILL if needed
+- ✅ Cleans up PID file
+
+**Process**:
+1. Checks if PID file exists
+2. Verifies process is running
+3. Sends SIGTERM signal
+4. Waits for graceful shutdown
+5. Uses SIGKILL if process doesn't stop
+6. Removes PID file
+
+**Output Example**:
+```
+🛑 Trasabilitate Application - Production Stop
+==============================================
+Stopping Trasabilitate application (PID: 402172)...
+✅ Application stopped successfully
+
+✅ Trasabilitate application has been stopped
+```
+
+### status_production.sh
+Displays current application status and useful information.
+
+**Features**:
+- ✅ Auto-detects project location
+- ✅ Shows process information (CPU, memory, uptime)
+- ✅ Tests web server connectivity
+- ✅ Displays log file locations
+- ✅ Provides quick command reference
+
+**Output Example**:
+```
+📊 Trasabilitate Application - Status Check
+==============================================
+✅ Application is running (PID: 402172)
+
+📋 Process Information:
+   402172  1  3.3  0.5  00:58  gunicorn --config gunicorn.conf.py
+
+🌐 Server Information:
+   • Project:      quality_app
+   • Listening on: 0.0.0.0:8781
+   • Local URL:    http://127.0.0.1:8781
+   • Network URL:  http://192.168.0.205:8781
+
+📁 Log Files:
+   • Access Log: /srv/quality_app/logs/access.log
+   • Error Log:  /srv/quality_app/logs/error.log
+
+🔧 Quick Commands:
+   • Stop server:     ./stop_production.sh
+   • Restart server:  ./stop_production.sh && ./start_production.sh
+   • View error log:  tail -f /srv/quality_app/logs/error.log
+   • View access log: tail -f /srv/quality_app/logs/access.log
+
+🌐 Connection Test:
+✅ Web server is responding
+```
+
+## File Locations
+
+### Script Locations
+```
+/srv/quality_app/py_app/
+├── start_production.sh     # Start the application
+├── stop_production.sh      # Stop the application
+├── status_production.sh    # Check status
+├── gunicorn.conf.py        # Gunicorn configuration
+├── wsgi.py                 # WSGI entry point
+└── run.py                  # Flask application entry
+```
+
+### Runtime Files
+```
+/srv/quality_app/
+├── py_app/
+│   └── run/
+│       └── trasabilitate.pid    # Process ID file
+├── logs/
+│   ├── access.log               # Access logs
+│   └── error.log                # Error logs
+└── backups/                     # Database backups
+```
+
+### Virtual Environment
+```
+/srv/quality_recticel/recticel/   # Shared virtual environment
+```
+
+## Log Monitoring
+
+### View Real-Time Logs
+
+**Error Log** (application errors, debugging):
+```bash
+tail -f /srv/quality_app/logs/error.log
+```
+
+**Access Log** (HTTP requests):
+```bash
+tail -f /srv/quality_app/logs/access.log
+```
+
+**Filter for Errors**:
+```bash
+grep ERROR /srv/quality_app/logs/error.log
+grep "500\|404" /srv/quality_app/logs/access.log
+```
+
+### Log Rotation
+
+Logs grow over time. To prevent disk space issues:
+
+**Manual Rotation**:
+```bash
+# Backup current logs
+mv /srv/quality_app/logs/error.log /srv/quality_app/logs/error.log.$(date +%Y%m%d)
+mv /srv/quality_app/logs/access.log /srv/quality_app/logs/access.log.$(date +%Y%m%d)
+
+# Restart to create new logs
+cd /srv/quality_app/py_app
+bash stop_production.sh && bash start_production.sh
+```
+
+**Setup Logrotate** (recommended):
+```bash
+sudo nano /etc/logrotate.d/trasabilitate
+```
+
+Add:
+```
+/srv/quality_app/logs/*.log {
+    daily
+    rotate 30
+    compress
+    delaycompress
+    notifempty
+    missingok
+    create 0644 ske087 ske087
+    postrotate
+        kill -HUP `cat /srv/quality_app/py_app/run/trasabilitate.pid 2>/dev/null` 2>/dev/null || true
+    endscript
+}
+```
+
+## Process Management
+
+### Check if Running
+```bash
+ps aux | grep gunicorn | grep trasabilitate
+```
+
+### Get Process ID
+```bash
+cat /srv/quality_app/py_app/run/trasabilitate.pid
+```
+
+### View Process Tree
+```bash
+pstree -p $(cat /srv/quality_app/py_app/run/trasabilitate.pid)
+```
+
+### Monitor Resources
+```bash
+# CPU and Memory usage
+top -p $(cat /srv/quality_app/py_app/run/trasabilitate.pid)
+
+# Detailed stats
+ps -p $(cat /srv/quality_app/py_app/run/trasabilitate.pid) -o pid,ppid,cmd,%cpu,%mem,vsz,rss,etime
+```
+
+### Kill Process (Emergency)
+```bash
+# Graceful
+kill $(cat /srv/quality_app/py_app/run/trasabilitate.pid)
+
+# Force kill
+kill -9 $(cat /srv/quality_app/py_app/run/trasabilitate.pid)
+
+# Clean up PID file
+rm /srv/quality_app/py_app/run/trasabilitate.pid
+```
+
+## Common Tasks
+
+### Restart Application
+```bash
+cd /srv/quality_app/py_app
+bash stop_production.sh && bash start_production.sh
+```
+
+### Deploy Code Changes
+```bash
+# 1. Stop application
+cd /srv/quality_app/py_app
+bash stop_production.sh
+
+# 2. Pull latest code (if using git)
+cd /srv/quality_app
+git pull
+
+# 3. Update dependencies if needed
+source /srv/quality_recticel/recticel/bin/activate
+pip install -r py_app/requirements.txt
+
+# 4. Start application
+cd py_app
+bash start_production.sh
+```
+
+### Change Port or Workers
+
+Edit `gunicorn.conf.py` or set environment variables:
+
+```bash
+# Temporary (current session)
+export GUNICORN_BIND="0.0.0.0:8080"
+export GUNICORN_WORKERS="16"
+cd /srv/quality_app/py_app
+bash start_production.sh
+
+# Permanent (edit config file)
+nano gunicorn.conf.py
+# Change: bind = "0.0.0.0:8781"
+# Restart application
+```
+
+### Update Configuration
+
+**Database Settings**:
+```bash
+nano /srv/quality_app/py_app/instance/external_server.conf
+# Restart required
+```
+
+**Application Settings**:
+```bash
+nano /srv/quality_app/py_app/app/__init__.py
+# Restart required
+```
+
+## Troubleshooting
+
+### Application Won't Start
+
+**1. Check if already running**:
+```bash
+bash status_production.sh
+```
+
+**2. Check database connection**:
+```bash
+mysql -u trasabilitate -p -e "SELECT 1;"
+```
+
+**3. Check virtual environment**:
+```bash
+ls -l /srv/quality_recticel/recticel/bin/python3
+```
+
+**4. Check permissions**:
+```bash
+ls -l /srv/quality_app/py_app/*.sh
+chmod +x /srv/quality_app/py_app/*.sh
+```
+
+**5. Check error logs**:
+```bash
+tail -100 /srv/quality_app/logs/error.log
+```
+
+### Application Crashes
+
+**View crash logs**:
+```bash
+tail -100 /srv/quality_app/logs/error.log | grep -i "error\|exception\|traceback"
+```
+
+**Check system resources**:
+```bash
+df -h                    # Disk space
+free -h                  # Memory
+top                      # CPU usage
+```
+
+**Check for out of memory**:
+```bash
+dmesg | grep -i "out of memory"
+```
+
+### Workers Dying
+
+Workers restart automatically after max_requests (1000). If workers crash frequently:
+
+**1. Check error logs for exceptions**
+**2. Increase worker timeout** (edit gunicorn.conf.py)
+**3. Reduce number of workers**
+**4. Check for memory leaks**
+
+### Port Already in Use
+
+```bash
+# Find process using port 8781
+sudo lsof -i :8781
+
+# Kill the process
+sudo kill -9 <PID>
+
+# Or change port in gunicorn.conf.py
+```
+
+### Stale PID File
+
+```bash
+# Remove stale PID file
+rm /srv/quality_app/py_app/run/trasabilitate.pid
+
+# Start application
+bash start_production.sh
+```
+
+## Performance Tuning
+
+### Worker Configuration
+
+**Calculate optimal workers**:
+```
+Workers = (2 × CPU cores) + 1
+```
+
+For 4-core CPU: 9 workers (default)
+For 8-core CPU: 17 workers
+
+Edit `gunicorn.conf.py`:
+```python
+workers = int(os.getenv("GUNICORN_WORKERS", "17"))
+```
+
+### Timeout Configuration
+
+**For large database operations**:
+```python
+timeout = int(os.getenv("GUNICORN_TIMEOUT", "1800"))  # 30 minutes
+```
+
+**For normal operations**:
+```python
+timeout = int(os.getenv("GUNICORN_TIMEOUT", "120"))   # 2 minutes
+```
+
+### Memory Management
+
+**Worker recycling**:
+```python
+max_requests = 1000           # Restart after 1000 requests
+max_requests_jitter = 100     # Add randomness to prevent simultaneous restarts
+```
+
+### Connection Pooling
+
+Configure in application code for better database performance.
+
+## Security Considerations
+
+### Change Default Credentials
+```sql
+-- Connect to database
+mysql trasabilitate
+
+-- Update superadmin password
+UPDATE users SET password = '<hashed_password>' WHERE username = 'superadmin';
+```
+
+### Firewall Configuration
+```bash
+# Allow only from specific IPs
+sudo ufw allow from 192.168.0.0/24 to any port 8781
+
+# Or use reverse proxy (nginx/apache)
+```
+
+### SSL/HTTPS
+
+Use a reverse proxy (nginx) for SSL:
+
+```nginx
+server {
+    listen 443 ssl;
+    server_name your-domain.com;
+    
+    ssl_certificate /path/to/cert.pem;
+    ssl_certificate_key /path/to/key.pem;
+    
+    location / {
+        proxy_pass http://127.0.0.1:8781;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+    }
+}
+```
+
+## Systemd Service (Optional)
+
+For automatic startup on boot, create a systemd service:
+
+**Create service file**:
+```bash
+sudo nano /etc/systemd/system/trasabilitate.service
+```
+
+**Service configuration**:
+```ini
+[Unit]
+Description=Trasabilitate Quality Management Application
+After=network.target mariadb.service
+
+[Service]
+Type=forking
+User=ske087
+Group=ske087
+WorkingDirectory=/srv/quality_app/py_app
+Environment="PATH=/srv/quality_recticel/recticel/bin:/usr/local/bin:/usr/bin:/bin"
+ExecStart=/srv/quality_app/py_app/start_production.sh
+ExecStop=/srv/quality_app/py_app/stop_production.sh
+PIDFile=/srv/quality_app/py_app/run/trasabilitate.pid
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=multi-user.target
+```
+
+**Enable and start**:
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable trasabilitate
+sudo systemctl start trasabilitate
+sudo systemctl status trasabilitate
+```
+
+**Manage with systemctl**:
+```bash
+sudo systemctl start trasabilitate
+sudo systemctl stop trasabilitate
+sudo systemctl restart trasabilitate
+sudo systemctl status trasabilitate
+```
+
+## Monitoring and Alerts
+
+### Basic Health Check Script
+
+Create `/srv/quality_app/py_app/healthcheck.sh`:
+```bash
+#!/bin/bash
+RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8781)
+
+if [ "$RESPONSE" = "200" ] || [ "$RESPONSE" = "302" ]; then
+    echo "OK: Application is running"
+    exit 0
+else
+    echo "ERROR: Application not responding (HTTP $RESPONSE)"
+    exit 1
+fi
+```
+
+### Scheduled Health Checks (Cron)
+```bash
+crontab -e
+# Add: Check every 5 minutes
+*/5 * * * * /srv/quality_app/py_app/healthcheck.sh || /srv/quality_app/py_app/start_production.sh
+```
+
+## Summary
+
+**Start Application**:
+```bash
+cd /srv/quality_app/py_app && bash start_production.sh
+```
+
+**Stop Application**:
+```bash
+cd /srv/quality_app/py_app && bash stop_production.sh
+```
+
+**Check Status**:
+```bash
+cd /srv/quality_app/py_app && bash status_production.sh
+```
+
+**View Logs**:
+```bash
+tail -f /srv/quality_app/logs/error.log
+```
+
+**Restart**:
+```bash
+cd /srv/quality_app/py_app && bash stop_production.sh && bash start_production.sh
+```
+
+For more information, see:
+- [DATABASE_RESTORE_GUIDE.md](DATABASE_RESTORE_GUIDE.md) - Backup and restore procedures
+- [DATABASE_BACKUP_GUIDE.md](DATABASE_BACKUP_GUIDE.md) - Backup management
+- [DOCKER_DEPLOYMENT.md](../old%20code/DOCKER_DEPLOYMENT.md) - Docker deployment options
+
+---
+
+**Last Updated**: November 3, 2025  
+**Application**: Quality Recticel Traceability System  
+**Version**: 1.0.0