# Production Startup Guide

## Overview
This guide covers starting, stopping, and managing the Quality Recticel application in production using the provided management scripts.

## Quick Start

### Start Application
```bash
cd /srv/quality_app/py_app
bash start_production.sh
```

### Stop Application
```bash
cd /srv/quality_app/py_app
bash stop_production.sh
```

### Check Status
```bash
cd /srv/quality_app/py_app
bash status_production.sh
```

## Management Scripts

### start_production.sh
Production startup script that launches the application using Gunicorn WSGI server.

**Features**:
- ✅ Validates prerequisites (virtual environment, Gunicorn)
- ✅ Tests database connection before starting
- ✅ Auto-detects project location (quality_app vs quality_recticel)
- ✅ Creates PID file for process management
- ✅ Starts Gunicorn in daemon mode (background)
- ✅ Displays comprehensive startup information

**Prerequisites Checked**:
1. Virtual environment exists (`../recticel`)
2. Gunicorn is installed
3. Database connection is working
4. No existing instance running

**Configuration**:
- **Workers**: CPU count × 2 + 1 (default: 9 workers)
- **Port**: 8781
- **Bind**: 0.0.0.0 (all interfaces)
- **Config**: gunicorn.conf.py
- **Timeout**: 1800 seconds (30 minutes)
- **Max Upload**: 10GB

**Output Example**:
```
🚀 Trasabilitate Application - Production Startup
==============================================

📋 Checking Prerequisites
----------------------------------------
✅ Virtual environment found
✅ Gunicorn is available
✅ Database connection verified

📋 Starting Production Server
----------------------------------------
Starting Gunicorn WSGI server...
Configuration: gunicorn.conf.py
Workers: 9
Binding to: 0.0.0.0:8781

✅ Application started successfully!

==============================================
🎉 PRODUCTION SERVER RUNNING
==============================================

📋 Server Information:
   • Process ID: 402172
   • Configuration: gunicorn.conf.py
   • Project: quality_app
   • Access Log: /srv/quality_app/logs/access.log
   • Error Log: /srv/quality_app/logs/error.log

🌐 Application URLs:
   • Local:   http://127.0.0.1:8781
   • Network: http://192.168.0.205:8781

👤 Default Login:
   • Username: superadmin
   • Password: superadmin123

🔧 Management Commands:
   • Stop server:    kill 402172 && rm ../run/trasabilitate.pid
   • View logs:      tail -f /srv/quality_app/logs/error.log
   • Monitor access: tail -f /srv/quality_app/logs/access.log
   • Server status:  ps -p 402172

⚠️  Server is running in daemon mode (background)
```

### stop_production.sh
Gracefully stops the running application.

**Features**:
- ✅ Reads PID from file
- ✅ Sends SIGTERM (graceful shutdown)
- ✅ Waits 3 seconds for graceful exit
- ✅ Falls back to SIGKILL if needed
- ✅ Cleans up PID file

**Process**:
1. Checks if PID file exists
2. Verifies process is running
3. Sends SIGTERM signal
4. Waits for graceful shutdown
5. Uses SIGKILL if process doesn't stop
6. Removes PID file

**Output Example**:
```
🛑 Trasabilitate Application - Production Stop
==============================================
Stopping Trasabilitate application (PID: 402172)...
✅ Application stopped successfully

✅ Trasabilitate application has been stopped
```

### status_production.sh
Displays current application status and useful information.

**Features**:
- ✅ Auto-detects project location
- ✅ Shows process information (CPU, memory, uptime)
- ✅ Tests web server connectivity
- ✅ Displays log file locations
- ✅ Provides quick command reference

**Output Example**:
```
📊 Trasabilitate Application - Status Check
==============================================
✅ Application is running (PID: 402172)

📋 Process Information:
   402172  1  3.3  0.5  00:58  gunicorn --config gunicorn.conf.py

🌐 Server Information:
   • Project:      quality_app
   • Listening on: 0.0.0.0:8781
   • Local URL:    http://127.0.0.1:8781
   • Network URL:  http://192.168.0.205:8781

📁 Log Files:
   • Access Log: /srv/quality_app/logs/access.log
   • Error Log:  /srv/quality_app/logs/error.log

🔧 Quick Commands:
   • Stop server:     ./stop_production.sh
   • Restart server:  ./stop_production.sh && ./start_production.sh
   • View error log:  tail -f /srv/quality_app/logs/error.log
   • View access log: tail -f /srv/quality_app/logs/access.log

🌐 Connection Test:
✅ Web server is responding
```

## File Locations

### Script Locations
```
/srv/quality_app/py_app/
├── start_production.sh     # Start the application
├── stop_production.sh      # Stop the application
├── status_production.sh    # Check status
├── gunicorn.conf.py        # Gunicorn configuration
├── wsgi.py                 # WSGI entry point
└── run.py                  # Flask application entry
```

### Runtime Files
```
/srv/quality_app/
├── py_app/
│   └── run/
│       └── trasabilitate.pid    # Process ID file
├── logs/
│   ├── access.log               # Access logs
│   └── error.log                # Error logs
└── backups/                     # Database backups
```

### Virtual Environment
```
/srv/quality_recticel/recticel/   # Shared virtual environment
```

## Log Monitoring

### View Real-Time Logs

**Error Log** (application errors, debugging):
```bash
tail -f /srv/quality_app/logs/error.log
```

**Access Log** (HTTP requests):
```bash
tail -f /srv/quality_app/logs/access.log
```

**Filter for Errors**:
```bash
grep ERROR /srv/quality_app/logs/error.log
grep "500\|404" /srv/quality_app/logs/access.log
```

### Log Rotation

Logs grow over time. To prevent disk space issues:

**Manual Rotation**:
```bash
# Backup current logs
mv /srv/quality_app/logs/error.log /srv/quality_app/logs/error.log.$(date +%Y%m%d)
mv /srv/quality_app/logs/access.log /srv/quality_app/logs/access.log.$(date +%Y%m%d)

# Restart to create new logs
cd /srv/quality_app/py_app
bash stop_production.sh && bash start_production.sh
```

**Setup Logrotate** (recommended):
```bash
sudo nano /etc/logrotate.d/trasabilitate
```

Add:
```
/srv/quality_app/logs/*.log {
    daily
    rotate 30
    compress
    delaycompress
    notifempty
    missingok
    create 0644 ske087 ske087
    postrotate
        kill -HUP `cat /srv/quality_app/py_app/run/trasabilitate.pid 2>/dev/null` 2>/dev/null || true
    endscript
}
```

## Process Management

### Check if Running
```bash
ps aux | grep gunicorn | grep trasabilitate
```

### Get Process ID
```bash
cat /srv/quality_app/py_app/run/trasabilitate.pid
```

### View Process Tree
```bash
pstree -p $(cat /srv/quality_app/py_app/run/trasabilitate.pid)
```

### Monitor Resources
```bash
# CPU and Memory usage
top -p $(cat /srv/quality_app/py_app/run/trasabilitate.pid)

# Detailed stats
ps -p $(cat /srv/quality_app/py_app/run/trasabilitate.pid) -o pid,ppid,cmd,%cpu,%mem,vsz,rss,etime
```

### Kill Process (Emergency)
```bash
# Graceful
kill $(cat /srv/quality_app/py_app/run/trasabilitate.pid)

# Force kill
kill -9 $(cat /srv/quality_app/py_app/run/trasabilitate.pid)

# Clean up PID file
rm /srv/quality_app/py_app/run/trasabilitate.pid
```

## Common Tasks

### Restart Application
```bash
cd /srv/quality_app/py_app
bash stop_production.sh && bash start_production.sh
```

### Deploy Code Changes
```bash
# 1. Stop application
cd /srv/quality_app/py_app
bash stop_production.sh

# 2. Pull latest code (if using git)
cd /srv/quality_app
git pull

# 3. Update dependencies if needed
source /srv/quality_recticel/recticel/bin/activate
pip install -r py_app/requirements.txt

# 4. Start application
cd py_app
bash start_production.sh
```

### Change Port or Workers

Edit `gunicorn.conf.py` or set environment variables:

```bash
# Temporary (current session)
export GUNICORN_BIND="0.0.0.0:8080"
export GUNICORN_WORKERS="16"
cd /srv/quality_app/py_app
bash start_production.sh

# Permanent (edit config file)
nano gunicorn.conf.py
# Change: bind = "0.0.0.0:8781"
# Restart application
```

### Update Configuration

**Database Settings**:
```bash
nano /srv/quality_app/py_app/instance/external_server.conf
# Restart required
```

**Application Settings**:
```bash
nano /srv/quality_app/py_app/app/__init__.py
# Restart required
```

## Troubleshooting

### Application Won't Start

**1. Check if already running**:
```bash
bash status_production.sh
```

**2. Check database connection**:
```bash
mysql -u trasabilitate -p -e "SELECT 1;"
```

**3. Check virtual environment**:
```bash
ls -l /srv/quality_recticel/recticel/bin/python3
```

**4. Check permissions**:
```bash
ls -l /srv/quality_app/py_app/*.sh
chmod +x /srv/quality_app/py_app/*.sh
```

**5. Check error logs**:
```bash
tail -100 /srv/quality_app/logs/error.log
```

### Application Crashes

**View crash logs**:
```bash
tail -100 /srv/quality_app/logs/error.log | grep -i "error\|exception\|traceback"
```

**Check system resources**:
```bash
df -h                    # Disk space
free -h                  # Memory
top                      # CPU usage
```

**Check for out of memory**:
```bash
dmesg | grep -i "out of memory"
```

### Workers Dying

Workers restart automatically after max_requests (1000). If workers crash frequently:

**1. Check error logs for exceptions**
**2. Increase worker timeout** (edit gunicorn.conf.py)
**3. Reduce number of workers**
**4. Check for memory leaks**

### Port Already in Use

```bash
# Find process using port 8781
sudo lsof -i :8781

# Kill the process
sudo kill -9 <PID>

# Or change port in gunicorn.conf.py
```

### Stale PID File

```bash
# Remove stale PID file
rm /srv/quality_app/py_app/run/trasabilitate.pid

# Start application
bash start_production.sh
```

## Performance Tuning

### Worker Configuration

**Calculate optimal workers**:
```
Workers = (2 × CPU cores) + 1
```

For 4-core CPU: 9 workers (default)
For 8-core CPU: 17 workers

Edit `gunicorn.conf.py`:
```python
workers = int(os.getenv("GUNICORN_WORKERS", "17"))
```

### Timeout Configuration

**For large database operations**:
```python
timeout = int(os.getenv("GUNICORN_TIMEOUT", "1800"))  # 30 minutes
```

**For normal operations**:
```python
timeout = int(os.getenv("GUNICORN_TIMEOUT", "120"))   # 2 minutes
```

### Memory Management

**Worker recycling**:
```python
max_requests = 1000           # Restart after 1000 requests
max_requests_jitter = 100     # Add randomness to prevent simultaneous restarts
```

### Connection Pooling

Configure in application code for better database performance.

## Security Considerations

### Change Default Credentials
```sql
-- Connect to database
mysql trasabilitate

-- Update superadmin password
UPDATE users SET password = '<hashed_password>' WHERE username = 'superadmin';
```

### Firewall Configuration
```bash
# Allow only from specific IPs
sudo ufw allow from 192.168.0.0/24 to any port 8781

# Or use reverse proxy (nginx/apache)
```

### SSL/HTTPS

Use a reverse proxy (nginx) for SSL:

```nginx
server {
    listen 443 ssl;
    server_name your-domain.com;
    
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    
    location / {
        proxy_pass http://127.0.0.1:8781;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
```

## Systemd Service (Optional)

For automatic startup on boot, create a systemd service:

**Create service file**:
```bash
sudo nano /etc/systemd/system/trasabilitate.service
```

**Service configuration**:
```ini
[Unit]
Description=Trasabilitate Quality Management Application
After=network.target mariadb.service

[Service]
Type=forking
User=ske087
Group=ske087
WorkingDirectory=/srv/quality_app/py_app
Environment="PATH=/srv/quality_recticel/recticel/bin:/usr/local/bin:/usr/bin:/bin"
ExecStart=/srv/quality_app/py_app/start_production.sh
ExecStop=/srv/quality_app/py_app/stop_production.sh
PIDFile=/srv/quality_app/py_app/run/trasabilitate.pid
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
```

**Enable and start**:
```bash
sudo systemctl daemon-reload
sudo systemctl enable trasabilitate
sudo systemctl start trasabilitate
sudo systemctl status trasabilitate
```

**Manage with systemctl**:
```bash
sudo systemctl start trasabilitate
sudo systemctl stop trasabilitate
sudo systemctl restart trasabilitate
sudo systemctl status trasabilitate
```

## Monitoring and Alerts

### Basic Health Check Script

Create `/srv/quality_app/py_app/healthcheck.sh`:
```bash
#!/bin/bash
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8781)

if [ "$RESPONSE" = "200" ] || [ "$RESPONSE" = "302" ]; then
    echo "OK: Application is running"
    exit 0
else
    echo "ERROR: Application not responding (HTTP $RESPONSE)"
    exit 1
fi
```

### Scheduled Health Checks (Cron)
```bash
crontab -e
# Add: Check every 5 minutes
*/5 * * * * /srv/quality_app/py_app/healthcheck.sh || /srv/quality_app/py_app/start_production.sh
```

## Summary

**Start Application**:
```bash
cd /srv/quality_app/py_app && bash start_production.sh
```

**Stop Application**:
```bash
cd /srv/quality_app/py_app && bash stop_production.sh
```

**Check Status**:
```bash
cd /srv/quality_app/py_app && bash status_production.sh
```

**View Logs**:
```bash
tail -f /srv/quality_app/logs/error.log
```

**Restart**:
```bash
cd /srv/quality_app/py_app && bash stop_production.sh && bash start_production.sh
```

For more information, see:
- [DATABASE_RESTORE_GUIDE.md](DATABASE_RESTORE_GUIDE.md) - Backup and restore procedures
- [DATABASE_BACKUP_GUIDE.md](DATABASE_BACKUP_GUIDE.md) - Backup management
- [DOCKER_DEPLOYMENT.md](../old%20code/DOCKER_DEPLOYMENT.md) - Docker deployment options

---

**Last Updated**: November 3, 2025  
**Application**: Quality Recticel Traceability System  
**Version**: 1.0.0