quality_app/documentation/PRODUCTION_STARTUP_GUIDE.md

Production Startup Guide

Overview

This guide covers starting, stopping, and managing the Quality Recticel application in production using the provided management scripts.

Quick Start

Start Application

cd /srv/quality_app/py_app
bash start_production.sh

Stop Application

cd /srv/quality_app/py_app
bash stop_production.sh

Check Status

cd /srv/quality_app/py_app
bash status_production.sh

Management Scripts

start_production.sh

Production startup script that launches the application using Gunicorn WSGI server.

Features:

• ✅ Validates prerequisites (virtual environment, Gunicorn)
• ✅ Tests database connection before starting
• ✅ Auto-detects project location (quality_app vs quality_recticel)
• ✅ Creates PID file for process management
• ✅ Starts Gunicorn in daemon mode (background)
• ✅ Displays comprehensive startup information

Prerequisites Checked:

1. Virtual environment exists (../recticel)
2. Gunicorn is installed
3. Database connection is working
4. No existing instance running

Configuration:

• Workers: CPU count × 2 + 1 (default: 9 workers)
• Port: 8781
• Bind: 0.0.0.0 (all interfaces)
• Config: gunicorn.conf.py
• Timeout: 1800 seconds (30 minutes)
• Max Upload: 10GB

Output Example:

🚀 Trasabilitate Application - Production Startup
==============================================

📋 Checking Prerequisites
----------------------------------------
✅ Virtual environment found
✅ Gunicorn is available
✅ Database connection verified

📋 Starting Production Server
----------------------------------------
Starting Gunicorn WSGI server...
Configuration: gunicorn.conf.py
Workers: 9
Binding to: 0.0.0.0:8781

✅ Application started successfully!

==============================================
🎉 PRODUCTION SERVER RUNNING
==============================================

📋 Server Information:
   • Process ID: 402172
   • Configuration: gunicorn.conf.py
   • Project: quality_app
   • Access Log: /srv/quality_app/logs/access.log
   • Error Log: /srv/quality_app/logs/error.log

🌐 Application URLs:
   • Local:   http://127.0.0.1:8781
   • Network: http://192.168.0.205:8781

👤 Default Login:
   • Username: superadmin
   • Password: superadmin123

🔧 Management Commands:
   • Stop server:    kill 402172 && rm ../run/trasabilitate.pid
   • View logs:      tail -f /srv/quality_app/logs/error.log
   • Monitor access: tail -f /srv/quality_app/logs/access.log
   • Server status:  ps -p 402172

⚠️  Server is running in daemon mode (background)

stop_production.sh

Gracefully stops the running application.

Features:

• ✅ Reads PID from file
• ✅ Sends SIGTERM (graceful shutdown)
• ✅ Waits 3 seconds for graceful exit
• ✅ Falls back to SIGKILL if needed
• ✅ Cleans up PID file

Process:

1. Checks if PID file exists
2. Verifies process is running
3. Sends SIGTERM signal
4. Waits for graceful shutdown
5. Uses SIGKILL if process doesn't stop
6. Removes PID file

Output Example:

🛑 Trasabilitate Application - Production Stop
==============================================
Stopping Trasabilitate application (PID: 402172)...
✅ Application stopped successfully

✅ Trasabilitate application has been stopped

status_production.sh

Displays current application status and useful information.

Features:

• ✅ Auto-detects project location
• ✅ Shows process information (CPU, memory, uptime)
• ✅ Tests web server connectivity
• ✅ Displays log file locations
• ✅ Provides quick command reference

Output Example:

📊 Trasabilitate Application - Status Check
==============================================
✅ Application is running (PID: 402172)

📋 Process Information:
   402172  1  3.3  0.5  00:58  gunicorn --config gunicorn.conf.py

🌐 Server Information:
   • Project:      quality_app
   • Listening on: 0.0.0.0:8781
   • Local URL:    http://127.0.0.1:8781
   • Network URL:  http://192.168.0.205:8781

📁 Log Files:
   • Access Log: /srv/quality_app/logs/access.log
   • Error Log:  /srv/quality_app/logs/error.log

🔧 Quick Commands:
   • Stop server:     ./stop_production.sh
   • Restart server:  ./stop_production.sh && ./start_production.sh
   • View error log:  tail -f /srv/quality_app/logs/error.log
   • View access log: tail -f /srv/quality_app/logs/access.log

🌐 Connection Test:
✅ Web server is responding

File Locations

Script Locations

/srv/quality_app/py_app/
├── start_production.sh     # Start the application
├── stop_production.sh      # Stop the application
├── status_production.sh    # Check status
├── gunicorn.conf.py        # Gunicorn configuration
├── wsgi.py                 # WSGI entry point
└── run.py                  # Flask application entry

Runtime Files

/srv/quality_app/
├── py_app/
│   └── run/
│       └── trasabilitate.pid    # Process ID file
├── logs/
│   ├── access.log               # Access logs
│   └── error.log                # Error logs
└── backups/                     # Database backups

Virtual Environment

/srv/quality_recticel/recticel/   # Shared virtual environment

Log Monitoring

View Real-Time Logs

Error Log (application errors, debugging):

tail -f /srv/quality_app/logs/error.log

Access Log (HTTP requests):

tail -f /srv/quality_app/logs/access.log

Filter for Errors:

grep ERROR /srv/quality_app/logs/error.log
grep "500\|404" /srv/quality_app/logs/access.log

Log Rotation

Logs grow over time. To prevent disk space issues:

Manual Rotation:

# Backup current logs
mv /srv/quality_app/logs/error.log /srv/quality_app/logs/error.log.$(date +%Y%m%d)
mv /srv/quality_app/logs/access.log /srv/quality_app/logs/access.log.$(date +%Y%m%d)

# Restart to create new logs
cd /srv/quality_app/py_app
bash stop_production.sh && bash start_production.sh

Setup Logrotate (recommended):

sudo nano /etc/logrotate.d/trasabilitate

Add:

/srv/quality_app/logs/*.log {
    daily
    rotate 30
    compress
    delaycompress
    notifempty
    missingok
    create 0644 ske087 ske087
    postrotate
        kill -HUP `cat /srv/quality_app/py_app/run/trasabilitate.pid 2>/dev/null` 2>/dev/null || true
    endscript
}

Process Management

Check if Running

ps aux | grep gunicorn | grep trasabilitate

Get Process ID

cat /srv/quality_app/py_app/run/trasabilitate.pid

View Process Tree

pstree -p $(cat /srv/quality_app/py_app/run/trasabilitate.pid)

Monitor Resources

# CPU and Memory usage
top -p $(cat /srv/quality_app/py_app/run/trasabilitate.pid)

# Detailed stats
ps -p $(cat /srv/quality_app/py_app/run/trasabilitate.pid) -o pid,ppid,cmd,%cpu,%mem,vsz,rss,etime

Kill Process (Emergency)

# Graceful
kill $(cat /srv/quality_app/py_app/run/trasabilitate.pid)

# Force kill
kill -9 $(cat /srv/quality_app/py_app/run/trasabilitate.pid)

# Clean up PID file
rm /srv/quality_app/py_app/run/trasabilitate.pid

Common Tasks

Restart Application

cd /srv/quality_app/py_app
bash stop_production.sh && bash start_production.sh

Deploy Code Changes

# 1. Stop application
cd /srv/quality_app/py_app
bash stop_production.sh

# 2. Pull latest code (if using git)
cd /srv/quality_app
git pull

# 3. Update dependencies if needed
source /srv/quality_recticel/recticel/bin/activate
pip install -r py_app/requirements.txt

# 4. Start application
cd py_app
bash start_production.sh

Change Port or Workers

Edit gunicorn.conf.py or set environment variables:

# Temporary (current session)
export GUNICORN_BIND="0.0.0.0:8080"
export GUNICORN_WORKERS="16"
cd /srv/quality_app/py_app
bash start_production.sh

# Permanent (edit config file)
nano gunicorn.conf.py
# Change: bind = "0.0.0.0:8781"
# Restart application

Update Configuration

Database Settings:

nano /srv/quality_app/py_app/instance/external_server.conf
# Restart required

Application Settings:

nano /srv/quality_app/py_app/app/__init__.py
# Restart required

Troubleshooting

Application Won't Start

1. Check if already running:

bash status_production.sh

2. Check database connection:

mysql -u trasabilitate -p -e "SELECT 1;"

3. Check virtual environment:

ls -l /srv/quality_recticel/recticel/bin/python3

4. Check permissions:

ls -l /srv/quality_app/py_app/*.sh
chmod +x /srv/quality_app/py_app/*.sh

5. Check error logs:

tail -100 /srv/quality_app/logs/error.log

Application Crashes

View crash logs:

tail -100 /srv/quality_app/logs/error.log | grep -i "error\|exception\|traceback"

Check system resources:

df -h                    # Disk space
free -h                  # Memory
top                      # CPU usage

Check for out of memory:

dmesg | grep -i "out of memory"

Workers Dying

Workers restart automatically after max_requests (1000). If workers crash frequently:

1. Check error logs for exceptions 2. Increase worker timeout (edit gunicorn.conf.py) 3. Reduce number of workers 4. Check for memory leaks

Port Already in Use

# Find process using port 8781
sudo lsof -i :8781

# Kill the process
sudo kill -9 <PID>

# Or change port in gunicorn.conf.py

Stale PID File

# Remove stale PID file
rm /srv/quality_app/py_app/run/trasabilitate.pid

# Start application
bash start_production.sh

Performance Tuning

Worker Configuration

Calculate optimal workers:

Workers = (2 × CPU cores) + 1

For 4-core CPU: 9 workers (default) For 8-core CPU: 17 workers

Edit gunicorn.conf.py:

workers = int(os.getenv("GUNICORN_WORKERS", "17"))

Timeout Configuration

For large database operations:

timeout = int(os.getenv("GUNICORN_TIMEOUT", "1800"))  # 30 minutes

For normal operations:

timeout = int(os.getenv("GUNICORN_TIMEOUT", "120"))   # 2 minutes

Memory Management

Worker recycling:

max_requests = 1000           # Restart after 1000 requests
max_requests_jitter = 100     # Add randomness to prevent simultaneous restarts

Connection Pooling

Configure in application code for better database performance.

Security Considerations

Change Default Credentials

-- Connect to database
mysql trasabilitate

-- Update superadmin password
UPDATE users SET password = '<hashed_password>' WHERE username = 'superadmin';

Firewall Configuration

# Allow only from specific IPs
sudo ufw allow from 192.168.0.0/24 to any port 8781

# Or use reverse proxy (nginx/apache)

SSL/HTTPS

Use a reverse proxy (nginx) for SSL:

server {
    listen 443 ssl;
    server_name your-domain.com;
    
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    
    location / {
        proxy_pass http://127.0.0.1:8781;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Systemd Service (Optional)

For automatic startup on boot, create a systemd service:

Create service file:

sudo nano /etc/systemd/system/trasabilitate.service

Service configuration:

[Unit]
Description=Trasabilitate Quality Management Application
After=network.target mariadb.service

[Service]
Type=forking
User=ske087
Group=ske087
WorkingDirectory=/srv/quality_app/py_app
Environment="PATH=/srv/quality_recticel/recticel/bin:/usr/local/bin:/usr/bin:/bin"
ExecStart=/srv/quality_app/py_app/start_production.sh
ExecStop=/srv/quality_app/py_app/stop_production.sh
PIDFile=/srv/quality_app/py_app/run/trasabilitate.pid
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable trasabilitate
sudo systemctl start trasabilitate
sudo systemctl status trasabilitate

Manage with systemctl:

sudo systemctl start trasabilitate
sudo systemctl stop trasabilitate
sudo systemctl restart trasabilitate
sudo systemctl status trasabilitate

Monitoring and Alerts

Basic Health Check Script

Create /srv/quality_app/py_app/healthcheck.sh:

#!/bin/bash
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8781)

if [ "$RESPONSE" = "200" ] || [ "$RESPONSE" = "302" ]; then
    echo "OK: Application is running"
    exit 0
else
    echo "ERROR: Application not responding (HTTP $RESPONSE)"
    exit 1
fi

Scheduled Health Checks (Cron)

crontab -e
# Add: Check every 5 minutes
*/5 * * * * /srv/quality_app/py_app/healthcheck.sh || /srv/quality_app/py_app/start_production.sh

Summary

Start Application:

cd /srv/quality_app/py_app && bash start_production.sh

Stop Application:

cd /srv/quality_app/py_app && bash stop_production.sh

Check Status:

cd /srv/quality_app/py_app && bash status_production.sh

View Logs:

tail -f /srv/quality_app/logs/error.log

Restart:

cd /srv/quality_app/py_app && bash stop_production.sh && bash start_production.sh

For more information, see:

• DATABASE_RESTORE_GUIDE.md - Backup and restore procedures
• DATABASE_BACKUP_GUIDE.md - Backup management
• DOCKER_DEPLOYMENT.md - Docker deployment options

---

Last Updated: November 3, 2025
Application: Quality Recticel Traceability System
Version: 1.0.0