quality_app/documentation/DATABASE_RESTORE_GUIDE.md

Database Restore Guide

Overview

The database restore functionality allows superadmins to restore the entire database from a backup file. This is essential for:

• Server Migration: Moving the application to a new server
• Disaster Recovery: Recovering from data corruption or loss
• Testing/Development: Restoring production data to test environment
• Rollback: Reverting to a previous state after issues

⚠️ CRITICAL WARNINGS

Data Loss Risk

• ALL CURRENT DATA WILL BE PERMANENTLY DELETED
• The restore operation is IRREVERSIBLE
• Once started, it cannot be stopped
• No "undo" functionality exists

Downtime Requirements

• Users may experience brief downtime during restore
• All database connections will be terminated
• Active sessions may be invalidated
• Plan restores during maintenance windows

Access Requirements

• SUPERADMIN ACCESS ONLY
• No other role has restore permissions
• This is by design for safety

Large Database Support

Supported File Sizes

The backup system is optimized for databases of all sizes:

• ✅ Small databases (< 100MB): Full validation, fast operations
• ✅ Medium databases (100MB - 2GB): Partial validation (first 10MB), normal operations
• ✅ Large databases (2GB - 10GB): Basic validation only, longer operations
• ✅ Very large databases (> 10GB): Can be configured by increasing limits

Upload Limits

• Maximum upload size: 10GB
• Warning threshold: 1GB (user confirmation required)
• Timeout: 30 minutes for upload + validation + restore

Performance Estimates

Database Size	Backup Creation	Upload Time*	Validation	Restore Time
100MB	~5 seconds	~10 seconds	~1 second	~15 seconds
500MB	~15 seconds	~1 minute	~2 seconds	~45 seconds
1GB	~30 seconds	~2 minutes	~3 seconds	~2 minutes
5GB	~2-3 minutes	~10-15 minutes	~1 second	~10 minutes
10GB	~5-7 minutes	~25-35 minutes	~1 second	~20 minutes

*Upload times assume 100Mbps network connection

Smart Validation

The system intelligently adjusts validation based on file size:

Small Files (< 100MB):

• Full line-by-line validation
• Checks for users table, INSERT statements, database structure
• Detects suspicious commands

Medium Files (100MB - 2GB):

• Validates only first 10MB in detail
• Quick structure check
• Performance optimized (~1-3 seconds)

Large Files (2GB - 10GB):

• Basic validation only (file size, extension)
• Skips detailed content check for performance
• Validation completes in ~1 second
• Message: "Large backup file accepted - detailed validation skipped for performance"

Memory Efficiency

All backup operations use streaming - no memory concerns:

• ✅ Backup creation: mysqldump streams directly to disk
• ✅ File upload: Saved directly to disk (no RAM buffering)
• ✅ Restore: mysql reads from disk in chunks
• ✅ Memory usage: < 100MB regardless of database size

System Requirements

For 5GB Database:

• Disk space: 10GB free (2x database size)
• Memory: < 100MB (streaming operations)
• Network: 100Mbps or faster recommended
• Time: ~30 minutes total (upload + restore)

For 10GB Database:

• Disk space: 20GB free
• Memory: < 100MB
• Network: 1Gbps recommended
• Time: ~1 hour total

How to Restore Database

Step 1: Access Settings Page

1. Log in as superadmin
2. Navigate to Settings page
3. Scroll down to Database Backup Management section
4. Find the ⚠️ Restore Database section (orange warning box)

Step 2: Upload or Select Backup File

Option A: Upload External Backup

1. Click "📁 Choose File" in the Upload section
2. Select your .sql backup file (up to 10GB)
3. If file is > 1GB, confirm the upload warning
4. Click "⬆️ Upload File" button
5. Wait for upload and validation (shows progress)
6. File appears in restore dropdown once complete

Option B: Use Existing Backup

1. Skip upload if backup already exists on server
2. Proceed directly to dropdown selection

Step 3: Select Backup from Dropdown

1. Click the dropdown: "Select Backup to Restore"
2. Choose from available backup files
   • Files are listed with size and creation date
   • Example: backup_trasabilitate_20251103_212929.sql (318 KB - 2025-11-03 21:29:29)
   • Uploaded files: backup_uploaded_20251103_214500_mybackup.sql (5.2 GB - ...)
3. The Restore Database button will enable once selected

Step 4: Confirm Restore (Double Confirmation)

First Confirmation Dialog

⚠️ CRITICAL WARNING ⚠️

You are about to RESTORE the database from:
backup_trasabilitate_20251103_212929.sql

This will PERMANENTLY DELETE all current data and replace it with the backup data.

This action CANNOT be undone!

Do you want to continue?

• Click OK to proceed or Cancel to abort

Second Confirmation (Type-to-Confirm)

⚠️ FINAL CONFIRMATION ⚠️

Type "RESTORE" in capital letters to confirm you understand:
• All current database data will be PERMANENTLY DELETED
• This action is IRREVERSIBLE
• Users may experience downtime during restore

Type RESTORE to continue:

• Type exactly: RESTORE (all capitals)
• Any other text will cancel the operation

Step 4: Restore Process

1. Button changes to: "⏳ Restoring database... Please wait..."
2. Backend performs restore operation:
   • Drops existing database
   • Creates new empty database
   • Imports backup SQL file
   • Verifies restoration
3. On success:
   • Success message displays
   • Page automatically reloads
   • All data is now from the backup file

UI Features

Visual Safety Indicators

• Orange Warning Box: Highly visible restore section
• Warning Icons: ⚠️ symbols throughout
• Explicit Text: Clear warnings about data loss
• Color Coding: Orange (#ff9800) for danger

Dark Mode Support

• Restore section adapts to dark theme
• Warning colors remain visible in both modes
• Light mode: Light orange background (#fff3e0)
• Dark mode: Dark brown background (#3a2a1f) with orange text

Button States

• Disabled: Grey button when no backup selected
• Enabled: Red button (#ff5722) when backup selected
• Processing: Loading indicator during restore

Technical Implementation

API Endpoint

POST /api/backup/restore/<filename>

Access Control: @superadmin_only decorator

Parameters:

• filename: Name of backup file to restore (in URL path)

Response:

{
    "success": true,
    "message": "Database restored successfully from backup_trasabilitate_20251103_212929.sql"
}

Backend Process (DatabaseBackupManager.restore_backup)

def restore_backup(self, filename: str) -> dict:
    """
    Restore database from a backup file
    
    Process:
    1. Verify backup file exists
    2. Drop existing database
    3. Create new database
    4. Import SQL dump
    5. Grant permissions
    6. Verify restoration
    """

Commands Executed:

-- Drop existing database
DROP DATABASE IF EXISTS trasabilitate;

-- Create new database
CREATE DATABASE trasabilitate;

-- Import backup (via mysql command)
mysql trasabilitate < /srv/quality_app/backups/backup_trasabilitate_20251103_212929.sql

-- Grant permissions
GRANT ALL PRIVILEGES ON trasabilitate.* TO 'your_user'@'localhost';
FLUSH PRIVILEGES;

Security Features

1. Double Confirmation: Prevents accidental restores
2. Type-to-Confirm: Requires typing "RESTORE" exactly
3. Superadmin Only: No other roles can access
4. Audit Trail: All restores logged in error.log
5. Session Check: Requires valid superadmin session

Server Migration Procedure

Migrating to New Server

On Old Server:

1. Create Final Backup

   • Go to Settings → Database Backup Management
   • Click ⚡ Backup Now
   • Wait for backup to complete (see performance estimates above)
   • Download the backup file (⬇️ Download button)
   • Save file securely (e.g., backup_trasabilitate_20251103.sql)
   • Note: Large databases (5GB+) will take 5-10 minutes to backup

2. Stop Application (optional but recommended)

   cd /srv/quality_app/py_app
   bash stop_production.sh
   

On New Server:

1. Install Application

   • Clone repository
   • Set up Python environment
   • Install dependencies
   • Configure external_server.conf

2. Initialize Empty Database

   sudo mysql -e "CREATE DATABASE trasabilitate;"
   sudo mysql -e "GRANT ALL PRIVILEGES ON trasabilitate.* TO 'your_user'@'localhost';"
   

3. Transfer Backup File

   Option A: Direct Upload via UI (Recommended for files < 5GB)

   • Start application
   • Login as superadmin → Settings
   • Use "Upload Backup File" section
   • Select your backup file (up to 10GB supported)
   • System will validate and add to restore list automatically
   • Estimated time: 10-30 minutes for 5GB file on 100Mbps network

   Option B: Manual Copy (Faster for very large files)

   • Copy backup file directly to server: scp backup_file.sql user@newserver:/srv/quality_app/backups/
   • Or use external storage/USB drive
   • Ensure permissions: chmod 644 /srv/quality_app/backups/backup_*.sql
   • File appears in restore dropdown immediately

4. Start Application (if not already running)

   cd /srv/quality_app/py_app
   bash start_production.sh
   

5. Restore Database via UI

   • Log in as superadmin
   • Go to Settings → Database Backup Management
   • Upload Section: Upload file OR skip if already copied
   • Restore Section: Select backup from dropdown
   • Click Restore Database
   • Complete double-confirmation
   • Wait for restore to complete
   • Estimated time: 5-20 minutes for 5GB database

6. Verify Migration

   • Check that all users exist
   • Verify data integrity
   • Test all modules (Quality, Warehouse, Labels, Daily Mirror)
   • Confirm permissions are correct

Large Database Migration Tips

For Databases > 5GB:

1. ✅ Use Manual Copy (Option B) instead of upload - Much faster
2. ✅ Schedule migration during off-hours to avoid user impact
3. ✅ Expect 30-60 minutes total time for 10GB database
4. ✅ Ensure sufficient disk space (2x database size)
5. ✅ Monitor progress in logs: tail -f /srv/quality_app/logs/error.log
6. ✅ Keep old server running until verification complete

Network Transfer Time Examples:

• 5GB @ 100Mbps network: ~7 minutes via scp, ~15 minutes via browser upload
• 5GB @ 1Gbps network: ~40 seconds via scp, ~2 minutes via browser upload
• 10GB @ 100Mbps network: ~14 minutes via scp, ~30 minutes via browser upload

Alternative: Command-Line Restore

If UI is not available, restore manually:

# Stop application
cd /srv/quality_app/py_app
bash stop_production.sh

# Drop and recreate database
sudo mysql -e "DROP DATABASE IF EXISTS trasabilitate;"
sudo mysql -e "CREATE DATABASE trasabilitate;"

# Restore from backup
sudo mysql trasabilitate < /srv/quality_app/backups/backup_trasabilitate_20251103.sql

# Grant permissions
sudo mysql -e "GRANT ALL PRIVILEGES ON trasabilitate.* TO 'your_user'@'localhost';"
sudo mysql -e "FLUSH PRIVILEGES;"

# Restart application
bash start_production.sh

Troubleshooting

Error: "Backup file not found"

Cause: Selected backup file doesn't exist in backup directory

Solution:

# Check backup directory
ls -lh /srv/quality_app/backups/

# Verify file exists and is readable
ls -l /srv/quality_app/backups/backup_trasabilitate_*.sql

Error: "Permission denied"

Cause: Insufficient MySQL privileges

Solution:

# Grant all privileges to database user
sudo mysql -e "GRANT ALL PRIVILEGES ON *.* TO 'your_user'@'localhost';"
sudo mysql -e "FLUSH PRIVILEGES;"

Error: "Database connection failed"

Cause: MySQL server not running or wrong credentials

Solution:

# Check MySQL status
sudo systemctl status mariadb

# Verify credentials in external_server.conf
cat /srv/quality_app/py_app/instance/external_server.conf

# Test connection
mysql -u your_user -p -e "SELECT 1;"

Error: "Restore partially completed"

Cause: SQL syntax errors in backup file

Solution:

1. Check error logs:

   tail -f /srv/quality_app/logs/error.log
   

2. Try manual restore to see specific errors:

   sudo mysql trasabilitate < backup_file.sql
   

3. Fix issues in backup file if possible
4. Create new backup from source database

Application Won't Start After Restore

Cause: Database structure mismatch or missing tables

Solution:

# Verify all tables exist
mysql trasabilitate -e "SHOW TABLES;"

# Check for specific required tables
mysql trasabilitate -e "SELECT COUNT(*) FROM users;"

# If tables missing, restore from a known-good backup

Best Practices

Before Restoring

1. ✅ Create a current backup before restoring older one
2. ✅ Notify users of planned downtime
3. ✅ Test restore in development environment first
4. ✅ Verify backup integrity (download and check file)
5. ✅ Plan rollback strategy if restore fails

During Restore

1. ✅ Monitor logs in real-time:

   tail -f /srv/quality_app/logs/error.log
   

2. ✅ Don't interrupt the process
3. ✅ Keep backup window as short as possible

After Restore

1. ✅ Verify data integrity
2. ✅ Test all features (login, modules, reports)
3. ✅ Check user permissions are correct
4. ✅ Monitor application for errors
5. ✅ Document restore in change log

Related Documentation

• DATABASE_BACKUP_GUIDE.md - Creating backups
• DATABASE_DOCKER_SETUP.md - Database configuration
• DOCKER_DEPLOYMENT.md - Deployment procedures

Summary

The restore functionality provides a safe and reliable way to restore database backups for server migration and disaster recovery. The double-confirmation system prevents accidental data loss, while the UI provides clear visibility into available backups. Always create a current backup before restoring, and test the restore process in a non-production environment when possible.