Fixed the scan error and backup problems

2025-11-05 21:25:02 +02:00
parent 9020f2c1cf
commit c91b7d0a4d
15 changed files with 4873 additions and 429 deletions
--- a/documentation/BACKUP_SCHEDULE_FEATURE.md
+++ b/documentation/BACKUP_SCHEDULE_FEATURE.md
@@ -0,0 +1,484 @@
+# Backup Schedule Feature - Complete Guide
+
+## Overview
+
+The backup schedule feature allows administrators to configure automated backups that run at specified times with customizable frequency. This ensures regular, consistent backups without manual intervention.
+
+**Added:** November 5, 2025
+**Version:** 1.1.0
+
+---
+
+## Key Features
+
+### 1. Automated Scheduling
+- **Daily Backups:** Run every day at specified time
+- **Weekly Backups:** Run once per week
+- **Monthly Backups:** Run once per month
+- **Custom Time:** Choose exact time (24-hour format)
+
+### 2. Backup Type Selection ✨ NEW
+- **Full Backup:** Complete database with schema, triggers, and data
+- **Data-Only Backup:** Only table data (faster, smaller files)
+
+### 3. Retention Management
+- **Automatic Cleanup:** Delete backups older than X days
+- **Configurable Period:** Keep backups from 1 to 365 days
+- **Smart Storage:** Prevents disk space issues
+
+### 4. Easy Management
+- **Enable/Disable:** Toggle scheduled backups on/off
+- **Visual Interface:** Clear, intuitive settings panel
+- **Status Tracking:** See current schedule at a glance
+
+---
+
+## Configuration Options
+
+### Schedule Settings
+
+| Setting | Options | Default | Description |
+|---------|---------|---------|-------------|
+| **Enabled** | On/Off | Off | Enable or disable scheduled backups |
+| **Time** | 00:00 - 23:59 | 02:00 | Time to run backup (24-hour format) |
+| **Frequency** | Daily, Weekly, Monthly | Daily | How often to run backup |
+| **Backup Type** | Full, Data-Only | Full | Type of backup to create |
+| **Retention** | 1-365 days | 30 | Days to keep old backups |
+
+---
+
+## Recommended Configurations
+
+### Configuration 1: Daily Data Snapshots
+**Best for:** Production environments with frequent data changes
+
+```json
+{
+  "enabled": true,
+  "time": "02:00",
+  "frequency": "daily",
+  "backup_type": "data-only",
+  "retention_days": 7
+}
+```
+
+**Why:**
+- ✅ Fast daily backups (data-only is 30-40% faster)
+- ✅ Smaller file sizes
+- ✅ 7-day retention keeps recent history without filling disk
+- ✅ Schema changes handled separately
+
+### Configuration 2: Weekly Full Backups
+**Best for:** Stable environments, comprehensive safety
+
+```json
+{
+  "enabled": true,
+  "time": "03:00",
+  "frequency": "weekly",
+  "backup_type": "full",
+  "retention_days": 60
+}
+```
+
+**Why:**
+- ✅ Complete database backup with schema and triggers
+- ✅ Less frequent (lower storage usage)
+- ✅ 60-day retention for long-term recovery
+- ✅ Safe for disaster recovery
+
+### Configuration 3: Hybrid Approach (Recommended)
+**Best for:** Most production environments
+
+**Schedule 1 - Daily Data:**
+```json
+{
+  "enabled": true,
+  "time": "02:00",
+  "frequency": "daily",
+  "backup_type": "data-only",
+  "retention_days": 7
+}
+```
+
+**Schedule 2 - Weekly Full (manual or separate scheduler):**
+- Run manual full backup every Sunday
+- Keep for 90 days
+
+**Why:**
+- ✅ Daily data snapshots for quick recovery
+- ✅ Weekly full backups for complete safety
+- ✅ Balanced storage usage
+- ✅ Multiple recovery points
+
+---
+
+## How to Configure
+
+### Via Web Interface
+
+1. **Navigate to Settings:**
+   - Log in as Admin or Superadmin
+   - Go to **Settings** page
+   - Scroll to **Database Backup Management** section
+
+2. **Configure Schedule:**
+   - Check **"Enable Scheduled Backups"** checkbox
+   - Set **Backup Time** (e.g., 02:00)
+   - Choose **Frequency** (Daily/Weekly/Monthly)
+   - Select **Backup Type:**
+     - **Full Backup** for complete safety
+     - **Data-Only Backup** for faster, smaller backups
+   - Set **Retention Days** (1-365)
+
+3. **Save Configuration:**
+   - Click **💾 Save Schedule** button
+   - Confirm settings in alert message
+
+### Via Configuration File
+
+**File Location:** `/srv/quality_app/backups/backup_schedule.json`
+
+**Example:**
+```json
+{
+  "enabled": true,
+  "time": "02:00",
+  "frequency": "daily",
+  "backup_type": "data-only",
+  "retention_days": 30
+}
+```
+
+**Note:** Changes take effect on next scheduled run.
+
+---
+
+## Technical Implementation
+
+### 1. Schedule Storage
+- **File:** `backup_schedule.json` in backups directory
+- **Format:** JSON
+- **Persistence:** Survives application restarts
+
+### 2. Backup Execution
+The schedule configuration is stored, but actual execution requires a cron job or scheduler:
+
+**Recommended: Use system cron**
+```bash
+# Edit crontab
+crontab -e
+
+# Add entry for 2 AM daily
+0 2 * * * cd /srv/quality_app/py_app && /srv/quality_recticel/recticel/bin/python3 -c "from app.database_backup import DatabaseBackupManager; from app import create_app; app = create_app(); app.app_context().push(); mgr = DatabaseBackupManager(); schedule = mgr.get_backup_schedule(); mgr.create_data_only_backup() if schedule['backup_type'] == 'data-only' else mgr.create_backup()"
+```
+
+**Alternative: APScheduler (application-level)**
+```python
+from apscheduler.schedulers.background import BackgroundScheduler
+
+scheduler = BackgroundScheduler()
+
+def scheduled_backup():
+    schedule = backup_manager.get_backup_schedule()
+    if schedule['enabled']:
+        if schedule['backup_type'] == 'data-only':
+            backup_manager.create_data_only_backup()
+        else:
+            backup_manager.create_backup()
+        backup_manager.cleanup_old_backups(schedule['retention_days'])
+
+# Schedule based on configuration
+scheduler.add_job(scheduled_backup, 'cron', hour=2, minute=0)
+scheduler.start()
+```
+
+### 3. Cleanup Process
+Automated cleanup runs after each backup:
+- Scans backup directory
+- Identifies files older than retention_days
+- Deletes old backups
+- Logs deletion activity
+
+---
+
+## Backup Type Comparison
+
+### Full Backup (Schema + Data + Triggers)
+
+**mysqldump command:**
+```bash
+mysqldump \
+  --single-transaction \
+  --skip-lock-tables \
+  --force \
+  --routines \
+  --triggers \        # ✅ Included
+  --events \
+  --add-drop-database \
+  --databases trasabilitate
+```
+
+**Typical size:** 1-2 MB (schema) + data size
+**Backup time:** ~15-30 seconds
+**Restore:** Complete replacement
+
+### Data-Only Backup
+
+**mysqldump command:**
+```bash
+mysqldump \
+  --no-create-info \      # ❌ Skip CREATE TABLE
+  --skip-triggers \       # ❌ Skip triggers
+  --no-create-db \        # ❌ Skip CREATE DATABASE
+  --complete-insert \
+  --extended-insert \
+  --single-transaction \
+  trasabilitate
+```
+
+**Typical size:** Data size only
+**Backup time:** ~10-20 seconds (30-40% faster)
+**Restore:** Data only (schema must exist)
+
+---
+
+## Understanding the UI
+
+### Schedule Form Fields
+
+```
+┌─────────────────────────────────────────────┐
+│ ☑ Enable Scheduled Backups                 │
+├─────────────────────────────────────────────┤
+│ Backup Time: [02:00]                        │
+│ Frequency: [Daily ▼]                        │
+│ Backup Type: [Full Backup ▼]               │
+│ Keep backups for: [30] days                │
+├─────────────────────────────────────────────┤
+│        [💾 Save Schedule]                   │
+└─────────────────────────────────────────────┘
+
+💡 Recommendation: Use Full Backup for weekly/
+   monthly schedules (complete safety), and
+   Data-Only for daily schedules (faster,
+   smaller files).
+```
+
+### Success Message Format
+When saving schedule:
+```
+✅ Backup schedule saved successfully
+
+Scheduled [Full/Data-Only] backups will run 
+[daily/weekly/monthly] at [HH:MM].
+```
+
+---
+
+## API Endpoints
+
+### Get Current Schedule
+```
+GET /api/backup/schedule
+```
+
+**Response:**
+```json
+{
+  "success": true,
+  "schedule": {
+    "enabled": true,
+    "time": "02:00",
+    "frequency": "daily",
+    "backup_type": "data-only",
+    "retention_days": 30
+  }
+}
+```
+
+### Save Schedule
+```
+POST /api/backup/schedule
+Content-Type: application/json
+
+{
+  "enabled": true,
+  "time": "02:00",
+  "frequency": "daily",
+  "backup_type": "data-only",
+  "retention_days": 30
+}
+```
+
+**Response:**
+```json
+{
+  "success": true,
+  "message": "Backup schedule saved successfully"
+}
+```
+
+---
+
+## Monitoring and Logs
+
+### Check Backup Files
+```bash
+ls -lh /srv/quality_app/backups/*.sql | tail -10
+```
+
+### Verify Schedule Configuration
+```bash
+cat /srv/quality_app/backups/backup_schedule.json
+```
+
+### Check Application Logs
+```bash
+tail -f /srv/quality_app/logs/error.log | grep -i backup
+```
+
+### Monitor Disk Usage
+```bash
+du -sh /srv/quality_app/backups/
+```
+
+---
+
+## Troubleshooting
+
+### Issue: Scheduled backups not running
+
+**Check 1:** Is schedule enabled?
+```bash
+cat /srv/quality_app/backups/backup_schedule.json | grep enabled
+```
+
+**Check 2:** Is cron job configured?
+```bash
+crontab -l | grep backup
+```
+
+**Check 3:** Are there permission issues?
+```bash
+ls -la /srv/quality_app/backups/
+```
+
+**Solution:** Ensure cron job exists and has proper permissions.
+
+---
+
+### Issue: Backup files growing too large
+
+**Check disk usage:**
+```bash
+du -sh /srv/quality_app/backups/
+ls -lh /srv/quality_app/backups/*.sql | wc -l
+```
+
+**Solutions:**
+1. Reduce retention_days (e.g., from 30 to 7)
+2. Use data-only backups (smaller files)
+3. Store old backups on external storage
+4. Compress backups: `gzip /srv/quality_app/backups/*.sql`
+
+---
+
+### Issue: Data-only restore fails
+
+**Error:** "Table doesn't exist"
+
+**Cause:** Database schema not present
+
+**Solution:** 
+1. Run full backup restore first, OR
+2. Ensure database structure exists via setup script
+
+---
+
+## Best Practices
+
+### ✅ DO:
+1. **Enable scheduled backups** - Automate for consistency
+2. **Use data-only for daily** - Faster, smaller files
+3. **Use full for weekly** - Complete safety net
+4. **Test restore regularly** - Verify backups work
+5. **Monitor disk space** - Prevent storage issues
+6. **Store off-site copies** - Disaster recovery
+7. **Adjust retention** - Balance safety vs. storage
+
+### ❌ DON'T:
+1. **Don't disable all backups** - Always have some backup
+2. **Don't set retention too low** - Keep at least 7 days
+3. **Don't ignore disk warnings** - Monitor storage
+4. **Don't forget to test restores** - Untested backups are useless
+5. **Don't rely only on scheduled** - Manual backups before major changes
+
+---
+
+## Security and Access
+
+### Required Roles
+- **View Schedule:** Admin, Superadmin
+- **Edit Schedule:** Admin, Superadmin
+- **Execute Manual Backup:** Admin, Superadmin
+- **Restore Database:** Superadmin only
+
+### File Permissions
+```bash
+# Backup directory
+drwxrwxr-x /srv/quality_app/backups/
+
+# Schedule file
+-rw-rw-r-- backup_schedule.json
+
+# Backup files
+-rw-rw-r-- *.sql
+```
+
+---
+
+## Migration Guide
+
+### Upgrading from Previous Version (without backup_type)
+
+**Automatic:** Schedule automatically gets `backup_type: "full"` on first load
+
+**Manual update:**
+```bash
+cd /srv/quality_app/backups/
+# Backup current schedule
+cp backup_schedule.json backup_schedule.json.bak
+
+# Add backup_type field
+cat backup_schedule.json | jq '. + {"backup_type": "full"}' > backup_schedule_new.json
+mv backup_schedule_new.json backup_schedule.json
+```
+
+---
+
+## Related Documentation
+
+- [DATA_ONLY_BACKUP_FEATURE.md](DATA_ONLY_BACKUP_FEATURE.md) - Data-only backup details
+- [BACKUP_SYSTEM.md](BACKUP_SYSTEM.md) - Complete backup system overview
+- [QUICK_BACKUP_REFERENCE.md](QUICK_BACKUP_REFERENCE.md) - Quick reference guide
+
+---
+
+## Future Enhancements
+
+### Planned Features:
+- [ ] Multiple schedules (daily data + weekly full)
+- [ ] Email notifications on backup completion
+- [ ] Backup to remote storage (S3, FTP)
+- [ ] Backup compression (gzip)
+- [ ] Backup encryption
+- [ ] Web-based backup browsing
+- [ ] Automatic restore testing
+
+---
+
+**Last Updated:** November 5, 2025
+**Module:** `app/database_backup.py`
+**UI Template:** `app/templates/settings.html`
+**Application:** Quality Recticel - Trasabilitate System