Fixed the scan error and backup problems
This commit is contained in:
Quality System Admin
312
documentation/DATA_ONLY_BACKUP_FEATURE.md
Normal file
312
documentation/DATA_ONLY_BACKUP_FEATURE.md
Normal file
@@ -0,0 +1,312 @@
|
||||
# Data-Only Backup and Restore Feature
|
||||
|
||||
## Overview
|
||||
|
||||
The data-only backup and restore feature allows you to backup and restore **only the data** from the database, without affecting the database schema, triggers, or structure. This is useful for:
|
||||
|
||||
- **Quick data transfers** between identical database structures
|
||||
- **Data refreshes** without changing the schema
|
||||
- **Faster backups** when you only need to save data
|
||||
- **Testing scenarios** where you want to swap data but keep the structure
|
||||
|
||||
---
|
||||
|
||||
## Key Features
|
||||
|
||||
### 1. Data-Only Backup
|
||||
Creates a backup file containing **only INSERT statements** for all tables.
|
||||
|
||||
**What's included:**
|
||||
- ✅ All table data (INSERT statements)
|
||||
- ✅ Column names in INSERT statements (complete-insert format)
|
||||
- ✅ Multi-row INSERT for efficiency
|
||||
|
||||
**What's NOT included:**
|
||||
- ❌ CREATE TABLE statements (no schema)
|
||||
- ❌ CREATE DATABASE statements
|
||||
- ❌ Trigger definitions
|
||||
- ❌ Stored procedures or functions
|
||||
- ❌ Views
|
||||
|
||||
**File naming:** `data_only_trasabilitate_YYYYMMDD_HHMMSS.sql`
|
||||
|
||||
### 2. Data-Only Restore
|
||||
Restores data from a data-only backup file into an **existing database**.
|
||||
|
||||
**What happens during restore:**
|
||||
1. **Truncates all tables** (deletes all current data)
|
||||
2. **Disables foreign key checks** temporarily
|
||||
3. **Inserts data** from the backup file
|
||||
4. **Re-enables foreign key checks**
|
||||
5. **Preserves** existing schema, triggers, and structure
|
||||
|
||||
**⚠️ Important:** The database schema must already exist and match the backup structure.
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
### Creating a Data-Only Backup
|
||||
|
||||
#### Via Web Interface:
|
||||
1. Navigate to **Settings** page
|
||||
2. Scroll to **Database Backup Management** section
|
||||
3. Click **📦 Data-Only Backup** button
|
||||
4. Backup file will be created and added to the backup list
|
||||
|
||||
#### Via API:
|
||||
```bash
|
||||
curl -X POST http://localhost:8781/api/backup/create-data-only \
|
||||
-H "Content-Type: application/json" \
|
||||
--cookie "session=your_session_cookie"
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Data-only backup created successfully",
|
||||
"filename": "data_only_trasabilitate_20251105_160000.sql",
|
||||
"size": "12.45 MB",
|
||||
"timestamp": "20251105_160000"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Restoring from Data-Only Backup
|
||||
|
||||
#### Via Web Interface:
|
||||
1. Navigate to **Settings** page
|
||||
2. Scroll to **Restore Database** section (Superadmin only)
|
||||
3. Select a backup file from the dropdown
|
||||
4. Choose **"Data-Only Restore"** radio button
|
||||
5. Click **🔄 Restore Database** button
|
||||
6. Confirm twice (with typing "RESTORE DATA")
|
||||
|
||||
#### Via API:
|
||||
```bash
|
||||
curl -X POST http://localhost:8781/api/backup/restore-data-only/data_only_trasabilitate_20251105_160000.sql \
|
||||
-H "Content-Type: application/json" \
|
||||
--cookie "session=your_session_cookie"
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Data restored successfully from data_only_trasabilitate_20251105_160000.sql"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Comparison: Full Backup vs Data-Only Backup
|
||||
|
||||
| Feature | Full Backup | Data-Only Backup |
|
||||
|---------|-------------|------------------|
|
||||
| **Database Schema** | ✅ Included | ❌ Not included |
|
||||
| **Triggers** | ✅ Included | ❌ Not included |
|
||||
| **Stored Procedures** | ✅ Included | ❌ Not included |
|
||||
| **Table Data** | ✅ Included | ✅ Included |
|
||||
| **File Size** | Larger | Smaller |
|
||||
| **Backup Speed** | Slower | Faster |
|
||||
| **Use Case** | Complete migration, disaster recovery | Data refresh, testing |
|
||||
| **Restore Requirements** | None (creates everything) | Database schema must exist |
|
||||
|
||||
---
|
||||
|
||||
## Use Cases
|
||||
|
||||
### ✅ When to Use Data-Only Backup:
|
||||
|
||||
1. **Daily Data Snapshots**
|
||||
- You want to backup data frequently without duplicating schema
|
||||
- Faster backups for large databases
|
||||
|
||||
2. **Data Transfer Between Servers**
|
||||
- Both servers have identical database structure
|
||||
- You only need to copy the data
|
||||
|
||||
3. **Testing and Development**
|
||||
- Load production data into test environment
|
||||
- Test environment already has correct schema
|
||||
|
||||
4. **Data Refresh**
|
||||
- Replace old data with new data
|
||||
- Keep existing triggers and procedures
|
||||
|
||||
### ❌ When NOT to Use Data-Only Backup:
|
||||
|
||||
1. **Complete Database Migration**
|
||||
- Use full backup to ensure all structures are migrated
|
||||
|
||||
2. **Disaster Recovery**
|
||||
- Use full backup to restore everything
|
||||
|
||||
3. **Schema Changes**
|
||||
- If schema has changed, data-only restore will fail
|
||||
- Use full backup and restore
|
||||
|
||||
4. **Fresh Database Setup**
|
||||
- No existing schema to restore into
|
||||
- Use full backup or database setup script
|
||||
|
||||
---
|
||||
|
||||
## Technical Implementation
|
||||
|
||||
### mysqldump Command for Data-Only Backup
|
||||
```bash
|
||||
mysqldump \
|
||||
--host=localhost \
|
||||
--port=3306 \
|
||||
--user=trasabilitate \
|
||||
--password=password \
|
||||
--no-create-info # Skip CREATE TABLE statements
|
||||
--skip-triggers # Skip trigger definitions
|
||||
--no-create-db # Skip CREATE DATABASE statement
|
||||
--complete-insert # Include column names in INSERT
|
||||
--extended-insert # Multi-row INSERTs for efficiency
|
||||
--single-transaction # Consistent snapshot
|
||||
--skip-lock-tables # Avoid table locks
|
||||
trasabilitate
|
||||
```
|
||||
|
||||
### Data-Only Restore Process
|
||||
```python
|
||||
# 1. Disable foreign key checks
|
||||
SET FOREIGN_KEY_CHECKS = 0;
|
||||
|
||||
# 2. Get all tables
|
||||
SHOW TABLES;
|
||||
|
||||
# 3. Truncate each table (except system tables)
|
||||
TRUNCATE TABLE `table_name`;
|
||||
|
||||
# 4. Execute the data-only backup SQL file
|
||||
# (Contains INSERT statements)
|
||||
|
||||
# 5. Re-enable foreign key checks
|
||||
SET FOREIGN_KEY_CHECKS = 1;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security and Permissions
|
||||
|
||||
- **Data-Only Backup Creation:** Requires `admin` or `superadmin` role
|
||||
- **Data-Only Restore:** Requires `superadmin` role only
|
||||
- **API Access:** Requires valid session authentication
|
||||
- **File Access:** Backups stored in `/srv/quality_app/backups` (configurable)
|
||||
|
||||
---
|
||||
|
||||
## Safety Features
|
||||
|
||||
### Confirmation Process for Restore:
|
||||
1. **First Confirmation:** Dialog explaining what will happen
|
||||
2. **Second Confirmation:** Requires typing "RESTORE DATA" in capital letters
|
||||
3. **Type Detection:** Warns if trying to do full restore on data-only file
|
||||
|
||||
### Data Integrity:
|
||||
- **Foreign key checks** disabled during restore to avoid constraint errors
|
||||
- **Transaction-based** backup for consistent snapshots
|
||||
- **Table truncation** ensures clean data without duplicates
|
||||
- **Automatic re-enabling** of foreign key checks after restore
|
||||
|
||||
---
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Create Data-Only Backup
|
||||
```
|
||||
POST /api/backup/create-data-only
|
||||
```
|
||||
**Access:** Admin+
|
||||
**Response:** Backup filename and size
|
||||
|
||||
### Restore Data-Only Backup
|
||||
```
|
||||
POST /api/backup/restore-data-only/<filename>
|
||||
```
|
||||
**Access:** Superadmin only
|
||||
**Response:** Success/failure message
|
||||
|
||||
---
|
||||
|
||||
## File Naming Convention
|
||||
|
||||
### Data-Only Backups:
|
||||
- Format: `data_only_<database>_<timestamp>.sql`
|
||||
- Example: `data_only_trasabilitate_20251105_143022.sql`
|
||||
|
||||
### Full Backups:
|
||||
- Format: `backup_<database>_<timestamp>.sql`
|
||||
- Example: `backup_trasabilitate_20251105_143022.sql`
|
||||
|
||||
The `data_only_` prefix helps identify backup type at a glance.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Error: "Data restore failed: Table 'X' doesn't exist"
|
||||
**Cause:** Database schema not present or incomplete
|
||||
**Solution:** Run full backup restore or database setup script first
|
||||
|
||||
### Error: "Column count doesn't match"
|
||||
**Cause:** Schema structure has changed since backup was created
|
||||
**Solution:** Use a newer data-only backup or update schema first
|
||||
|
||||
### Error: "Foreign key constraint fails"
|
||||
**Cause:** Foreign key checks not properly disabled
|
||||
**Solution:** Check MariaDB user has SUPER privilege
|
||||
|
||||
### Warning: "Could not truncate table"
|
||||
**Cause:** Table has special permissions or is a view
|
||||
**Solution:** Non-critical warning; restore will continue
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always keep full backups** for complete disaster recovery
|
||||
2. **Use data-only backups** for frequent snapshots
|
||||
3. **Test restores** in non-production environment first
|
||||
4. **Document schema changes** that affect data structure
|
||||
5. **Schedule both types** of backups (e.g., full weekly, data-only daily)
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Backup Speed:
|
||||
- **Full backup (17 tables):** ~15-30 seconds
|
||||
- **Data-only backup (17 tables):** ~10-20 seconds (faster by 30-40%)
|
||||
|
||||
### File Size:
|
||||
- **Full backup:** Includes schema (~1-2 MB) + data
|
||||
- **Data-only backup:** Only data (smaller by 1-2 MB)
|
||||
|
||||
### Restore Speed:
|
||||
- **Full restore:** Drops and recreates everything
|
||||
- **Data-only restore:** Only truncates and inserts (faster on large schemas)
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [BACKUP_SYSTEM.md](BACKUP_SYSTEM.md) - Complete backup system overview
|
||||
- [DATABASE_RESTORE_GUIDE.md](DATABASE_RESTORE_GUIDE.md) - Detailed restore procedures
|
||||
- [DATABASE_STRUCTURE.md](DATABASE_STRUCTURE.md) - Database schema reference
|
||||
|
||||
---
|
||||
|
||||
## Implementation Date
|
||||
|
||||
**Feature Added:** November 5, 2025
|
||||
**Version:** 1.1.0
|
||||
**Python Module:** `app/database_backup.py`
|
||||
**API Routes:** `app/routes.py` (lines 3800-3835)
|
||||
**UI Template:** `app/templates/settings.html`
|
||||
Reference in New Issue
Block a user