updated structure and app
This commit is contained in:
Quality System Admin
342
DATABASE_DOCKER_SETUP.md
Normal file
342
DATABASE_DOCKER_SETUP.md
Normal file
@@ -0,0 +1,342 @@
|
||||
# Database Setup for Docker Deployment
|
||||
|
||||
## Overview
|
||||
The Recticel Quality Application uses a **dual-database approach**:
|
||||
1. **MariaDB** (Primary) - Production data, users, permissions, orders
|
||||
2. **SQLite** (Backup/Legacy) - Local user authentication fallback
|
||||
|
||||
## Database Configuration Flow
|
||||
|
||||
### 1. Docker Environment Variables → Database Connection
|
||||
|
||||
```
|
||||
Docker .env file
|
||||
↓
|
||||
docker-compose.yml (environment section)
|
||||
↓
|
||||
Docker container environment variables
|
||||
↓
|
||||
setup_complete_database.py (reads from env)
|
||||
↓
|
||||
external_server.conf file (generated)
|
||||
↓
|
||||
Application runtime (reads conf file)
|
||||
```
|
||||
|
||||
### 2. Environment Variables Used
|
||||
|
||||
| Variable | Default | Purpose | Used By |
|
||||
|----------|---------|---------|---------|
|
||||
| `DB_HOST` | `db` | Database server hostname | All DB operations |
|
||||
| `DB_PORT` | `3306` | MariaDB port | All DB operations |
|
||||
| `DB_NAME` | `trasabilitate` | Database name | All DB operations |
|
||||
| `DB_USER` | `trasabilitate` | Database username | All DB operations |
|
||||
| `DB_PASSWORD` | `Initial01!` | Database password | All DB operations |
|
||||
| `MYSQL_ROOT_PASSWORD` | `rootpassword` | MariaDB root password | DB initialization |
|
||||
| `INIT_DB` | `true` | Run schema setup | docker-entrypoint.sh |
|
||||
| `SEED_DB` | `true` | Create superadmin user | docker-entrypoint.sh |
|
||||
|
||||
### 3. Database Initialization Process
|
||||
|
||||
#### Phase 1: MariaDB Container Startup
|
||||
```bash
|
||||
# docker-compose.yml starts MariaDB container
|
||||
# init-db.sql runs automatically:
|
||||
1. CREATE DATABASE trasabilitate
|
||||
2. CREATE USER 'trasabilitate'@'%'
|
||||
3. GRANT ALL PRIVILEGES
|
||||
```
|
||||
|
||||
#### Phase 2: Application Container Waits
|
||||
```bash
|
||||
# docker-entrypoint.sh:
|
||||
1. Waits for MariaDB to be ready (health check)
|
||||
2. Tests connection with credentials
|
||||
3. Retries up to 60 times (2s intervals = 120s timeout)
|
||||
```
|
||||
|
||||
#### Phase 3: Configuration File Generation
|
||||
```bash
|
||||
# docker-entrypoint.sh creates:
|
||||
/app/instance/external_server.conf
|
||||
server_domain=db # From DB_HOST
|
||||
port=3306 # From DB_PORT
|
||||
database_name=trasabilitate # From DB_NAME
|
||||
username=trasabilitate # From DB_USER
|
||||
password=Initial01! # From DB_PASSWORD
|
||||
```
|
||||
|
||||
#### Phase 4: Schema Creation (if INIT_DB=true)
|
||||
```bash
|
||||
# setup_complete_database.py creates:
|
||||
- scan1_orders (quality scans - station 1)
|
||||
- scanfg_orders (quality scans - finished goods)
|
||||
- order_for_labels (production orders for labels)
|
||||
- warehouse_locations (warehouse management)
|
||||
- users (user authentication)
|
||||
- roles (user roles)
|
||||
- permissions (permission definitions)
|
||||
- role_permissions (role-permission mappings)
|
||||
- role_hierarchy (role inheritance)
|
||||
- permission_audit_log (permission change tracking)
|
||||
|
||||
# Also creates triggers:
|
||||
- increment_approved_quantity (auto-count approved items)
|
||||
- increment_approved_quantity_fg (auto-count finished goods)
|
||||
```
|
||||
|
||||
#### Phase 5: Data Seeding (if SEED_DB=true)
|
||||
```bash
|
||||
# seed.py creates:
|
||||
- Superadmin user (username: superadmin, password: superadmin123)
|
||||
|
||||
# setup_complete_database.py also creates:
|
||||
- Default permission set (35+ permissions)
|
||||
- Role hierarchy (7 roles: superadmin → admin → manager → workers)
|
||||
- Role-permission mappings
|
||||
```
|
||||
|
||||
### 4. How Application Connects to Database
|
||||
|
||||
#### A. Settings Module (app/settings.py)
|
||||
```python
|
||||
def get_external_db_connection():
|
||||
# Reads /app/instance/external_server.conf
|
||||
# Returns mariadb.connect() using conf values
|
||||
```
|
||||
|
||||
#### B. Other Modules (order_labels.py, print_module.py, warehouse.py)
|
||||
```python
|
||||
def get_db_connection():
|
||||
# Also reads external_server.conf
|
||||
# Each module manages its own connections
|
||||
```
|
||||
|
||||
#### C. SQLAlchemy (app/__init__.py)
|
||||
```python
|
||||
# Currently hardcoded to SQLite (NOT DOCKER-FRIENDLY!)
|
||||
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///users.db'
|
||||
```
|
||||
|
||||
## Current Issues & Recommendations
|
||||
|
||||
### ❌ Problem 1: Hardcoded SQLite in __init__.py
|
||||
**Issue:** `app/__init__.py` uses hardcoded SQLite connection
|
||||
```python
|
||||
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///users.db'
|
||||
```
|
||||
|
||||
**Impact:**
|
||||
- Not using environment variables
|
||||
- SQLAlchemy not connected to MariaDB
|
||||
- Inconsistent with external_server.conf approach
|
||||
|
||||
**Solution:** Update to read from environment:
|
||||
```python
|
||||
import os
|
||||
|
||||
def create_app():
|
||||
app = Flask(__name__)
|
||||
|
||||
# Database configuration from environment
|
||||
db_user = os.getenv('DB_USER', 'trasabilitate')
|
||||
db_pass = os.getenv('DB_PASSWORD', 'Initial01!')
|
||||
db_host = os.getenv('DB_HOST', 'localhost')
|
||||
db_port = os.getenv('DB_PORT', '3306')
|
||||
db_name = os.getenv('DB_NAME', 'trasabilitate')
|
||||
|
||||
# Use MariaDB/MySQL connection
|
||||
app.config['SQLALCHEMY_DATABASE_URI'] = (
|
||||
f'mysql+mariadb://{db_user}:{db_pass}@{db_host}:{db_port}/{db_name}'
|
||||
)
|
||||
```
|
||||
|
||||
### ❌ Problem 2: Dual Connection Methods
|
||||
**Issue:** Application uses two different connection methods:
|
||||
1. SQLAlchemy ORM (for User model)
|
||||
2. Direct mariadb.connect() (for everything else)
|
||||
|
||||
**Impact:**
|
||||
- Complexity in maintenance
|
||||
- Potential connection pool exhaustion
|
||||
- Inconsistent transaction handling
|
||||
|
||||
**Recommendation:** Standardize on one approach:
|
||||
- **Option A:** Use SQLAlchemy for everything (preferred)
|
||||
- **Option B:** Use direct mariadb connections everywhere
|
||||
|
||||
### ❌ Problem 3: external_server.conf Redundancy
|
||||
**Issue:** Configuration is duplicated:
|
||||
1. Environment variables → external_server.conf
|
||||
2. Application reads external_server.conf
|
||||
|
||||
**Impact:**
|
||||
- Unnecessary file I/O
|
||||
- Potential sync issues
|
||||
- Not 12-factor app compliant
|
||||
|
||||
**Recommendation:** Read directly from environment variables
|
||||
|
||||
## Docker Deployment Database Schema
|
||||
|
||||
### MariaDB Container Configuration
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
db:
|
||||
image: mariadb:11.3
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: rootpassword
|
||||
MYSQL_DATABASE: trasabilitate
|
||||
MYSQL_USER: trasabilitate
|
||||
MYSQL_PASSWORD: Initial01!
|
||||
volumes:
|
||||
- /srv/docker-test/mariadb:/var/lib/mysql # Persistent storage
|
||||
- ./init-db.sql:/docker-entrypoint-initdb.d/01-init.sql
|
||||
```
|
||||
|
||||
### Database Tables Created
|
||||
|
||||
| Table | Purpose | Records |
|
||||
|-------|---------|---------|
|
||||
| `scan1_orders` | Quality scan records (station 1) | 1000s |
|
||||
| `scanfg_orders` | Finished goods scan records | 1000s |
|
||||
| `order_for_labels` | Production orders needing labels | 100s |
|
||||
| `warehouse_locations` | Warehouse location codes | 50-200 |
|
||||
| `users` | User accounts | 10-50 |
|
||||
| `roles` | Role definitions | 7 |
|
||||
| `permissions` | Permission definitions | 35+ |
|
||||
| `role_permissions` | Role-permission mappings | 100+ |
|
||||
| `role_hierarchy` | Role inheritance tree | 7 |
|
||||
| `permission_audit_log` | Permission change audit trail | Growing |
|
||||
|
||||
### Default Users & Roles
|
||||
|
||||
**Superadmin User:**
|
||||
- Username: `superadmin`
|
||||
- Password: `superadmin123`
|
||||
- Role: `superadmin`
|
||||
- Access: Full system access
|
||||
|
||||
**Role Hierarchy:**
|
||||
```
|
||||
superadmin (level 1)
|
||||
└─ admin (level 2)
|
||||
└─ manager (level 3)
|
||||
├─ quality_manager (level 4)
|
||||
│ └─ quality_worker (level 5)
|
||||
└─ warehouse_manager (level 4)
|
||||
└─ warehouse_worker (level 5)
|
||||
```
|
||||
|
||||
## Production Deployment Checklist
|
||||
|
||||
- [ ] Change `MYSQL_ROOT_PASSWORD` from default
|
||||
- [ ] Change `DB_PASSWORD` from default (Initial01!)
|
||||
- [ ] Change superadmin password from default (superadmin123)
|
||||
- [ ] Set `INIT_DB=false` after first deployment
|
||||
- [ ] Set `SEED_DB=false` after first deployment
|
||||
- [ ] Set strong `SECRET_KEY` in environment
|
||||
- [ ] Backup MariaDB data directory regularly
|
||||
- [ ] Enable MariaDB binary logging for point-in-time recovery
|
||||
- [ ] Configure proper `DB_MAX_RETRIES` and `DB_RETRY_INTERVAL`
|
||||
- [ ] Monitor database connections and performance
|
||||
- [ ] Set up database user with minimal required privileges
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Database Connection Failed
|
||||
```bash
|
||||
# Check if MariaDB container is running
|
||||
docker-compose ps
|
||||
|
||||
# Check MariaDB logs
|
||||
docker-compose logs db
|
||||
|
||||
# Test connection from app container
|
||||
docker-compose exec web python3 -c "
|
||||
import mariadb
|
||||
conn = mariadb.connect(
|
||||
user='trasabilitate',
|
||||
password='Initial01!',
|
||||
host='db',
|
||||
port=3306,
|
||||
database='trasabilitate'
|
||||
)
|
||||
print('Connection successful!')
|
||||
"
|
||||
```
|
||||
|
||||
### Tables Not Created
|
||||
```bash
|
||||
# Run setup script manually
|
||||
docker-compose exec web python3 /app/app/db_create_scripts/setup_complete_database.py
|
||||
|
||||
# Check tables
|
||||
docker-compose exec db mysql -utrasabilitate -pInitial01! trasabilitate -e "SHOW TABLES;"
|
||||
```
|
||||
|
||||
### external_server.conf Not Found
|
||||
```bash
|
||||
# Verify file exists
|
||||
docker-compose exec web cat /app/instance/external_server.conf
|
||||
|
||||
# Recreate if missing (entrypoint should do this automatically)
|
||||
docker-compose restart web
|
||||
```
|
||||
|
||||
## Migration from Non-Docker to Docker
|
||||
|
||||
If migrating from a non-Docker deployment:
|
||||
|
||||
1. **Backup existing MariaDB database:**
|
||||
```bash
|
||||
mysqldump -u trasabilitate -p trasabilitate > backup.sql
|
||||
```
|
||||
|
||||
2. **Update docker-compose.yml paths to existing data:**
|
||||
```yaml
|
||||
db:
|
||||
volumes:
|
||||
- /path/to/existing/mariadb:/var/lib/mysql
|
||||
```
|
||||
|
||||
3. **Or restore to new Docker MariaDB:**
|
||||
```bash
|
||||
docker-compose exec -T db mysql -utrasabilitate -pInitial01! trasabilitate < backup.sql
|
||||
```
|
||||
|
||||
4. **Verify data:**
|
||||
```bash
|
||||
docker-compose exec db mysql -utrasabilitate -pInitial01! trasabilitate -e "SELECT COUNT(*) FROM users;"
|
||||
```
|
||||
|
||||
## Environment Variable Examples
|
||||
|
||||
### Development (.env)
|
||||
```bash
|
||||
DB_HOST=db
|
||||
DB_PORT=3306
|
||||
DB_NAME=trasabilitate
|
||||
DB_USER=trasabilitate
|
||||
DB_PASSWORD=Initial01!
|
||||
MYSQL_ROOT_PASSWORD=rootpassword
|
||||
INIT_DB=true
|
||||
SEED_DB=true
|
||||
FLASK_ENV=development
|
||||
GUNICORN_LOG_LEVEL=debug
|
||||
```
|
||||
|
||||
### Production (.env)
|
||||
```bash
|
||||
DB_HOST=db
|
||||
DB_PORT=3306
|
||||
DB_NAME=trasabilitate
|
||||
DB_USER=trasabilitate
|
||||
DB_PASSWORD=SuperSecurePassword123!@#
|
||||
MYSQL_ROOT_PASSWORD=SuperSecureRootPass456!@#
|
||||
INIT_DB=false
|
||||
SEED_DB=false
|
||||
FLASK_ENV=production
|
||||
GUNICORN_LOG_LEVEL=info
|
||||
SECRET_KEY=your-super-secret-key-change-this
|
||||
```
|
||||
Reference in New Issue
Block a user