Installation

Installation#

This guide provides comprehensive instructions for setting up the Nopayloaddb project. Choose the installation method that best fits your needs:

Quick Start with Docker (Recommended): Get up and running in minutes
Manual Installation: For development or custom setups
Production Deployment: For production environments

Note

TL;DR - Quick Start: If you just want to try Nopayloaddb quickly, jump to Quick Start with Docker (Recommended).

Quick Start with Docker (Recommended)#

The fastest way to get Nopayloaddb running:

Prerequisites: - Docker and Docker Compose installed on your system

Steps:

Clone and Setup:

git clone https://github.com/BNLNPPS/nopayloaddb.git
cd nopayloaddb

Create Environment File:

cat > .env << 'EOF'
# Basic configuration for development
SECRET_KEY='development-secret-key-change-in-production'
DJANGO_LOGPATH='/tmp'

# Database configuration
POSTGRES_DB_W=nopayloaddb
POSTGRES_USER_W=npdb
POSTGRES_PASSWORD_W=password
POSTGRES_HOST_W=db
POSTGRES_PORT_W=5432
EOF

Start the Application:
```
docker-compose up --build
```
Access the Application:

API Endpoints: - API Documentation: http://localhost:8000/api/cdb_rest/ - Sample API call:
curl http://localhost:8000/api/cdb_rest/gt

That’s it! You now have Nopayloaddb running with a PostgreSQL database.

To stop the application:

docker-compose down

Manual Installation#

For developers who want more control or need to customize the setup.

Prerequisites#

System Requirements#

Required Software:

Python 3.8+ (Python 3.9+ recommended)
PostgreSQL 12+ (13+ recommended for better performance)
Git (for cloning the repository)

System Dependencies:

The following system packages are required for PostgreSQL connectivity:

sudo apt-get update
sudo apt-get install -y \
    python3-dev \
    libpq-dev \
    postgresql \
    postgresql-contrib \
    git

# CentOS/RHEL
sudo yum install -y \
    python3-devel \
    postgresql-devel \
    postgresql-server \
    postgresql-contrib \
    git

# Fedora
sudo dnf install -y \
    python3-devel \
    postgresql-devel \
    postgresql-server \
    postgresql-contrib \
    git

# Using Homebrew
brew install postgresql git

# Start PostgreSQL service
brew services start postgresql

Installation Steps#

Clone the Repository

git clone https://github.com/BNLNPPS/nopayloaddb.git
cd nopayloaddb

Create Virtual Environment

python3 -m venv venv

# Activate virtual environment
# Linux/macOS:
source venv/bin/activate

# Windows:
# venv\Scripts\activate

Tip

Always use a virtual environment to avoid conflicts with system packages.

Install Python Dependencies

pip install --upgrade pip
pip install -r requirements.txt

Database Setup

Create PostgreSQL Database and User:

# Connect to PostgreSQL as superuser
sudo -u postgres psql

-- Create database
CREATE DATABASE nopayloaddb_dev;

-- Create user with password
CREATE USER npdb_dev WITH PASSWORD 'secure_dev_password';

-- Grant privileges
GRANT ALL PRIVILEGES ON DATABASE nopayloaddb_dev TO npdb_dev;

-- Exit PostgreSQL
\q

Note

For production, use separate read/write users. See Production Deployment.

Environment Configuration

Create a .env file in the project root:

cat > .env << 'EOF'
# Security
SECRET_KEY='your-very-secure-secret-key-here'

# Logging
DJANGO_LOGPATH='/tmp'

# Write Database (Primary)
POSTGRES_DB_W=nopayloaddb_dev
POSTGRES_USER_W=npdb_dev
POSTGRES_PASSWORD_W=secure_dev_password
POSTGRES_HOST_W=localhost
POSTGRES_PORT_W=5432

# Read Replicas (Optional - can use same values as write DB for development)
POSTGRES_DB_R1=nopayloaddb_dev
POSTGRES_USER_R1=npdb_dev
POSTGRES_PASSWORD_R1=secure_dev_password
POSTGRES_HOST_R1=localhost
POSTGRES_PORT_R1=5432

POSTGRES_DB_R2=nopayloaddb_dev
POSTGRES_USER_R2=npdb_dev
POSTGRES_PASSWORD_R2=secure_dev_password
POSTGRES_HOST_R2=localhost
POSTGRES_PORT_R2=5432
EOF

Warning

Generate a secure SECRET_KEY: You can generate one using:

python -c "from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())"

Apply Database Migrations

# Load environment variables
source .env

# Apply migrations
python manage.py migrate

Create Superuser (Optional)
```
python manage.py createsuperuser
```
Run Development Server
```
python manage.py runserver
```
Access the application at http://127.0.0.1:8000/

Docker Compose Setup (Detailed)#

For a more robust development environment with persistent data and easier management.

Prerequisites#

Install Docker:

# Ubuntu/Debian
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

# Install Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/download/v2.17.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Setup Steps#

Clone Repository

git clone https://github.com/BNLNPPS/nopayloaddb.git
cd nopayloaddb

Configure Environment Variables

Create a comprehensive .env file:

cat > .env << 'EOF'
# Django Configuration
SECRET_KEY='your-docker-development-secret-key'
DJANGO_LOGPATH='/npdb/logs'
DEBUG=True

# Database Configuration
POSTGRES_DB_W=nopayloaddb
POSTGRES_USER_W=npdb
POSTGRES_PASSWORD_W=secure_password_123
POSTGRES_HOST_W=db
POSTGRES_PORT_W=5432

# Read replicas (using same DB for development)
POSTGRES_DB_R1=nopayloaddb
POSTGRES_USER_R1=npdb
POSTGRES_PASSWORD_R1=secure_password_123
POSTGRES_HOST_R1=db
POSTGRES_PORT_R1=5432

POSTGRES_DB_R2=nopayloaddb
POSTGRES_USER_R2=npdb
POSTGRES_PASSWORD_R2=secure_password_123
POSTGRES_HOST_R2=db
POSTGRES_PORT_R2=5432
EOF

Start Services

# Build and start in foreground
docker-compose up --build

# Or start in background (detached mode)
docker-compose up --build -d

Verify Installation

# Check running services
docker-compose ps

# Check logs
docker-compose logs webapp

# Test API endpoint
curl http://localhost:8000/api/cdb_rest/gt

Managing the Development Environment

# View logs in real-time
docker-compose logs -f webapp

# Execute commands in the webapp container
docker-compose exec webapp python manage.py shell

# Create a superuser
docker-compose exec webapp python manage.py createsuperuser

# Run migrations (if needed)
docker-compose exec webapp python manage.py migrate

# Stop services
docker-compose down

# Remove all data (caution!)
docker-compose down -v

Environment Variables Reference#

Complete reference for all supported environment variables:

Core Settings#

Variable	Description	Default
`SECRET_KEY`	Django secret key (required)	`'changetosomething'` (insecure)
`DEBUG`	Enable Django debug mode	`False`
`DJANGO_LOGPATH`	Path for Django log files	`'/var/log'`

Database Configuration#

Write Database (Primary):

Variable	Description	Default
`POSTGRES_DB_W`	Write database name	`'dbname'`
`POSTGRES_USER_W`	Write database user	`'login'`
`POSTGRES_PASSWORD_W`	Write database password	`'password'`
`POSTGRES_HOST_W`	Write database host	`'localhost'`
`POSTGRES_PORT_W`	Write database port	`'5432'`

Read Replicas (Optional):

Replace _W with _R1 or _R2 for read replica configuration.

Production Deployment#

Warning

This section is for production deployments only. Do not use these settings for development.

For production environments, additional security and performance considerations apply:

Security Checklist#

Never use DEBUG=True in production
Use a secure, randomly generated SECRET_KEY
Configure HTTPS/TLS encryption
Use separate database users for read/write operations
Set proper file permissions on configuration files
Use environment-specific secret management

Production Configuration Example#

# Production environment variables (store securely)
SECRET_KEY='your-production-secret-key-50-characters-long'
DEBUG=False
DJANGO_LOGPATH='/var/log/nopayloaddb'

# Production database (write)
POSTGRES_DB_W=nopayloaddb_prod
POSTGRES_USER_W=npdb_write
POSTGRES_PASSWORD_W='very-secure-write-password'
POSTGRES_HOST_W=db-primary.example.com
POSTGRES_PORT_W=5432

# Production read replicas
POSTGRES_DB_R1=nopayloaddb_prod
POSTGRES_USER_R1=npdb_read
POSTGRES_PASSWORD_R1='very-secure-read-password'
POSTGRES_HOST_R1=db-replica1.example.com
POSTGRES_PORT_R1=5432

Helm Charts for Production (Recommended)

For production deployments on Kubernetes/OpenShift, we recommend using the official Helm charts:

# Clone the official Helm charts
git clone https://github.com/BNLNPPS/nopayloaddb-charts.git
cd nopayloaddb-charts

# Choose your experiment configuration:
# For sPHENIX:
cp your-values.yaml npdbchart_sphenix/values.yaml
helm install sphenix-npdb npdbchart_sphenix/

# For Belle2:
cp your-values.yaml npdbchart_belle2_java/values.yaml
helm install belle2-npdb npdbchart_belle2_java/

The Helm charts include:

Pre-configured security settings
Database setup and migration jobs
Monitoring and health checks
Experiment-specific configurations
Load balancing and scaling options

For detailed production deployment instructions, see Deployment Guide.

Troubleshooting#

Common Issues and Solutions#

Database Connection Errors

django.db.utils.OperationalError: could not connect to server

Solutions:

Verify PostgreSQL is running:

# Linux/macOS
sudo systemctl status postgresql
# or
brew services list | grep postgresql

Check database credentials in your .env file
Ensure the database exists:
```
psql -h localhost -U postgres -l
```

Permission Denied on Log Directory

PermissionError: [Errno 13] Permission denied: '/var/log/django-hostname.log'

Solution:

Set DJANGO_LOGPATH to a writable directory:

export DJANGO_LOGPATH='/tmp'
# or create logs directory in project
mkdir -p logs
export DJANGO_LOGPATH='./logs'

Module Import Errors

ModuleNotFoundError: No module named 'psycopg2'

Solutions:

Ensure virtual environment is activated
Install system dependencies (see Prerequisites)

Reinstall requirements:

pip install --upgrade -r requirements.txt

Docker Issues

Port Already in Use:

# Find and stop conflicting process
sudo lsof -i :8000
sudo kill <PID>

Container Build Failures:

# Clean Docker cache and rebuild
docker system prune -f
docker-compose build --no-cache

Getting Help#

If you encounter issues not covered here:

Check the GitHub Issues
Review the Django logs for detailed error messages
Ensure all prerequisites are correctly installed
Try the Docker setup if manual installation fails

Useful Commands for Debugging:

# Check Python environment
python --version
pip list

# Check PostgreSQL connection
psql -h localhost -U your_user -d your_database -c "SELECT 1;"

# Check Django configuration
python manage.py check

# View detailed Django errors
python manage.py runserver --verbosity=2

Next Steps#

After successful installation:

Read the Usage Guide: See Usage Guide for API examples
Development: See Development Guide for development guidelines
Architecture: Learn about the system in Architecture

Tip

Quick API Test: Try this command to verify everything is working:

curl -H "Content-Type: application/json" http://localhost:8000/api/cdb_rest/gt

Installation

Contents

Installation#

Quick Start with Docker (Recommended)#

Manual Installation#

Prerequisites#

System Requirements#

Installation Steps#

Docker Compose Setup (Detailed)#

Prerequisites#

Setup Steps#

Environment Variables Reference#

Core Settings#

Database Configuration#

Production Deployment#

Security Checklist#

Production Configuration Example#

Troubleshooting#

Common Issues and Solutions#

Getting Help#

Next Steps#