wp-statpress-analyze/README.md

# IP Geo Visitor Analysis Toolkit

A set of three Python scripts to process large-scale web visitor logs from MySQL, enrich them with geographic data, detect bots, and generate monthly analytics reports.

---

## Overview

| Script | Purpose |
|---|---|
| `ip_geo_report.py` | Read visitor records from MySQL, resolve IPs to countries, export daily visit counts per country |
| `analyze.py` | Read the CSV output from `ip_geo_report.py` and generate monthly visit stats with MoM changes and charts |
| `bot_analyzer.py` | Select records from a date range and score each visit to classify as human, bot, or suspicious |

---

## Requirements

- Python 3.8+
- MySQL database with visitor records
- ipinfo `bundle_location_lite.mmdb` (free, download from [ipinfo.io](https://ipinfo.io/account/data-downloads))

---

## Installation

```bash
# Clone or copy scripts to your project folder
mkdir visitor-analysis && cd visitor-analysis

# Create virtual environment
python3 -m venv venv
source venv/bin/activate        # Linux / macOS
# venv\Scripts\activate         # Windows

# Install dependencies
pip install mysql-connector-python maxminddb matplotlib
```

---

## Project Structure

```
visitor-analysis/
├── venv/                           ← virtual environment (do not commit)
├── bundle_location_lite.mmdb       ← ipinfo offline GeoIP database
├── ip_geo_report.py                ← Script 1: IP → Country report
├── analyze.py                      ← Script 2: Monthly analytics + charts
├── bot_analyzer.py                 ← Script 3: Bot detection
├── requirements.txt
├── README.md
├── cache/                          ← Auto-created: LRU disk cache + checkpoints
└── report/                         ← Auto-created: CSV output from Script 1
```

---

## Script 1 — `ip_geo_report.py`

Reads all visitor records from MySQL in batches, resolves each IP address to a country using the ipinfo mmdb database, and produces a CSV report of daily visit counts per country.

### Features

- Batch processing with cursor-based pagination (handles 100M+ rows)
- Two-level cache: LRU memory (100k IPs) + disk shelve spillover
- Resume from checkpoint on crash or interruption
- ipinfo mmdb offline lookup → ipinfo API fallback
- Bot/attack filtering before GeoIP lookup

### Configuration

Edit the top section of `ip_geo_report.py`:

```python
DB_CONFIG = {
    "host":     "localhost",
    "user":     "your_user",
    "password": "your_password",
    "database": "your_database",
}

TABLE_NAME      = "visitors"        # your table name
IP_COLUMN       = "ip"              # IP address column
DATE_COLUMN     = "visit_date"      # DATE or DATETIME column
UA_COLUMN       = "user_agent"      # set None if not available
PATH_COLUMN     = "path"            # set None if not available
GEOIP_DB_PATH   = "./bundle_location_lite.mmdb"
IPINFO_TOKEN    = ""                # optional API fallback token
BATCH_SIZE      = 50_000
LRU_MAX_SIZE    = 100_000
```

### Usage

```bash
source venv/bin/activate
python3 ip_geo_report.py
```

If interrupted, simply re-run — it resumes from the last checkpoint automatically.

### Output

```
report/
├── report_YYYY-MM-DD.csv       ← Country, Date, Visit Count
└── summary_YYYY-MM-DD.csv      ← Country, Total Visits (sorted)
```

---

## Script 2 — `analyze.py`

Reads the CSV output from `ip_geo_report.py` and generates monthly analytics including MoM (Month-over-Month) percentage changes and a 4-panel visualization chart.

### Features

- Auto-detects latest `report_*.csv` in `./report/`
- Handles multiple date formats (`YYYY-MM-DD`, `YYYYMMDD`, `YYYY/MM/DD`)
- Monthly visit totals with MoM % change
- Monthly unique country counts with MoM % change
- Top N countries per month
- 4-panel PNG chart (bar charts + MoM line + stacked country chart)

### Usage

```bash
# Auto-detect latest report
python3 analyze.py

# Specify input file
python3 analyze.py --input ./report/report_2026-09-27.csv

# Custom output dir and top N countries
python3 analyze.py \
    --input  ./report/report_2026-09-27.csv \
    --output ./analysis \
    --top    10
```

### Arguments

| Argument | Default | Description |
|---|---|---|
| `--input`, `-i` | latest in `./report/` | Path to input CSV |
| `--output`, `-o` | `./analysis/` | Output directory |
| `--top`, `-n` | `10` | Top N countries per month |

### Output

```
analysis/
├── monthly_summary_YYYYMMDD.csv    ← Month, Visits, MoM%, Countries, MoM%
├── top_countries_YYYYMMDD.csv      ← Month, Rank, Country, Visits, %
└── analysis_chart_YYYYMMDD.png     ← 4-panel visualization chart
```

### Chart Panels

| Panel | Content |
|---|---|
| Top-left | Monthly total visits bar + MoM % line |
| Top-right | Monthly unique countries bar + MoM % line |
| Bottom-left | MoM % comparison line chart with +/- shading |
| Bottom-right | Stacked bar — top 5 countries (last 6 months) |

---

## Script 3 — `bot_analyzer.py`

Selects visitor records from a MySQL date range and scores each visit across multiple signals to classify it as human, bot, or suspicious.

### Features

- Two-pass processing: frequency counting then per-record scoring
- Windowed frequency detection (per day, per hour, traffic ratio)
- Pattern-based signals: user agent, request path, IP range
- Outputs three separate CSVs: human / bot / suspicious
- Detailed summary with top high-frequency IPs

### Scoring Rules

| Signal | Score | Trigger |
|---|---|---|
| Invalid / Private IP | +10 | Reserved IP ranges (RFC 1918 etc.) |
| Bot User Agent | +8 | googlebot, scrapers, crawlers |
| High Freq Per Hour | +8 | IP hits > threshold in single hour |
| Attack Path | +9 | SQLi, shells, scanner paths |
| Tool User Agent | +7 | curl, wget, python, selenium |
| No User Agent | +7 | Empty or missing UA |
| High Freq Per Day | +7 | IP hits > threshold in single day |
| Old IE Browser | +5 | IE 6/7/8 (commonly spoofed) |
| High Traffic Ratio | +5 | IP > 1% of all traffic in range |
| Suspicious Combo | +4 | Old UA + attack path together |
| Empty Path | +3 | Request path is blank or just `/` |

**Classification thresholds:**

| Score | Classification |
|---|---|
| 0 – 2 | ✅ Human |
| 3 – 4 | ⚠️ Suspicious |
| 5+ | 🤖 Bot |

### Configuration

```python
HIGH_FREQ_PER_DAY     = 100     # flag if IP hits > N times in a single day
HIGH_FREQ_PER_HOUR    = 30      # flag if IP hits > N times in a single hour
HIGH_FREQ_TOTAL_RATIO = 0.01    # flag if IP > 1% of total traffic in range
BOT_SCORE_THRESHOLD   = 5       # score >= this = bot
SUSPICIOUS_THRESHOLD  = 3       # score >= this = suspicious
```

### Usage

```bash
# Basic usage
python3 bot_analyzer.py --start 2026-01-01 --end 2026-09-30

# Custom thresholds
python3 bot_analyzer.py \
    --start         2026-01-01 \
    --end           2026-09-30 \
    --bot-threshold 5 \
    --freq-per-day  100 \
    --freq-per-hour 30 \
    --freq-ratio    0.01 \
    --output        ./bot_analysis
```

### Arguments

| Argument | Default | Description |
|---|---|---|
| `--start`, `-s` | required | Start date `YYYY-MM-DD` |
| `--end`, `-e` | required | End date `YYYY-MM-DD` |
| `--output`, `-o` | `./bot_analysis/` | Output directory |
| `--bot-threshold` | `5` | Minimum score to classify as bot |
| `--freq-per-day` | `100` | Max hits per day before flagging |
| `--freq-per-hour` | `30` | Max hits per hour before flagging |
| `--freq-ratio` | `0.01` | Max fraction of total traffic per IP |

### Output

```
bot_analysis/
├── human_DATERANGE_TIMESTAMP.csv       ← Confirmed human visits
├── bot_DATERANGE_TIMESTAMP.csv         ← Confirmed bot visits + score + reasons
├── suspicious_DATERANGE_TIMESTAMP.csv  ← Borderline visits for manual review
└── summary_DATERANGE_TIMESTAMP.csv     ← Stats, reason counts, top IPs
```

---

## Recommended Workflow

```
1. Run ip_geo_report.py
   → Processes all historical data
   → Outputs report/report_YYYY-MM-DD.csv

2. Run analyze.py
   → Reads the report CSV
   → Outputs monthly stats + chart

3. Run bot_analyzer.py --start YYYY-MM-DD --end YYYY-MM-DD
   → Focuses on a specific time window
   → Outputs human / bot / suspicious CSVs
```

---

## GeoIP Database Setup

1. Register for a free account at [https://ipinfo.io/signup](https://ipinfo.io/signup)
2. Go to [https://ipinfo.io/account/data-downloads](https://ipinfo.io/account/data-downloads)
3. Download **`bundle_location_lite.mmdb`**
4. Place it in the project root directory

The database provides: `country`, `country_code`, `continent`, `asn`, `as_name`, `as_domain`

---

## Database Table Requirements

Your MySQL visitor table should have at minimum:

| Column | Type | Required | Description |
|---|---|---|---|
| `id` | INT | ✅ Yes | Primary key for cursor pagination |
| `ip` | VARCHAR | ✅ Yes | Visitor IP address |
| `visit_date` | DATE / DATETIME | ✅ Yes | Visit timestamp |
| `user_agent` | VARCHAR | ⚠️ Optional | Browser user agent string |
| `path` | VARCHAR | ⚠️ Optional | Request path / URL |

> Set `UA_COLUMN = None` and `PATH_COLUMN = None` in scripts if those columns do not exist.

---

## requirements.txt

```
mysql-connector-python
maxminddb
matplotlib
```

Install with:
```bash
pip install -r requirements.txt
```

---

## Notes

- All scripts support resuming interrupted runs (Script 1 via checkpoint file, Script 3 via re-running with same date range)
- Cache files in `./cache/` persist between runs to avoid redundant GeoIP lookups
- Partial report data is saved every 10 batches to survive crashes
- Scripts are designed for Python 3.8+ and tested on Ubuntu 20.04