init commit

2026-09-30 21:05:53 +08:00
commit f7410dcfe4
6 changed files with 2219 additions and 0 deletions
@@ -0,0 +1,322 @@
+# IP Geo Visitor Analysis Toolkit
+
+A set of three Python scripts to process large-scale web visitor logs from MySQL, enrich them with geographic data, detect bots, and generate monthly analytics reports.
+
+---
+
+## Overview
+
+| Script | Purpose |
+|---|---|
+| `ip_geo_report.py` | Read visitor records from MySQL, resolve IPs to countries, export daily visit counts per country |
+| `analyze.py` | Read the CSV output from `ip_geo_report.py` and generate monthly visit stats with MoM changes and charts |
+| `bot_analyzer.py` | Select records from a date range and score each visit to classify as human, bot, or suspicious |
+
+---
+
+## Requirements
+
+- Python 3.8+
+- MySQL database with visitor records
+- ipinfo `bundle_location_lite.mmdb` (free, download from [ipinfo.io](https://ipinfo.io/account/data-downloads))
+
+---
+
+## Installation
+
+```bash
+# Clone or copy scripts to your project folder
+mkdir visitor-analysis && cd visitor-analysis
+
+# Create virtual environment
+python3 -m venv venv
+source venv/bin/activate        # Linux / macOS
+# venv\Scripts\activate         # Windows
+
+# Install dependencies
+pip install mysql-connector-python maxminddb matplotlib
+```
+
+---
+
+## Project Structure
+
+```
+visitor-analysis/
+├── venv/                           ← virtual environment (do not commit)
+├── bundle_location_lite.mmdb       ← ipinfo offline GeoIP database
+├── ip_geo_report.py                ← Script 1: IP → Country report
+├── analyze.py                      ← Script 2: Monthly analytics + charts
+├── bot_analyzer.py                 ← Script 3: Bot detection
+├── requirements.txt
+├── README.md
+├── cache/                          ← Auto-created: LRU disk cache + checkpoints
+└── report/                         ← Auto-created: CSV output from Script 1
+```
+
+---
+
+## Script 1 — `ip_geo_report.py`
+
+Reads all visitor records from MySQL in batches, resolves each IP address to a country using the ipinfo mmdb database, and produces a CSV report of daily visit counts per country.
+
+### Features
+
+- Batch processing with cursor-based pagination (handles 100M+ rows)
+- Two-level cache: LRU memory (100k IPs) + disk shelve spillover
+- Resume from checkpoint on crash or interruption
+- ipinfo mmdb offline lookup → ipinfo API fallback
+- Bot/attack filtering before GeoIP lookup
+
+### Configuration
+
+Edit the top section of `ip_geo_report.py`:
+
+```python
+DB_CONFIG = {
+    "host":     "localhost",
+    "user":     "your_user",
+    "password": "your_password",
+    "database": "your_database",
+}
+
+TABLE_NAME      = "visitors"        # your table name
+IP_COLUMN       = "ip"              # IP address column
+DATE_COLUMN     = "visit_date"      # DATE or DATETIME column
+UA_COLUMN       = "user_agent"      # set None if not available
+PATH_COLUMN     = "path"            # set None if not available
+GEOIP_DB_PATH   = "./bundle_location_lite.mmdb"
+IPINFO_TOKEN    = ""                # optional API fallback token
+BATCH_SIZE      = 50_000
+LRU_MAX_SIZE    = 100_000
+```
+
+### Usage
+
+```bash
+source venv/bin/activate
+python3 ip_geo_report.py
+```
+
+If interrupted, simply re-run — it resumes from the last checkpoint automatically.
+
+### Output
+
+```
+report/
+├── report_YYYY-MM-DD.csv       ← Country, Date, Visit Count
+└── summary_YYYY-MM-DD.csv      ← Country, Total Visits (sorted)
+```
+
+---
+
+## Script 2 — `analyze.py`
+
+Reads the CSV output from `ip_geo_report.py` and generates monthly analytics including MoM (Month-over-Month) percentage changes and a 4-panel visualization chart.
+
+### Features
+
+- Auto-detects latest `report_*.csv` in `./report/`
+- Handles multiple date formats (`YYYY-MM-DD`, `YYYYMMDD`, `YYYY/MM/DD`)
+- Monthly visit totals with MoM % change
+- Monthly unique country counts with MoM % change
+- Top N countries per month
+- 4-panel PNG chart (bar charts + MoM line + stacked country chart)
+
+### Usage
+
+```bash
+# Auto-detect latest report
+python3 analyze.py
+
+# Specify input file
+python3 analyze.py --input ./report/report_2026-09-27.csv
+
+# Custom output dir and top N countries
+python3 analyze.py \
+    --input  ./report/report_2026-09-27.csv \
+    --output ./analysis \
+    --top    10
+```
+
+### Arguments
+
+| Argument | Default | Description |
+|---|---|---|
+| `--input`, `-i` | latest in `./report/` | Path to input CSV |
+| `--output`, `-o` | `./analysis/` | Output directory |
+| `--top`, `-n` | `10` | Top N countries per month |
+
+### Output
+
+```
+analysis/
+├── monthly_summary_YYYYMMDD.csv    ← Month, Visits, MoM%, Countries, MoM%
+├── top_countries_YYYYMMDD.csv      ← Month, Rank, Country, Visits, %
+└── analysis_chart_YYYYMMDD.png     ← 4-panel visualization chart
+```
+
+### Chart Panels
+
+| Panel | Content |
+|---|---|
+| Top-left | Monthly total visits bar + MoM % line |
+| Top-right | Monthly unique countries bar + MoM % line |
+| Bottom-left | MoM % comparison line chart with +/- shading |
+| Bottom-right | Stacked bar — top 5 countries (last 6 months) |
+
+---
+
+## Script 3 — `bot_analyzer.py`
+
+Selects visitor records from a MySQL date range and scores each visit across multiple signals to classify it as human, bot, or suspicious.
+
+### Features
+
+- Two-pass processing: frequency counting then per-record scoring
+- Windowed frequency detection (per day, per hour, traffic ratio)
+- Pattern-based signals: user agent, request path, IP range
+- Outputs three separate CSVs: human / bot / suspicious
+- Detailed summary with top high-frequency IPs
+
+### Scoring Rules
+
+| Signal | Score | Trigger |
+|---|---|---|
+| Invalid / Private IP | +10 | Reserved IP ranges (RFC 1918 etc.) |
+| Bot User Agent | +8 | googlebot, scrapers, crawlers |
+| High Freq Per Hour | +8 | IP hits > threshold in single hour |
+| Attack Path | +9 | SQLi, shells, scanner paths |
+| Tool User Agent | +7 | curl, wget, python, selenium |
+| No User Agent | +7 | Empty or missing UA |
+| High Freq Per Day | +7 | IP hits > threshold in single day |
+| Old IE Browser | +5 | IE 6/7/8 (commonly spoofed) |
+| High Traffic Ratio | +5 | IP > 1% of all traffic in range |
+| Suspicious Combo | +4 | Old UA + attack path together |
+| Empty Path | +3 | Request path is blank or just `/` |
+
+**Classification thresholds:**
+
+| Score | Classification |
+|---|---|
+| 0 – 2 | ✅ Human |
+| 3 – 4 | ⚠️ Suspicious |
+| 5+ | 🤖 Bot |
+
+### Configuration
+
+```python
+HIGH_FREQ_PER_DAY     = 100     # flag if IP hits > N times in a single day
+HIGH_FREQ_PER_HOUR    = 30      # flag if IP hits > N times in a single hour
+HIGH_FREQ_TOTAL_RATIO = 0.01    # flag if IP > 1% of total traffic in range
+BOT_SCORE_THRESHOLD   = 5       # score >= this = bot
+SUSPICIOUS_THRESHOLD  = 3       # score >= this = suspicious
+```
+
+### Usage
+
+```bash
+# Basic usage
+python3 bot_analyzer.py --start 2026-01-01 --end 2026-09-30
+
+# Custom thresholds
+python3 bot_analyzer.py \
+    --start         2026-01-01 \
+    --end           2026-09-30 \
+    --bot-threshold 5 \
+    --freq-per-day  100 \
+    --freq-per-hour 30 \
+    --freq-ratio    0.01 \
+    --output        ./bot_analysis
+```
+
+### Arguments
+
+| Argument | Default | Description |
+|---|---|---|
+| `--start`, `-s` | required | Start date `YYYY-MM-DD` |
+| `--end`, `-e` | required | End date `YYYY-MM-DD` |
+| `--output`, `-o` | `./bot_analysis/` | Output directory |
+| `--bot-threshold` | `5` | Minimum score to classify as bot |
+| `--freq-per-day` | `100` | Max hits per day before flagging |
+| `--freq-per-hour` | `30` | Max hits per hour before flagging |
+| `--freq-ratio` | `0.01` | Max fraction of total traffic per IP |
+
+### Output
+
+```
+bot_analysis/
+├── human_DATERANGE_TIMESTAMP.csv       ← Confirmed human visits
+├── bot_DATERANGE_TIMESTAMP.csv         ← Confirmed bot visits + score + reasons
+├── suspicious_DATERANGE_TIMESTAMP.csv  ← Borderline visits for manual review
+└── summary_DATERANGE_TIMESTAMP.csv     ← Stats, reason counts, top IPs
+```
+
+---
+
+## Recommended Workflow
+
+```
+1. Run ip_geo_report.py
+   → Processes all historical data
+   → Outputs report/report_YYYY-MM-DD.csv
+
+2. Run analyze.py
+   → Reads the report CSV
+   → Outputs monthly stats + chart
+
+3. Run bot_analyzer.py --start YYYY-MM-DD --end YYYY-MM-DD
+   → Focuses on a specific time window
+   → Outputs human / bot / suspicious CSVs
+```
+
+---
+
+## GeoIP Database Setup
+
+1. Register for a free account at [https://ipinfo.io/signup](https://ipinfo.io/signup)
+2. Go to [https://ipinfo.io/account/data-downloads](https://ipinfo.io/account/data-downloads)
+3. Download **`bundle_location_lite.mmdb`**
+4. Place it in the project root directory
+
+The database provides: `country`, `country_code`, `continent`, `asn`, `as_name`, `as_domain`
+
+---
+
+## Database Table Requirements
+
+Your MySQL visitor table should have at minimum:
+
+| Column | Type | Required | Description |
+|---|---|---|---|
+| `id` | INT | ✅ Yes | Primary key for cursor pagination |
+| `ip` | VARCHAR | ✅ Yes | Visitor IP address |
+| `visit_date` | DATE / DATETIME | ✅ Yes | Visit timestamp |
+| `user_agent` | VARCHAR | ⚠️ Optional | Browser user agent string |
+| `path` | VARCHAR | ⚠️ Optional | Request path / URL |
+
+> Set `UA_COLUMN = None` and `PATH_COLUMN = None` in scripts if those columns do not exist.
+
+---
+
+## requirements.txt
+
+```
+mysql-connector-python
+maxminddb
+matplotlib
+```
+
+Install with:
+```bash
+pip install -r requirements.txt
+```
+
+---
+
+## Notes
+
+- All scripts support resuming interrupted runs (Script 1 via checkpoint file, Script 3 via re-running with same date range)
+- Cache files in `./cache/` persist between runs to avoid redundant GeoIP lookups
+- Partial report data is saved every 10 batches to survive crashes
+- Scripts are designed for Python 3.8+ and tested on Ubuntu 20.04