wp-statpress-analyze/README.md

IP Geo Visitor Analysis Toolkit

A set of three Python scripts to process large-scale web visitor logs from MySQL, enrich them with geographic data, detect bots, and generate monthly analytics reports.

---

Overview

Script	Purpose
ip_geo_report.py	Read visitor records from MySQL, resolve IPs to countries, export daily visit counts per country
analyze.py	Read the CSV output from ip_geo_report.py and generate monthly visit stats with MoM changes and charts
bot_analyzer.py	Select records from a date range and score each visit to classify as human, bot, or suspicious

---

Requirements

• Python 3.8+
• MySQL database with visitor records
• ipinfo bundle_location_lite.mmdb (free, download from ipinfo.io)

---

Installation

# Clone or copy scripts to your project folder
mkdir visitor-analysis && cd visitor-analysis

# Create virtual environment
python3 -m venv venv
source venv/bin/activate        # Linux / macOS
# venv\Scripts\activate         # Windows

# Install dependencies
pip install mysql-connector-python maxminddb matplotlib

---

Project Structure

visitor-analysis/
├── venv/                           ← virtual environment (do not commit)
├── bundle_location_lite.mmdb       ← ipinfo offline GeoIP database
├── ip_geo_report.py                ← Script 1: IP → Country report
├── analyze.py                      ← Script 2: Monthly analytics + charts
├── bot_analyzer.py                 ← Script 3: Bot detection
├── requirements.txt
├── README.md
├── cache/                          ← Auto-created: LRU disk cache + checkpoints
└── report/                         ← Auto-created: CSV output from Script 1

---

Script 1 — ip_geo_report.py

Reads all visitor records from MySQL in batches, resolves each IP address to a country using the ipinfo mmdb database, and produces a CSV report of daily visit counts per country.

Features

• Batch processing with cursor-based pagination (handles 100M+ rows)
• Two-level cache: LRU memory (100k IPs) + disk shelve spillover
• Resume from checkpoint on crash or interruption
• ipinfo mmdb offline lookup → ipinfo API fallback
• Bot/attack filtering before GeoIP lookup

Configuration

Edit the top section of ip_geo_report.py:

DB_CONFIG = {
    "host":     "localhost",
    "user":     "your_user",
    "password": "your_password",
    "database": "your_database",
}

TABLE_NAME      = "visitors"        # your table name
IP_COLUMN       = "ip"              # IP address column
DATE_COLUMN     = "visit_date"      # DATE or DATETIME column
UA_COLUMN       = "user_agent"      # set None if not available
PATH_COLUMN     = "path"            # set None if not available
GEOIP_DB_PATH   = "./bundle_location_lite.mmdb"
IPINFO_TOKEN    = ""                # optional API fallback token
BATCH_SIZE      = 50_000
LRU_MAX_SIZE    = 100_000

Usage

source venv/bin/activate
python3 ip_geo_report.py

If interrupted, simply re-run — it resumes from the last checkpoint automatically.

Output

report/
├── report_YYYY-MM-DD.csv       ← Country, Date, Visit Count
└── summary_YYYY-MM-DD.csv      ← Country, Total Visits (sorted)

---

Script 2 — analyze.py

Reads the CSV output from ip_geo_report.py and generates monthly analytics including MoM (Month-over-Month) percentage changes and a 4-panel visualization chart.

Features

• Auto-detects latest report_*.csv in ./report/
• Handles multiple date formats (YYYY-MM-DD, YYYYMMDD, YYYY/MM/DD)
• Monthly visit totals with MoM % change
• Monthly unique country counts with MoM % change
• Top N countries per month
• 4-panel PNG chart (bar charts + MoM line + stacked country chart)

Usage

# Auto-detect latest report
python3 analyze.py

# Specify input file
python3 analyze.py --input ./report/report_2026-09-27.csv

# Custom output dir and top N countries
python3 analyze.py \
    --input  ./report/report_2026-09-27.csv \
    --output ./analysis \
    --top    10

Arguments

Argument	Default	Description
--input, -i	latest in ./report/	Path to input CSV
--output, -o	./analysis/	Output directory
--top, -n	10	Top N countries per month

Output

analysis/
├── monthly_summary_YYYYMMDD.csv    ← Month, Visits, MoM%, Countries, MoM%
├── top_countries_YYYYMMDD.csv      ← Month, Rank, Country, Visits, %
└── analysis_chart_YYYYMMDD.png     ← 4-panel visualization chart

Chart Panels

Panel	Content
Top-left	Monthly total visits bar + MoM % line
Top-right	Monthly unique countries bar + MoM % line
Bottom-left	MoM % comparison line chart with +/- shading
Bottom-right	Stacked bar — top 5 countries (last 6 months)

---

Script 3 — bot_analyzer.py

Selects visitor records from a MySQL date range and scores each visit across multiple signals to classify it as human, bot, or suspicious.

Features

• Two-pass processing: frequency counting then per-record scoring
• Windowed frequency detection (per day, per hour, traffic ratio)
• Pattern-based signals: user agent, request path, IP range
• Outputs three separate CSVs: human / bot / suspicious
• Detailed summary with top high-frequency IPs

Scoring Rules

Signal	Score	Trigger
Invalid / Private IP	+10	Reserved IP ranges (RFC 1918 etc.)
Bot User Agent	+8	googlebot, scrapers, crawlers
High Freq Per Hour	+8	IP hits > threshold in single hour
Attack Path	+9	SQLi, shells, scanner paths
Tool User Agent	+7	curl, wget, python, selenium
No User Agent	+7	Empty or missing UA
High Freq Per Day	+7	IP hits > threshold in single day
Old IE Browser	+5	IE 6/7/8 (commonly spoofed)
High Traffic Ratio	+5	IP > 1% of all traffic in range
Suspicious Combo	+4	Old UA + attack path together
Empty Path	+3	Request path is blank or just /

Classification thresholds:

Score	Classification
0 – 2	✅ Human
3 – 4	⚠️ Suspicious
5+	🤖 Bot

Configuration

HIGH_FREQ_PER_DAY     = 100     # flag if IP hits > N times in a single day
HIGH_FREQ_PER_HOUR    = 30      # flag if IP hits > N times in a single hour
HIGH_FREQ_TOTAL_RATIO = 0.01    # flag if IP > 1% of total traffic in range
BOT_SCORE_THRESHOLD   = 5       # score >= this = bot
SUSPICIOUS_THRESHOLD  = 3       # score >= this = suspicious

Usage

# Basic usage
python3 bot_analyzer.py --start 2026-01-01 --end 2026-09-30

# Custom thresholds
python3 bot_analyzer.py \
    --start         2026-01-01 \
    --end           2026-09-30 \
    --bot-threshold 5 \
    --freq-per-day  100 \
    --freq-per-hour 30 \
    --freq-ratio    0.01 \
    --output        ./bot_analysis

Arguments

Argument	Default	Description
--start, -s	required	Start date YYYY-MM-DD
--end, -e	required	End date YYYY-MM-DD
--output, -o	./bot_analysis/	Output directory
--bot-threshold	5	Minimum score to classify as bot
--freq-per-day	100	Max hits per day before flagging
--freq-per-hour	30	Max hits per hour before flagging
--freq-ratio	0.01	Max fraction of total traffic per IP

Output

bot_analysis/
├── human_DATERANGE_TIMESTAMP.csv       ← Confirmed human visits
├── bot_DATERANGE_TIMESTAMP.csv         ← Confirmed bot visits + score + reasons
├── suspicious_DATERANGE_TIMESTAMP.csv  ← Borderline visits for manual review
└── summary_DATERANGE_TIMESTAMP.csv     ← Stats, reason counts, top IPs

---

Recommended Workflow

1. Run ip_geo_report.py
   → Processes all historical data
   → Outputs report/report_YYYY-MM-DD.csv

2. Run analyze.py
   → Reads the report CSV
   → Outputs monthly stats + chart

3. Run bot_analyzer.py --start YYYY-MM-DD --end YYYY-MM-DD
   → Focuses on a specific time window
   → Outputs human / bot / suspicious CSVs

---

GeoIP Database Setup

1. Register for a free account at https://ipinfo.io/signup
2. Go to https://ipinfo.io/account/data-downloads
3. Download bundle_location_lite.mmdb
4. Place it in the project root directory

The database provides: country, country_code, continent, asn, as_name, as_domain

---

Database Table Requirements

Your MySQL visitor table should have at minimum:

Column	Type	Required	Description
id	INT	✅ Yes	Primary key for cursor pagination
ip	VARCHAR	✅ Yes	Visitor IP address
visit_date	DATE / DATETIME	✅ Yes	Visit timestamp
user_agent	VARCHAR	⚠️ Optional	Browser user agent string
path	VARCHAR	⚠️ Optional	Request path / URL

Set UA_COLUMN = None and PATH_COLUMN = None in scripts if those columns do not exist.

---

requirements.txt

mysql-connector-python
maxminddb
matplotlib

Install with:

pip install -r requirements.txt

---

Notes

• All scripts support resuming interrupted runs (Script 1 via checkpoint file, Script 3 via re-running with same date range)
• Cache files in ./cache/ persist between runs to avoid redundant GeoIP lookups
• Partial report data is saved every 10 batches to survive crashes
• Scripts are designed for Python 3.8+ and tested on Ubuntu 20.04