AM-WEB/arisingmedia-web-sops
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
94f7a1f72a9e39c89bb7303363ded859f8553063
arisingmedia-web-sops/STACK.md
T
acobham 94f7a1f72a recent updates
2026-06-09 18:31:59 +02:00

1771 lines
62 KiB
Markdown
Raw Blame History

# STACK — Architecture, Deployment, and Build Pipeline
Author: Andre Cobham / Arising Media
Updated: 2026-06-09
## Stack Philosophy
Two primary stacks. Pick based on page count and update frequency.
### Stack A — PHP Router + SQLite (50+ pages, standard as of 2026-05-21)
- **PHP Router** — `router.php` dispatches every content URL to the correct PHP template. Edit one template = entire page class updates on next request. No find-and-replace. No file edits.
- **SQLite** — single-file content DB. `pages.sqlite` holds all page content (title, meta, sections JSON, schema). 10,000 rows = 5MB. Sub-millisecond reads. No server process.
- **Vanilla JavaScript** — no frameworks. `fetch`, `IntersectionObserver`, `querySelector`
- **Plain CSS** — `tokens.css` (design tokens) + `main.css` (components). No Sass, no Tailwind
- **Docker + nginx** — nginx routes `/assets/*` directly; all content URLs → PHP-FPM → router.php
- **Resend** — transactional email via `/api/contact.php`
- **Reference:** `arisingmedia.us` — 10,000+ pages
### Stack B — Static HTML (fewer than 50 pages)
- **Static HTML** — every page is a `.html` file on disk
- Same JS, CSS, Docker, nginx, Resend as Stack A
- Python 3 stdlib for build scripts (no pip)
- **Reference:** `lahrcarpetcleaning.com`
### Never Use (Both Stacks)
- Node.js / npm packages on the website. Front-end JS uses ZERO packages
- WordPress for new builds (we migrate clients OUT of WordPress)
- CSS frameworks (Bootstrap, Tailwind, Bulma)
- JS frameworks (React, Vue, Angular, Svelte)
- jQuery, Lodash, Moment, axios, or any utility library
- CSS-in-JS, styled-components
- Build tools that require `node_modules` (webpack, vite, parcel, esbuild)
- Tracking pixels other than what the client explicitly requests
### Why This Stack
1. **Performance** — a static HTML page with vanilla JS loads in <100ms with no parse cost from frameworks
2. **Longevity** — no dependency rot. A site we build today still works in 10 years with no maintenance
3. **Security** — no `npm audit` warnings, no supply-chain attack vectors, no transitive deps to patch
4. **Auditability** — every line on the site is something we wrote and can read in plain text
5. **Hosting** — a static folder + tiny Python container fits in the smallest VM tier any provider sells
### When to Add a Server-Side Service
Static-only is the default. Add a small Python service ONLY when needed for:
- Form submission (handled via Resend in the stdlib HTTP server pattern)
- A specific dynamic feature the client paid for (e.g., booking widget, AI chat)
Each service is its own Docker container. Keep them small (single file when possible).
Use Python `http.server` + `urllib` from stdlib. Do not introduce Flask, FastAPI, Django, or any third-party HTTP framework.
---
## Project Structure
Two folders per project: source and deployment.
### Source Folder
Lives in the dev tree under `concept-agent/projects/{domain}/site/`.
Contains everything needed to maintain and rebuild the site.
```
{domain}/site/
├── index.html # home page
├── about/index.html # /about/
├── contact/index.html # /contact/
├── reviews/index.html # /reviews/
├── blog/index.html # /blog/
├── locations/ # location pages
│ ├── index.html # /locations/
│ ├── _template.html # template stamped with JSON
│ ├── buffalo.html # generated, flat URL
│ ├── amherst.html
│ └── ...
├── services/
│ ├── index.html
│ ├── _template.html
│ ├── floor-refinishing.html
│ └── ...
├── components/
│ ├── header.html # loaded via fetch() by components.js
│ └── footer.html
├── data/
│ ├── locations.json # source data for build_locations.py
│ └── services.json # source data for build_services.py
├── assets/
│ ├── css/
│ │ ├── main.css # variables, reset, layout
│ │ └── components.css # cards, hero, header, footer, nav, responsive
│ ├── js/
│ │ ├── main.js # scroll animations, count-up, etc.
│ │ ├── components.js # fetch + inject header/footer
│ │ └── form.js # form validation + submit
│ ├── images/
│ ├── videos/ # hero video files (.mp4 + .webm)
│ └── fonts/ # only if not using Google Fonts CDN
├── build_locations.py # JSON → flat .html stamping
├── build_services.py
└── README.md # project notes, content sources, status
```
### Deployment Folder
Lives at `/home/sirdrez/arisingmedia-websites/{domain}/`.
Contains ONLY what's needed to run `docker compose up`.
```
{domain}/
├── index.html # all public website folders
├── about/ # ↑
├── assets/ # ↑
├── blog/ # ↑
├── components/ # ↑
├── contact/ # ↑
├── locations/ # ↑
├── reviews/ # ↑
├── services/ # ↑
├── api/ # form-submit Python service (if used)
│ ├── server.py
│ ├── Dockerfile
│ ├── .env # gitignored — Resend key, etc.
│ └── .env.example
├── Dockerfile # nginx web container
├── nginx.conf
├── docker-compose.yml
├── .dockerignore
├── .gitignore
└── .planning/ # everything not needed at runtime
├── build_locations.py # build scripts moved here
├── data/ # JSON sources moved here
├── README.md
├── DNS_*.txt # DNS notes
└── review_*.png # design review screenshots
```
### What Goes Where
**Source folder gets** every working file (build scripts, data JSON, screenshots,
notes, raw assets). This is the dev/maintenance copy. NOT what gets deployed.
**Deployment folder gets** ONLY the rendered website + the small API service.
Build scripts, JSON data, and notes go into `.planning/` to keep root clean and
prevent accidental web exposure.
### URL Structure — Two Valid Patterns
#### Pattern A: Flat HTML (default for Docker/nginx projects)
nginx `try_files $uri $uri/ $uri.html =404` serves `/locations/buffalo` and
`/locations/buffalo.html`. Canonical form: `/locations/buffalo.html`.
Why flat:
- One file = one page, no `/index.html` confusion
- Easier sitemap generation
- `<a href>` links are unambiguous
- Crawl budget benefit — Google indexes one URL per page, not two
#### Pattern B: Directory-style (default for cPanel/Apache projects)
Each page lives at `{slug}/index.html`. Apache auto-serves `index.html` when
visiting `/{slug}/`. Use this when deploying to cPanel shared hosting.
```
services/
├── carpet-cleaning/index.html → /services/carpet-cleaning/
├── stairs/index.html → /services/stairs/
commercial/
├── offices/index.html → /commercial/offices/
└── vacation-rentals/index.html → /commercial/vacation-rentals/
```
### Lahrcarpetcleaning.com Reference (Directory-Style, cPanel)
```
lahrcarpetcleaning.com/
├── index.html
├── about/index.html
├── contact/index.html
├── reviews/index.html
├── service-area/index.html
├── locations/
│ ├── index.html
│ ├── waterloo-ny/index.html
│ ├── geneva-ny/index.html
│ └── ... (20 location pages)
├── services/
│ ├── carpet-cleaning/index.html
│ ├── stairs/index.html
│ ├── upholstery/index.html
│ ├── floors/index.html
│ ├── area-rugs/index.html
│ ├── add-ons/index.html
│ └── commercial/index.html
├── commercial/
│ ├── offices/index.html
│ ├── vacation-rentals/index.html
│ ├── hotels-inns/index.html
│ ├── retail-showrooms/index.html
│ └── property-management/index.html
├── assets/
│ ├── css/styles.css?v=N ← always cache-bust on change
│ ├── js/
│ │ ├── main.js
│ │ └── components.js ← injects nav+footer via innerHTML
│ ├── images/
│ │ ├── hero/ ← hero-{slug}.webp, one per page
│ │ └── services/ ← {service}.webp card images
│ └── videos/hero/hero-reel.mp4
├── tools/ ← NOT deployed to webroot
│ ├── convert-to-webp.py
│ ├── gen-images-flux.py
│ └── gen-hero-images.py
├── .cpanel.yml
├── robots.txt
├── sitemap.xml
├── 404.html
└── 500.html
```
All images are `.webp`. cPanel deployment via `.cpanel.yml`.
---
## Build Pipeline
When a site has many similar pages (location pages, service pages, blog posts,
team-member pages), use a JSON + template + Python build script.
### When to Use a Build Script
Use it when there are 4+ pages with identical structure differing only in
content. For example: 6 location pages where only the city name and
city-specific copy differs.
For one-off pages (home, about, contact, services index), hand-write the HTML
directly. Build scripts are for repetition, not for everything.
### Pattern
Three files per template family:
1. **`data/{thing}.json`** — array of objects, one per page
2. **`{thing}/_template.html`** — HTML with `{{placeholder}}` markers
3. **`build_{thing}.py`** — stdlib Python, stamps template with data
#### Example: locations.json
```json
[
{
"slug": "buffalo",
"city": "Buffalo",
"state": "NY",
"title": "Hardwood Floor Refinishing in Buffalo, NY | Floor It",
"meta_description": "Professional hardwood floor refinishing...",
"canonical": "https://floorithardwoodfloors.com/locations/buffalo.html",
"hero_h1": "Hardwood Floor Refinishing in Buffalo, NY",
"hero_lead": "Western New York's most experienced...",
"overview_h2": "Buffalo's Trusted Floor Refinishing Specialists",
"overview_body_1": "...",
"overview_body_2": "...",
"faqs": [
{ "q": "...", "a": "..." }
]
}
]
```
#### Example: _template.html
```html
<!DOCTYPE html>
<html lang="en">
<head>
<title>{{title}}</title>
<meta name="description" content="{{meta_description}}">
<link rel="canonical" href="{{canonical}}">
...
</head>
<body>
<h1>{{hero_h1}}</h1>
<p>{{hero_lead}}</p>
...
</body>
</html>
```
#### Example: build_locations.py (skeleton)
```python
"""Build flat .html location pages from data/locations.json + locations/_template.html."""
import json, sys
from pathlib import Path
SITE_ROOT = Path(__file__).parent
DATA_FILE = SITE_ROOT / "data" / "locations.json"
TEMPLATE_FILE = SITE_ROOT / "locations" / "_template.html"
OUT_DIR = SITE_ROOT / "locations"
def render(template: str, item: dict) -> str:
out = template
for key, value in item.items():
if isinstance(value, (str, int, float)):
out = out.replace("{{" + key + "}}", str(value))
# Custom rendering for nested arrays (e.g. faqs)
# ... handle item['faqs'] etc.
return out
def main():
data = json.loads(DATA_FILE.read_text(encoding="utf-8"))
template = TEMPLATE_FILE.read_text(encoding="utf-8")
print(f"Building {len(data)} location pages...")
for item in data:
rendered = render(template, item)
outfile = OUT_DIR / f"{item['slug']}.html"
outfile.write_text(rendered, encoding="utf-8")
print(f" Built: {outfile.relative_to(SITE_ROOT)}")
print(f"Done. {len(data)} pages written.")
if __name__ == "__main__":
main()
```
### Rules
1. **Source of truth is JSON, not HTML.** When content needs to change, edit the
JSON and re-run the build script. Never hand-edit a generated `.html` file —
the next build will overwrite your changes.
2. **Generated files land in the same folder as their template.** Do not nest
into a subfolder. The template file is always named `_template.html` (leading
underscore so it sorts above the generated pages).
3. **Build script lives in the SOURCE root**, not in deployment. After running
the build, sync the rendered `.html` files (not the script, not the JSON) to
deployment.
4. **Verify zero unreplaced placeholders** after every build:
```bash
grep -rn "{{" {thing}/*.html # should return nothing
```
5. **Build is idempotent.** Running it twice produces identical files.
### Stamping Rules — Escaping
When a JSON value gets stamped into an HTML attribute or `<title>`, special
characters can break the page. Use these rules:
- Plain text in `<p>` or `<h1>`: ampersand-encode (`&` → `&amp;`)
- `<title>` content: ampersand-encode + strip line breaks
- `<meta>` content attribute: encode `&`, `"`, and remove line breaks
- `href` URL attribute: never put user input here, but if needed, urlencode
For our typical use case (controlled content authored by us), the simple
`str.replace("{{key}}", value)` is sufficient because we don't have hostile
input. Just don't put angle brackets or quotes in the JSON values.
### Re-Running the Build
```bash
cd {project}/site
python3 build_locations.py
python3 build_services.py
```
After build, sync the rendered files to deployment.
---
## WordPress to Static HTML Migration
The playbook for migrating a WordPress (Divi, Elementor, classic, whatever) site
to vanilla static HTML.
### Phase 1 — Capture Source
Before touching anything, capture the current site so nothing is lost.
1. **Database dump** — `wp db export ${domain}.sql --add-drop-table`
2. **Wp-content snapshot** — tar the entire `wp-content/` (themes, plugins, uploads)
3. **Crawl the live site** — use `wget --mirror --convert-links --adjust-extension --page-requisites --no-parent https://{domain}` to capture rendered HTML + all assets
4. **Inventory pages** — list every URL returning 200 (use the sitemap if it has one)
5. **Inventory forms** — note every Gravity Form / Contact Form 7 / etc. field-by-field
6. **Inventory dynamic features** — search, comments, members, anything truly dynamic
Save all of this in the project's `.planning/` folder.
### Phase 2 — Decide What to Keep
Re-design pass. Most WP sites have:
- Bloated copy → cut by 30-50%
- Outdated/inflated metrics → remove or replace with real, verifiable data
- Stock photos → replace with real client photos when available
- Cluttered layouts → strip back to one clear CTA per section
- Plugin features the client never uses → drop entirely
Show the client a wireframe of the simplified structure before building anything.
### Phase 3 — Information Architecture
Standard structure for a small business:
```
/ home
/about/ about / story / team
/services/ services index
/services/{slug}.html one detail page per service
/locations/ locations index
/locations/{city}.html one detail page per service area (SEO gold)
/reviews/ customer reviews
/contact/ contact + form
/blog/ optional blog index
```
For each location and each service: one flat `.html` page generated from JSON +
template.
### Phase 4 — Build
1. Set up source folder per `01-project-structure.md`
2. Write `assets/css/main.css` (variables, reset, typography, layout)
3. Write `assets/css/components.css` (header, footer, hero, cards, forms)
4. Write `components/header.html` and `components/footer.html`
5. Write `assets/js/components.js` (fetch + inject header/footer)
6. Write `assets/js/main.js` (scroll animations, anything page-wide)
7. Build `index.html` first — this is the design system in working form
8. Generate location and service detail pages from JSON
9. Build remaining pages: about, contact, reviews, blog index
### Phase 5 — Forms
If the WP site had Gravity Forms or similar, build a vanilla replacement:
- HTML form in `contact/index.html` (and inline on service/location pages if needed)
- Client-side validation in `assets/js/form.js`
- POST to `/api/estimate` (or similar) handled by Python stdlib service
- Server-side validation, reCAPTCHA verification, send via Resend
### Phase 6 — SEO Parity
Before launch, every old URL must either:
- Have a matching new URL with the same or better content, OR
- 301-redirect to a relevant new URL
Build a redirect map from the old WP sitemap. Add to `nginx.conf`:
```nginx
location = /old-page-slug { return 301 /new-slug.html; }
location = /?p=123 { return 301 /about/; }
```
Per-page parity checklist:
- `<title>` matches or improves on the WP title
- `<meta name="description">` matches or improves
- `<link rel="canonical">` is set to the new URL
- Headings (h1, h2, h3) preserve the topical structure
- Internal links updated to new URLs
- Image alt text preserved or improved
- Schema.org JSON-LD added (`LocalBusiness`, `Service`, `BreadcrumbList`)
### Phase 7 — Switch DNS / Cutover
1. Deploy the static site to a separate URL first (`new.{domain}`) for client review
2. Once approved, point production DNS to the new container
3. Keep the WP container running for 14 days as fallback
4. Submit new sitemap to Google Search Console
5. Use Search Console URL inspection on 5-10 key pages to confirm indexing
### Phase 8 — Post-Launch
- Monitor Search Console for crawl errors / 404s, fix in nginx as redirects
- Monitor form submissions — first real lead through the new form is the
ultimate "it works" check
- Decommission WP only after 30 days of clean operation
### What NOT to Do
- Do not run a "headless WordPress" or "WordPress as API" — that defeats the
whole point. Static means static.
- Do not use a static-site-generator tool (Hugo, 11ty, Jekyll, Astro, Next.js
static export). We hand-write HTML and use small Python build scripts only
where data is repeated.
- Do not migrate the database. Content gets re-written cleaner during migration.
---
## WP + Divi to AM HTML Pipeline Overview
End-to-end playbook for converting a WordPress / Divi site backup (.wpress)
into an Arising Media vanilla HTML + vanilla JS deployment.
### What This Pipeline Does
Takes a single `.wpress` archive (All-in-One WP Migration backup) and produces:
- A fully structured `src/` directory matching AM project layout
- A CSS design system derived from the original Divi theme settings
- All page content extracted, cleaned, and re-authored into AM HTML templates
- All media migrated to WebP and remapped to `/assets/images/`
- SEO metadata (titles, descriptions, canonicals, schema.org) preserved or improved
- Docker-ready deployment with nginx + PHP contact form
### Philosophy
The goal is NOT a 1:1 copy. The goal is:
1. Preserve all content, SEO equity, and brand identity
2. ENHANCE the design — cleaner, faster, more modern
3. Remove all WordPress / Divi bloat (plugin CSS, shortcode residue, 300KB JS bundles)
4. Produce a site that loads in <2s on mobile and scores 95+ on Lighthouse
Every migration is a design upgrade. The Divi site is the reference, not the target.
### Divi Version Matters
Two distinct extraction paths:
| Version | Content Storage | How to detect |
|---------|----------------|---------------|
| Divi 4 | `[et_pb_section]` shortcodes in `wp_posts.post_content` | `post_content` contains `[et_pb_` |
| Divi 5 | Gutenberg blocks (`<!-- wp:divi/section -->`) + JSON in `wp_postmeta` | `post_content` contains `<!-- wp:divi/` |
Run Phase 2 (database analysis) first to determine which version before choosing the extraction path.
### Pipeline Phases
```
Phase 0 Setup Verify .wpress location, create extraction directory
Phase 1 Extract Unpack .wpress binary archive to wpress-extract/
Phase 2 DB Analysis Inspect WordPress database dump, detect Divi version, inventory pages
Phase 3 Content Extract page content via Divi 4 or Divi 5 path
Phase 4 Design System Pull colors, fonts, spacing from wp_options → CSS custom properties
Phase 5 Media Catalog uploads/, convert to WebP, generate image manifest
Phase 6 Build HTML Map extracted content to AM templates, generate JSON data files
Phase 7 SEO Port titles, metas, canonicals, schema.org; build redirect map
Phase 8 Forms Replace Gravity Forms / CF7 with AM vanilla form + Python API
Phase 9 QA Lighthouse audit, grep for unreplaced placeholders, protection check
```
### Script Reference
All scripts live in `.am-webdesign-sops/wp-divi-pipeline/scripts/`.
| Script | Phase | Purpose |
|--------|-------|---------|
| `extract_wpress.py` | 1 | Unpack .wpress binary archive |
| `analyze_db.py` | 2 | Parse SQL dump, inventory pages + detect Divi version |
| `extract_divi4.py` | 3 | Parse et_pb_ shortcodes → structured content JSON |
| `extract_divi5.py` | 3 | Parse Gutenberg/Divi5 blocks → structured content JSON |
| `extract_design.py` | 4 | Pull Divi theme options → design-system.json |
| `extract_media.py` | 5 | Catalog uploads/, emit media-manifest.json |
| `convert_images.py` | 5 | Batch convert images → WebP |
| `run_pipeline.sh` | 0-7 | Master script — runs all phases in order |
### Per-Project Working Directory
```
{domain}/
└── .planning/
├── vibrantyou-yoga-YYYYMMDD-*.wpress ← source archive (never modify)
├── wpress-extract/ ← Phase 1 output (gitignored)
│ ├── package.json ← archive metadata
│ ├── database.sql ← MySQL dump
│ └── uploads/ ← all media (NOT in wp-content/)
├── data/
│ ├── pages.json ← Phase 2 output
│ ├── design-system.json ← Phase 3 output
│ └── media-manifest.json ← Phase 4 output
└── scripts/ ← project-specific overrides if needed
```
### .wpress Extraction Details
The `.wpress` binary format is NOT a standard zip or tar. Custom sequential binary format:
```
[HEADER 4377 bytes] [FILE DATA n bytes] [HEADER] [FILE DATA] ...
```
Header breakdown:
```
Offset Length Field
0 255 Filename (null-padded)
255 14 File size in bytes (ASCII decimal, null-padded)
269 12 mtime unix timestamp (ASCII decimal, null-padded)
281 4096 Relative path (null-padded)
4377 n Raw file bytes (size from header)
```
The archive ends when a header of all null bytes is encountered, or EOF.
Extraction script:
```bash
python3 /home/sirdrez/arisingmedia-websites/.am-webdesign-sops/wp-divi-pipeline/scripts/extract_wpress.py \
/home/sirdrez/arisingmedia-websites/{domain}/.planning/{file}.wpress \
/home/sirdrez/arisingmedia-websites/{domain}/.planning/wpress-extract/
```
### Database Analysis
Parse the WordPress MySQL dump to inventory pages, detect Divi version,
extract design settings, and build the data JSON files.
```bash
python3 /home/sirdrez/arisingmedia-websites/.am-webdesign-sops/wp-divi-pipeline/scripts/analyze_db.py \
{domain}/.planning/wpress-extract/ \
{domain}/.planning/data/
```
Outputs three files into `.planning/data/`:
- `pages.json` — all published pages/posts with content and SEO meta
- `design-system.json` — colors, fonts, Divi settings
- `site-info.json` — domain, plugin list, WP version, Divi version
### Divi 5 Content Extraction
Parse raw Divi page content from `pages.json` into clean, structured HTML
sections ready to map into AM templates.
```bash
python3 /home/sirdrez/arisingmedia-websites/.am-webdesign-sops/wp-divi-pipeline/scripts/extract_divi5.py \
{domain}/.planning/data/pages.json \
{domain}/.planning/data/content/
```
Produces one JSON file per page: `content/{slug}.json`
Key fields in page JSON:
- `slug`: page URL slug
- `title`: page title
- `seo_title`: SEO title (from Rank Math if available)
- `seo_description`: SEO description (from Rank Math if available)
- `sections`: array of content sections with type, background_color, and modules
Map each Divi module type to AM component:
| Divi module | Extract | Map to AM element |
|-------------|---------|-------------------|
| `divi/text` | inner HTML | `<section>`, `<p>`, headings as-is |
| `divi/button` | `text`, `url` | `<a class="btn-primary">` |
| `divi/image` | `src`, `alt`, `title` | `<img>` → rewrite to WebP path |
| `divi/blurb` | icon, title, body | `.am-card` component |
| `divi/testimonial` | quote, author, company | `.am-testimonial` component |
| `divi/video` | `src`, poster | `<video>` or YouTube embed |
| `divi/contact_form` | field list | → replace with AM form |
| `divi/accordion` | Q+A pairs | `<details><summary>` |
| `divi/fullwidth_header` | title, subhead, CTA | hero section |
Strip Divi class/attribute noise using `clean_divi_html()` from `divi_to_html.py`:
```python
from divi_to_html import clean_divi_html, rewrite_internal_links
cleaned = clean_divi_html(raw_html)
cleaned = rewrite_internal_links(cleaned, staging_hosts=("vibrantyou.yoga",))
```
### Design System Extraction
Convert Divi theme settings into AM CSS custom properties.
Input: `design-system.json` produced by `analyze_db.py` with fields:
- `primary_color`: main brand color
- `body_font`: font family name
- `header_font`: heading font name
- `body_font_size`: base font size in px
- `body_line_height`: line height ratio
- `divi_version`: "4" or "5"
- `wp_version`: WordPress version
- `site_url`: domain
- `site_name`: brand name
Never lift the Divi palette 1:1. Use extracted colors as the base and build a
full 5-step scale around the primary hue:
```css
:root {
--color-primary: {extracted-color};
--color-primary-dark: {darken-by-15%};
--color-primary-light: {lighten-by-40%};
--color-surface: #fafafa;
--color-surface-alt: #f0f7f6;
--color-text: #1a1a1a;
--color-text-muted: #5a6e6b;
--color-border: #c8dedd;
--color-white: #ffffff;
/* Fonts */
--font-body: '{body-font}', system-ui, sans-serif;
--font-heading: '{header-font}', Georgia, serif;
/* Modular scale (1.25 ratio) */
--text-xs: 0.75rem; --text-sm: 0.875rem;
--text-base: 1rem; --text-lg: 1.125rem;
--text-xl: 1.25rem; --text-2xl: 1.5rem;
--text-3xl: 1.875rem; --text-4xl: 2.25rem;
--text-5xl: 3rem; --text-6xl: 3.75rem;
/* Spacing scale */
--space-1: 0.25rem; --space-2: 0.5rem; --space-3: 0.75rem;
--space-4: 1rem; --space-5: 1.25rem; --space-6: 1.5rem;
--space-8: 2rem; --space-10: 2.5rem; --space-12: 3rem;
--space-16: 4rem; --space-20: 5rem; --space-24: 6rem;
--space-32: 8rem;
}
```
### Content Migration
Map extracted Divi content into AM HTML templates.
Build order:
1. `src/assets/css/main.css` — design tokens, reset, typography, layout grid
2. `src/assets/css/components.css` — header, footer, hero, cards, forms, nav
3. `src/components/header.html` — navigation
4. `src/components/footer.html` — footer links, contact info
5. `src/assets/js/components.js` — fetch + inject header/footer
6. `src/assets/js/main.js` — scroll animations, intersection observer
7. `src/index.html` — home page (this IS the design system in working form)
8. Remaining pages: about, classes, contact, blog
9. `src/robots.txt`, `src/sitemap.xml`, `src/404.html`, `src/500.html`
For 4+ similar pages (class types, locations), use JSON template build:
```
src/classes/
├── _template.html ← class detail page template
├── hatha.html ← generated from classes.json
├── vinyasa.html
└── yin.html
.planning/data/
└── classes.json ← array of class objects
```
### Media Assets
Migrate WordPress uploads to AM `/assets/images/`, convert to WebP, and
generate a media manifest for URL remapping.
Steps:
1. Catalog all original media (skip WordPress-generated size variants like `-150x150`)
2. Copy originals to `src/assets/images/`
3. Convert to WebP using `cwebp` or Python Pillow
4. Generate media manifest with old → new URL mapping
5. Apply manifest during HTML build to rewrite all image paths
```bash
# Catalog originals (skip WP size variants)
find .planning/wpress-extract/uploads -type f \( -name "*.jpg" -o -name "*.png" \) | \
grep -v -E "\-[0-9]+x[0-9]+\.(jpg|png)$" > .planning/data/media-originals.txt
# Copy and convert
while IFS= read -r src; do
cp "$src" "src/assets/images/$(basename $src)"
done < .planning/data/media-originals.txt
cd src/assets/images/
for img in *.jpg *.png; do
[ -f "$img" ] || continue
cwebp -q 82 "$img" -o "${img%.*}.webp" && rm "$img"
done
```
Remap URLs during HTML build:
```python
import json, re
manifest = json.loads(open('.planning/data/media-manifest.json').read())
url_map = {m['wp_url']: m['am_url'] for m in manifest}
def rewrite_media_urls(html: str) -> str:
for wp_url, am_url in url_map.items():
html = html.replace(wp_url, am_url)
return html
```
### SEO Preservation
Before building HTML, map every WordPress page URL to its new AM URL and
ensure title, description, canonical, and schema.org are preserved or improved.
Rank Math SEO extraction (already in `pages.json` as `seo_title` and `seo_description`).
Priority order for SEO fields:
1. `seo_title` from Rank Math (if not empty and not a template)
2. `post_title` with AM format appended: `{Title} | {Brand Name}`
3. Never leave title as the raw WP default
Rank Math title templates use `%` tokens — strip them and rebuild:
```python
import re
def clean_rm_title(rm_title: str, post_title: str, site_name: str) -> str:
if not rm_title or "%" in rm_title:
return f"{post_title} | {site_name}"
return rm_title
def clean_rm_desc(rm_desc: str) -> str:
return re.sub(r"%[a-z_]+%", "", rm_desc).strip(" -|")
```
Schema.org by page type:
| Page | Schema type | Required fields |
|------|------------|----------------|
| Home | `LocalBusiness` | name, url, telephone, address, areaServed, openingHours |
| About | `AboutPage` + `Organization` | name, description, founders |
| Contact | `ContactPage` | name, url, telephone, email, address |
| Blog post | `Article` | headline, datePublished, author, image |
Pre-launch SEO audit (all must return empty):
```bash
SITE=src
# Every page has title/description/canonical/JSON-LD
find $SITE -name "*.html" | xargs grep -L '<title>'
find $SITE -name "*.html" | xargs grep -L 'name="description"'
find $SITE -name "*.html" | xargs grep -L 'rel="canonical"'
find $SITE -name "*.html" | xargs grep -L 'application/ld+json'
# No WP URLs leaked
grep -r "wp-content\|wp-admin\|?p=\|?page_id=" $SITE --include="*.html"
# No unreplaced placeholders
grep -r "{{" $SITE --include="*.html"
# No Divi residue
grep -r "et_pb_\|wp:divi" $SITE --include="*.html"
```
### Run Order (Complete Execution Sequence)
```bash
export DOMAIN="vibrantyou.yoga"
export PROJECT="/home/sirdrez/arisingmedia-websites/$DOMAIN"
export SOPS="/home/sirdrez/arisingmedia-websites/.am-webdesign-sops"
export WPRESS=$(ls $PROJECT/.planning/*.wpress | head -1)
# Phase 0: Setup
mkdir -p $PROJECT/{src/{about,services,contact,blog,classes,components,assets/{css,js,images,svg,fonts}},build,infra,api,.planning/{data/{content},scripts,wpress-extract}}
# Phase 1: Extract archive
python3 $SOPS/wp-divi-pipeline/scripts/extract_wpress.py "$WPRESS" "$PROJECT/.planning/wpress-extract/"
# Phase 2: Database analysis
python3 $SOPS/wp-divi-pipeline/scripts/analyze_db.py "$PROJECT/.planning/wpress-extract/" "$PROJECT/.planning/data/"
# Phase 3: Content extraction (Divi 5 example)
python3 $SOPS/wp-divi-pipeline/scripts/extract_divi5.py "$PROJECT/.planning/data/pages.json" "$PROJECT/.planning/data/content/"
# Phase 4: Design system (manual — read design-system.json, write main.css)
# Phase 5: Media migration
find $PROJECT/.planning/wpress-extract/uploads -type f \( -name "*.jpg" -o -name "*.png" \) | \
grep -v -E "\-[0-9]+x[0-9]+\.(jpg|png)$" > $PROJECT/.planning/data/media-originals.txt
while IFS= read -r src; do
cp "$src" "$PROJECT/src/assets/images/$(basename $src)"
done < $PROJECT/.planning/data/media-originals.txt
cd $PROJECT/src/assets/images/
for img in *.jpg *.png; do
[ -f "$img" ] || continue
cwebp -q 82 "$img" -o "${img%.*}.webp" && rm "$img"
done
# Phase 6: Build HTML (manual — per 05-content-migration.md)
# Phase 7: SEO audit
cd $PROJECT/src
find . -name "*.html" | grep -v "_template" | xargs grep -L '<title>'
find . -name "*.html" | grep -v "_template" | xargs grep -L 'rel="canonical"'
# Phase 8: Docker setup
docker compose -f $PROJECT/docker-compose.yml build
docker compose -f $PROJECT/docker-compose.yml up -d
curl -I http://localhost:PORT/
# Phase 9: Protection check
bash $SOPS/tools/verify-protection.sh https://$DOMAIN
```
---
## Docker + Nginx Deployment
Every project ships with ALL deployment configs so it can go to either a
Docker VPS or a cPanel shared host without refactoring.
### docker-compose.yml
```yaml
services:
web:
image: {domain}-static
build:
context: .
dockerfile: Dockerfile
ports:
- "{port}:80"
depends_on:
api:
condition: service_healthy
restart: unless-stopped
api:
image: {domain}-api
build:
context: ./api
dockerfile: Dockerfile
env_file: ./api/.env
expose:
- "3001"
healthcheck:
test: ["CMD", "python3", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:3001/health',timeout=3).status==200 else 1)"]
interval: 10s
timeout: 5s
retries: 3
restart: unless-stopped
```
Port assignments are unique per project. Track in
`/home/sirdrez/arisingmedia-websites/PORTS.md` so no two projects collide.
### Dockerfile (nginx web container)
CRITICAL — the Dockerfile must explicitly list which folders to copy. Never use
`COPY . /usr/share/nginx/html/` because that copies `.env`, `Dockerfile`,
build scripts, etc. into the web root where they become URL-accessible.
```dockerfile
FROM nginx:alpine
# nginx config — server-only, never served as a static file
COPY nginx.conf /etc/nginx/conf.d/default.conf
# Public website only — explicit list, no wildcards
COPY index.html /usr/share/nginx/html/
COPY assets /usr/share/nginx/html/assets/
COPY components /usr/share/nginx/html/components/
COPY about /usr/share/nginx/html/about/
COPY blog /usr/share/nginx/html/blog/
COPY contact /usr/share/nginx/html/contact/
COPY locations /usr/share/nginx/html/locations/
COPY reviews /usr/share/nginx/html/reviews/
COPY services /usr/share/nginx/html/services/
EXPOSE 80
```
### Dockerfile (api Python container)
```dockerfile
FROM python:3.13-alpine
WORKDIR /app
COPY server.py .
EXPOSE 3001
CMD ["python3", "-u", "server.py"]
```
No pip, no requirements.txt, no node_modules. Python stdlib only.
### nginx.conf
```nginx
server {
listen 80;
server_name _;
root /usr/share/nginx/html;
index index.html;
# Defense in depth — deny dotfiles, configs, scripts, source files
location ~ /\. {
deny all;
return 404;
}
location ~* \.(env|env\.example|conf|yml|yaml|py|pyc|md|txt|sh|sql|log|bak|old|swp|dockerfile)$ {
deny all;
return 404;
}
location = /Dockerfile {
deny all;
return 404;
}
# API proxy — strip /api/ prefix, forward to Python service
location /api/ {
proxy_pass http://api:3001/;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 10s;
proxy_connect_timeout 5s;
}
# Flat HTML routing — /locations/buffalo serves /locations/buffalo.html
location / {
try_files $uri $uri/ $uri.html =404;
}
# Cache static assets aggressively
location ~* \.(jpg|jpeg|png|webp|svg|ico|css|js|woff2?|mp4|webm)$ {
expires 30d;
add_header Cache-Control "public, immutable";
access_log off;
}
# Security headers
add_header X-Frame-Options "SAMEORIGIN";
add_header X-Content-Type-Options "nosniff";
add_header X-XSS-Protection "1; mode=block";
add_header Referrer-Policy "strict-origin-when-cross-origin";
add_header Permissions-Policy "geolocation=(), microphone=(), camera=()";
add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' https://www.google.com https://www.gstatic.com https://www.recaptcha.net; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; font-src https://fonts.gstatic.com; img-src 'self' data: https:; object-src 'none'; frame-ancestors 'self'; form-action 'self'; base-uri 'self';";
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload";
# Disable server tokens
server_tokens off;
client_max_body_size 16k;
gzip on;
gzip_types text/html text/css application/javascript image/svg+xml;
gzip_min_length 1024;
error_page 404 /404.html;
error_page 500 /500.html;
}
```
### .dockerignore
Keeps sensitive files out of the build context:
```
.git
.gitignore
.dockerignore
api
build_*.py
__pycache__
*.pyc
*.md
*.txt
review_*.png
docker-compose.yml
.DS_Store
.planning
```
### .gitignore
```
api/.env
api/__pycache__/
__pycache__/
*.pyc
*.log
.DS_Store
```
The `api/.env` file is NEVER committed.
### Sync from Source to Deployment
After every change to source HTML/CSS/JS/assets:
```bash
SITE="/path/to/concept-agent/projects/{domain}/site"
DEPLOY="/home/sirdrez/arisingmedia-websites/{domain}"
rsync -a \
--exclude=.git --exclude=.planning --exclude=api \
--exclude=Dockerfile --exclude=nginx.conf --exclude=docker-compose.yml \
--exclude=.dockerignore --exclude=.gitignore \
--exclude='build_*.py' --exclude=__pycache__ --exclude=data \
--exclude='*.md' --exclude='*.txt' --exclude='review_*.png' \
"$SITE/" "$DEPLOY/"
cd "$DEPLOY"
docker compose up -d --build web
```
### Verify After Deploy
Every deploy MUST be audited with `tools/verify-protection.sh` before being
considered live. The script probes a fixed list of sensitive paths
(`Dockerfile`, `.env`, `nginx.conf`, `.planning/`, `__pycache__/`, build
scripts, `.git/`, etc.) and fails if any returns 200.
```bash
~/arisingmedia-websites/.am-webdesign-sops/tools/verify-protection.sh \
http://localhost:{port}
```
Exit codes:
- `0` PASS — every sensitive path 404, every required path reachable.
- `0` PASS (with warnings) — protection clean but `/robots.txt` or
`/sitemap.xml` missing (content gap, not a leak).
- `1` FAIL — at least one sensitive path returned 200, or `/` is unreachable.
Run it manually after every `docker compose up -d --build`. Wire it into CI
once the site has a remote pipeline. Treat a FAIL as a deploy rollback.
For ad-hoc spot checks:
```bash
curl -s -o /dev/null -w "site: %{http_code}\n" http://localhost:{port}/
curl -s -o /dev/null -w "css: %{http_code}\n" http://localhost:{port}/assets/css/main.css
curl -s -o /dev/null -w "api: %{http_code}\n" http://localhost:{port}/api/health
```
All public paths return 200. All sensitive paths return 404.
### Project Folder Rename Procedure
WHY: Docker Compose derives its project name from the folder the
`docker-compose.yml` lives in. Renaming the folder changes the compose project
name, which orphans any running containers under the old name.
The fix is to explicitly remove the old container before bringing up the new
compose project:
```bash
# Stop and remove the old container by its known name
docker stop {container-name}
docker rm {container-name}
# Now bring up from the renamed folder — clean start
docker compose -f /path/to/renamed-folder/docker-compose.yml up -d
```
Always confirm the env vars loaded correctly after restart:
```bash
docker exec {container-name} env | grep RESEND
```
---
## cPanel + Apache Deployment
Use this deployment method when the client's host is cPanel-based (shared hosting,
WHM, Bluehost, HostGator, SiteGround, etc.) instead of a VPS running Docker.
### Key Rule: Repo Path ≠ Webroot
cPanel Git requires an EMPTY directory as the repository path. The webroot
(`public_html/{domain}/`) is never the repo path — cPanel rejects it if it
already contains files.
```
Repo path (empty dir): /home/{username}/repositories/{domain}/
Deploy target (webroot): /home/{username}/public_html/{domain}/
```
### Setting Up the Repo in cPanel
1. cPanel → Git Version Control → Create Repository
2. Repository Path: `/home/{username}/repositories/{domain}/` (must be empty)
3. Clone URL: your Git remote (GitHub, Bitbucket, etc.)
4. cPanel clones into the repo path — never into the webroot
### .cpanel.yml
This file lives in the repo root and tells cPanel what to copy to the webroot
on every push/deploy. All paths are relative to the repo root.
```yaml
---
deployment:
tasks:
- export DEPLOYPATH=/home/{username}/public_html/{domain}/
- /bin/cp -r assets $DEPLOYPATH
- /bin/cp -r about $DEPLOYPATH
- /bin/cp -r commercial $DEPLOYPATH
- /bin/cp -r contact $DEPLOYPATH
- /bin/cp -r locations $DEPLOYPATH
- /bin/cp -r reviews $DEPLOYPATH
- /bin/cp -r service-area $DEPLOYPATH
- /bin/cp -r services $DEPLOYPATH
- /bin/cp index.html $DEPLOYPATH
- /bin/cp 404.html $DEPLOYPATH
- /bin/cp robots.txt $DEPLOYPATH
- /bin/cp sitemap.xml $DEPLOYPATH
```
Add or remove folder cp lines to match the project's actual directory structure.
Do NOT copy: `tools/`, `*.py`, `*.md`, `.git/`, `docker-compose.yml`, `Dockerfile`.
### Lahrcarpetcleaning.com Reference
```yaml
---
deployment:
tasks:
- export DEPLOYPATH=/home/dev1communitypro/public_html/lahrcarpetcleaning.dev1.communityproud.com/
- /bin/cp -r assets $DEPLOYPATH
- /bin/cp -r about $DEPLOYPATH
- /bin/cp -r commercial $DEPLOYPATH
- /bin/cp -r contact $DEPLOYPATH
- /bin/cp -r locations $DEPLOYPATH
- /bin/cp -r reviews $DEPLOYPATH
- /bin/cp -r service-area $DEPLOYPATH
- /bin/cp -r services $DEPLOYPATH
- /bin/cp index.html $DEPLOYPATH
- /bin/cp 404.html $DEPLOYPATH
- /bin/cp robots.txt $DEPLOYPATH
- /bin/cp sitemap.xml $DEPLOYPATH
```
### Deploying After a Push
1. Push to the connected remote (GitHub)
2. cPanel → Git Version Control → Manage → Pull or Deploy
3. cPanel runs the `.cpanel.yml` tasks, copying files to webroot
4. Apache serves from webroot automatically — no nginx, no Docker
### Apache vs nginx
cPanel hosts use Apache (not nginx). There is no nginx.conf to manage.
URL routing is handled by `.htaccess`:
```apache
Options -Indexes
RewriteEngine On
# Directory-style URLs: /services/carpet-cleaning/ → index.html inside that folder
# Apache handles this automatically with DirectoryIndex — no extra rules needed
# Deny sensitive files
<FilesMatch "\.(py|yml|yaml|md|log|sh|env|conf|dockerfile)$">
Order allow,deny
Deny from all
</FilesMatch>
# Security headers
<IfModule mod_headers.c>
Header set X-Frame-Options "SAMEORIGIN"
Header set X-Content-Type-Options "nosniff"
Header set X-XSS-Protection "1; mode=block"
Header set Referrer-Policy "strict-origin-when-cross-origin"
Header set Permissions-Policy "geolocation=(), microphone=(), camera=()"
Header set Strict-Transport-Security "max-age=31536000; includeSubDomains"
</IfModule>
ErrorDocument 404 /404.html
ErrorDocument 500 /500.html
```
### Cache Busting on cPanel
Apache does not auto-invalidate cached assets. Bump `?v=N` on CSS/JS in
all HTML files after every asset change:
```html
<link rel="stylesheet" href="/assets/css/styles.css?v=6">
<script src="/assets/js/main.js?v=3"></script>
```
Increment by 1 on every change. Apply across ALL HTML pages.
### Verify After cPanel Deploy
```bash
curl -s -o /dev/null -w "home: %{http_code}\n" https://{domain}/
curl -s -o /dev/null -w "css: %{http_code}\n" https://{domain}/assets/css/styles.css
curl -s -o /dev/null -w "404: %{http_code}\n" https://{domain}/page-that-does-not-exist
```
All public paths return 200. All non-existent paths return 404.
### Universal Project Checklist (Both Paths)
Every project must include ALL of these before first deploy:
```
Dockerfile ✓ Docker/VPS
docker-compose.yml ✓ Docker/VPS
nginx.conf ✓ Docker/VPS
.htaccess ✓ cPanel/Apache
.cpanel.yml ✓ cPanel Git
.dockerignore ✓ Docker build security
.gitignore ✓ keeps .env and secrets out of git
robots.txt ✓ both paths
sitemap.xml ✓ both paths
404.html ✓ both paths
500.html ✓ both paths
```
Lahrcarpetcleaning.com is the reference implementation for both paths.
---
## Domain, Email, DNS, and Resend
### Resend Account Setup
1. Sign up at https://resend.com
2. Generate an API key (one per project): https://resend.com/api-keys
3. Save the key in the project's `api/.env` as `RESEND_API_KEY=re_xxxx`
4. NEVER commit `.env`. NEVER paste the key in Slack, GitHub, or chat logs.
### Add and Verify the Sending Domain
1. https://resend.com/domains → **Add Domain**
2. Enter the domain (the one you'll send FROM, not necessarily the website domain)
3. Resend gives 3-4 DNS records. Add them all in Cloudflare (or whatever DNS host)
4. Wait 5-15 minutes, click **Verify** in Resend — all records must show green
### Records Resend Provides
| Type | Name | Value | Proxy | TTL |
|------|------|-------|-------|-----|
| TXT | `resend._domainkey` | `p=...long-rsa-key...` | DNS only | 1 hr |
| TXT | `send` | `v=spf1 include:amazonses.com ~all` | DNS only | 1 hr |
| MX | `send` | `feedback-smtp.{region}.amazonses.com` priority 10 | DNS only | 1 hr |
(Resend uses Amazon SES under the hood, hence `amazonses.com` in the SPF.)
### DMARC — REQUIRED for Inbox Placement
Without DMARC, Gmail flags otherwise-correctly-configured email as suspicious
and routes it to spam. Resend doesn't auto-create this record. You must add it.
| Type | Name | Value | Proxy | TTL |
|------|------|-------|-------|-----|
| TXT | `_dmarc` | `v=DMARC1; p=none; rua=mailto:dev@{domain}` | DNS only | Auto |
Components:
- `v=DMARC1` — declares a DMARC policy exists
- `p=none` — monitor mode, doesn't reject anything yet (safe to start)
- `rua=mailto:...` — DMARC failure reports go to this inbox (review weekly)
After 30 days of clean DMARC reports with no false positives, optionally
upgrade to `p=quarantine` then `p=reject`.
### Verify DNS is Live
```bash
dig +short TXT resend._domainkey.{domain} @8.8.8.8
dig +short TXT send.{domain} @8.8.8.8
dig +short TXT _dmarc.{domain} @8.8.8.8
dig +short MX send.{domain} @8.8.8.8
```
All four should return their expected values.
### From-Name Format
Always use a friendly From name, not bare email. Bare email looks robotic
and triggers spam filters.
```
FROM_EMAIL=Brand Name <webleads@{domain}>
```
### TO-Email Setup
The `TO_EMAIL` is wherever the lead actually goes. Often a Gmail group address
or the owner's personal inbox.
- During Resend domain verification (BEFORE green): you can ONLY send TO the
email tied to the Resend account
- After verification: send to anyone
For local testing without verification, use:
```
FROM_EMAIL=onboarding@resend.dev
TO_EMAIL={your-resend-account-email}
```
### When Emails Go to Spam
Run this checklist:
1. **All 4 DNS records green at Resend**? If not, deliverability suffers.
2. **DMARC TXT record exists**? Most common cause of spam folder.
3. **Friendly From name**? `Brand Name <webleads@...>` not bare `webleads@...`
4. **Both `html` and `text` parts in the payload**? HTML-only is suspicious.
5. **Subject line clean**? No em-dashes, no "Estimate Request URGENT", no all-caps.
6. **Recipient marked first emails as Not Spam**? Train Gmail.
### Cloudflare-Specific Notes
The user-agent quirk — Cloudflare in front of Resend's API blocks Python's default
`User-Agent: Python-urllib/3.x`. Always set a custom `User-Agent` in the API request headers.
If the DNS provider is Cloudflare, ensure all Resend records have **proxy status: DNS only**
(the gray cloud icon, not orange). Proxying these breaks authentication.
### Annual Key Rotation
Rotate Resend API keys annually:
1. Generate new key in Resend dashboard
2. Update `api/.env` on the server
3. `docker compose down && docker compose up -d` to reload env
4. Confirm a test submission still works
5. Revoke the old key in Resend dashboard
### Resend HTTP 403 — Domain Not Verified
A 403 from the Resend API does NOT mean the API key is wrong. The specific
error is:
```json
{"statusCode":403,"message":"The {domain} domain is not verified. Please, add and verify your domain on https://resend.com/domains","name":"validation_error"}
```
This means the key is valid and authenticated, but the FROM domain has not
been added or verified at resend.com/domains yet.
Rule: **verify the domain BEFORE testing the form endpoint.** If you test
before verification, `{"ok":false}` will be returned to the visitor even
though the API key is correct and the code is correct.
Sequence:
1. Set `RESEND_API_KEY` in `.env`
2. Add domain at resend.com/domains
3. Add DNS records in Cloudflare
4. Wait for green verification
5. Then test the form endpoint
### DKIM Key Rotation
Resend periodically rotates DKIM keys. They send email when this happens. Add
the new `resend2._domainkey` (or whichever selector they specify) TXT record
in Cloudflare, then click verify. Old key remains active until they remove it.
---
## Form Handling — Resend
Static sites can't send email by themselves. Every project that needs a
contact form gets a small Python service running in its own Docker container,
proxied by nginx.
### Architecture
```
Browser → POST /api/estimate (vanilla JS fetch in form.js)
↓
nginx → proxies /api/ to api:3001 (strips /api/ prefix)
↓
Python service (server.py, stdlib only)
- Validates fields server-side
- Verifies reCAPTCHA v3 with Google
- Sends via Resend HTTPS API
- Returns {ok: true} or {error: ...}
```
### Front-End (Vanilla JS)
`assets/js/form.js`:
- Real-time validation (blur events)
- Phone formatting `(###) ###-####`
- Email regex check
- Required-field check
- Async submit to `/api/estimate` with JSON body
- Disable submit button + show "Sending..." during request
- Show success/error message in `.form-status` span
- Reset form on success
- reCAPTCHA v3 token fetched before submit and included in body
### Back-End (Python stdlib)
`api/server.py` (skeleton):
```python
#!/usr/bin/env python3
import hashlib, http.server, json, os, re, socketserver, time
import urllib.parse, urllib.request
PORT = int(os.environ.get("PORT", "3001"))
RESEND_API_KEY = os.environ.get("RESEND_API_KEY", "")
RECAPTCHA_SECRET = os.environ.get("RECAPTCHA_SECRET", "")
TO_EMAIL = os.environ.get("TO_EMAIL", "")
FROM_EMAIL = os.environ.get("FROM_EMAIL", "")
RECAPTCHA_MIN = float(os.environ.get("RECAPTCHA_MIN", "0.5"))
PHONE_RE = re.compile(r"^\(?\d{3}\)?[\s.\-]?\d{3}[\s.\-]?\d{4}$")
EMAIL_RE = re.compile(r"^[^\s@]+@[^\s@]+\.[^\s@]+$")
# Rate limit: 5 requests / IP / 15 minutes
RATE_MAP = {}
RATE_WINDOW = 15 * 60
RATE_MAX = 5
def sanitize(s):
if not isinstance(s, str): return ""
return s.replace("&","&amp;").replace("<","&lt;").replace(">","&gt;").replace('"',"&quot;").strip()[:2000]
def validate_fields(body):
errors = []
if not body.get("name") or len((body["name"]).strip()) < 2: errors.append("name")
if not EMAIL_RE.match((body.get("email") or "").strip()): errors.append("email")
if not PHONE_RE.match((body.get("phone") or "").replace(" ", "")): errors.append("phone")
return errors
def verify_recaptcha(token):
if not RECAPTCHA_SECRET or not token: return 0.0
data = urllib.parse.urlencode({"secret": RECAPTCHA_SECRET, "response": token}).encode()
req = urllib.request.Request("https://www.google.com/recaptcha/api/siteverify", data=data)
try:
with urllib.request.urlopen(req, timeout=8) as resp:
return float(json.loads(resp.read()).get("score", 0))
except Exception:
return 0.0
def send_via_resend(fields):
safe = {k: sanitize(fields.get(k,"")) for k in ["name","email","phone","address","city","zip","service","condition","message"]}
html = f"""<!DOCTYPE html>...{safe['name']}..."""
text = f"New estimate request\n\nName: {safe['name']}\n..."
payload = json.dumps({
"from": FROM_EMAIL,
"to": [TO_EMAIL],
"reply_to": fields.get("email","").strip(),
"subject": f"New estimate request: {safe['name']} ({safe['city']})",
"html": html, "text": text,
}).encode("utf-8")
idem = hashlib.sha256(payload).hexdigest()[:64]
req = urllib.request.Request("https://api.resend.com/emails", data=payload, headers={
"Authorization": f"Bearer {RESEND_API_KEY}",
"Content-Type": "application/json",
"Idempotency-Key": idem,
"User-Agent": "{Brand}-Estimate-Form/1.0",
})
try:
with urllib.request.urlopen(req, timeout=10) as resp:
if resp.status >= 300: raise RuntimeError(f"Resend {resp.status}: {resp.read().decode('utf-8','ignore')}")
except urllib.error.HTTPError as e:
raise RuntimeError(f"Resend {e.code}: {e.read().decode('utf-8','ignore')}") from None
```
Reference implementation: `floorithardwoodfloors.com/api/server.py`.
### Critical: User-Agent Header
When calling the Resend API from Python, you MUST set a non-default User-Agent.
Cloudflare (which fronts Resend) blocks Python's default `Python-urllib/3.x`
with HTTP 403 / Cloudflare error code 1010.
```python
"User-Agent": "{ProjectName}-Form/1.0"
```
### Idempotency
Every Resend request includes an `Idempotency-Key` header set to the SHA-256
of the payload (truncated to 64 chars). Identical payloads within 24 hours
are deduplicated by Resend automatically. This prevents:
- Double-clicks creating two leads
- Browser retries after a network blip
- Honest user submitting twice
### Security Checklist
- API key in `.env` file, NOT in source control. `.gitignore` it.
- API key NEVER reaches the browser bundle (only the server has it)
- `.env` file lives in `api/`, NOT in the nginx web root
- Server-side validation on EVERY field — never trust client
- HTML-escape every field rendered into the email body to prevent injection
- Rate limit per IP (5 / 15 min default)
- 16 KB body cap — reject anything larger
- 10-second upstream timeout — don't hold connections open
- CORS locked to the production domain only (`Access-Control-Allow-Origin: https://{domain}`)
- reCAPTCHA v3 with score threshold (default 0.5) once secret is configured
### Environment Variables
`api/.env`:
```
RESEND_API_KEY=re_xxxxxxxxxxxx
RECAPTCHA_SECRET=6Ldq...
TO_EMAIL=leads@{domain}
FROM_EMAIL=Brand Name <webleads@{domain}>
RECAPTCHA_MIN=0.5
PORT=3001
```
`api/.env.example` (committed) is the same file with placeholder values.
### reCAPTCHA Setup
1. Create site at https://www.google.com/recaptcha/admin
2. Type: **reCAPTCHA v3** (not v2)
3. Add your domain
4. Copy the **site key** into `assets/js/form.js`:
```js
const RECAPTCHA_SITE_KEY = '6Ldq...';
```
5. Add the script tag to pages with the form:
```html
<script src="https://www.google.com/recaptcha/api.js?render=6Ldq..."></script>
```
6. Copy the **secret key** into `api/.env` as `RECAPTCHA_SECRET`
### Deliverability Checklist
When emails are landing in spam:
1. Verify Resend domain is fully green (SPF + DKIM + DMARC)
2. From name set, not bare email: `Brand Name <webleads@{domain}>`
3. Both `html` and `text` parts in every Resend payload (no HTML-only)
4. Subject line is descriptive, no em-dash, no spam-trigger words
5. Recipient marks first 2-3 emails as "Not Spam" in Gmail to train the filter
### Testing
```bash
# Validation rejection (expect 422)
curl -X POST http://localhost:8096/api/estimate \
-H "Content-Type: application/json" \
-d '{"name":"","email":"bad"}'
# Full valid submission (expect 200, real email sent)
curl -X POST http://localhost:8096/api/estimate \
-H "Content-Type: application/json" \
-d '{"name":"Test","email":"test@example.com","phone":"(716) 555-1234","address":"100 Test St","city":"Buffalo","zip":"14201","service":"refinishing","message":"Test","token":""}'
```
The first real test email confirms end-to-end works.
---
## PHP App Stack (Server-Side Processing)
Use this pattern when a project requires server-side processing that static HTML cannot handle: file conversion, at-rest encryption, payment processing, user authentication, or API-gated features.
**Reference implementation:** `quickconvert.us`
### When to Use This Pattern
- File uploads and processing (image conversion, PDF generation, etc.)
- At-rest encryption of user data
- Payment processing with Stripe subscriptions
- User authentication with magic link or password-based login
- Rate-limited APIs that must be server-enforced
**Do not** introduce this pattern just to add a contact form. Use the Python stdlib form service instead.
### Stack
- **PHP 8.3** (php:8.3-fpm-alpine base image)
- **Nginx** (Alpine package, same container via supervisord)
- **SQLite** (pdo_sqlite extension, no separate DB container needed)
- **libsodium** (built into PHP 8.x — use for all encryption)
- **ImageMagick** (pecl imagick for image processing)
- **msmtp** (SMTP relay for outbound email)
- **supervisord** (manages nginx + php-fpm + crond in one container)
### Project Structure
```
project/
├── src/ ← nginx document root
│ ├── index.php
│ ├── api/
│ │ ├── convert.php ← POST endpoint (CSRF + reCAPTCHA protected)
│ │ └── download.php ← GET endpoint (signed token)
│ ├── assets/css/
│ ├── assets/js/
│ └── assets/images/
├── includes/ ← PHP classes (above doc root, not web-accessible)
│ ├── bootstrap.php ← constants, session, autoload
│ ├── auth.php ← login, register, magic token
│ ├── csrf.php
│ ├── db.php ← SQLite PDO wrapper
│ ├── encryption.php ← libsodium wrappers
│ └── mailer.php
├── components/
│ ├── header.php
│ └── footer.php
├── storage/ ← volume-mounted, NOT in docker image
│ ├── uploads/ ← encrypted .enc files only
│ ├── converted/
│ ├── temp/
│ ├── .htaccess ← deny all direct access
│ └── {app}.db
├── infra/
│ ├── nginx.conf
│ ├── php.ini
│ ├── supervisord.conf
│ └── docker-entrypoint.sh
├── tools/
│ └── cleanup.php ← cron: delete expired tokens + files
├── Dockerfile
├── docker-compose.yml
└── .env ← gitignored, never committed
```
### Security Requirements (Non-Negotiable)
**CSRF** — every POST form and API endpoint must verify a CSRF token tied to the session.
**Rate limiting** — two layers:
1. nginx: `limit_req_zone` on /api/ (10 req/s, burst 20)
2. PHP: per-IP daily counter in SQLite rate_limits table
**reCAPTCHA v3** — on conversion/upload endpoints. Verify server-side via Google API. Cache result in session (verify once per session, not per request).
**At-rest encryption** — any user-uploaded file must be encrypted before writing to disk. Use `sodium_crypto_secretstream_xchacha20poly1305_*` for files, `sodium_crypto_secretbox` for strings. Key stored in `.env` as `QC_ENCRYPTION_KEY` (32 bytes hex).
**Signed download tokens** — never expose file paths. Issue a 64-char hex token stored in SQLite with expiry and single-use enforcement.
**Magic link auth** — prefer magic link over password. On register: create account unverified, send verify email, block login until verified. Token: 64-char hex, 1-hour expiry, stored in `magic_tokens` table, consumed on use.
### Nginx Security Headers
```nginx
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Permissions-Policy "camera=(), microphone=(), geolocation=()" always;
add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' https://www.google.com https://www.gstatic.com; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; object-src 'none'; base-uri 'self'; form-action 'self' https://checkout.stripe.com;" always;
# Stripe webhook — POST only
location = /api/stripe-webhook.php {
limit_except POST { deny all; }
}
# Block dotfiles
location ~ /\. { deny all; return 403; }
```
### Database Schema Pattern (SQLite, Idempotent)
Use `CREATE TABLE IF NOT EXISTS` for all tables. Use `ALTER TABLE ... ADD COLUMN` wrapped in try/catch for schema migrations.
```php
try { $pdo->exec("ALTER TABLE users ADD COLUMN verified_at INTEGER DEFAULT NULL"); }
catch (Throwable $e) { /* column already exists */ }
```
### Stripe Integration
- Checkout: create session server-side, redirect to Stripe-hosted page
- Webhook: verify `Stripe-Signature` header using HMAC-SHA256 (implement without Stripe SDK — use curl)
- Webhook tolerance: 300 seconds (5 min) on timestamp
- Register webhook endpoint at: `https://{domain}/api/stripe-webhook.php`
- Events to subscribe: `checkout.session.completed`, `customer.subscription.created`, `customer.subscription.updated`, `customer.subscription.deleted`, `invoice.payment_succeeded`, `invoice.payment_failed`
### .env Required Vars
```
APP_ENV=production
BASE_URL=https://{domain}
QC_ENCRYPTION_KEY={32-bytes-hex}
STRIPE_MODE=live
STRIPE_LIVE_SECRET_KEY=sk_live_...
STRIPE_LIVE_PUBLISHABLE_KEY=pk_live_...
STRIPE_WEBHOOK_SECRET=whsec_...
STRIPE_PRICE_ID=price_...
RECAPTCHA_SITE_KEY=...
RECAPTCHA_SECRET_KEY=...
SMTP_HOST=...
SMTP_PORT=587
SMTP_USER=...
SMTP_PASS=...
MAIL_FROM=noreply@{domain}
MAIL_FROM_NAME={Brand}
```
Generate encryption key: `php -r "echo bin2hex(random_bytes(32));"`
Reference in New Issue View Git Blame Copy Permalink