Live Web Application: https://elonuziel.github.io/Behatsdaa-site-ai-mesh/
Frontend Web Repository: elonuziel/Behatsdaa-site-ai-mesh
An automated data extraction suite, anti-bot bypass pipeline, and data normalization engine for the Behatsdaa club catalog:
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Behatsdaa-stores-list (This Repository) β
β - Python 3.10+ & Playwright scraping suite β
β - Imperva Incapsula WAF & anti-bot bypass β
β - Normalizes & exports verified data/ (*.json, *.csv) β
βββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββ
β git push (data/** changes)
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β π€ GitHub Action (.github/workflows/sync-data.yml) β
β - Automatically copies data/ to Behatsdaa-site-ai-mesh β
β - Commits & pushes with SYNC_DATA_PAT β
βββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββ
β triggers deploy workflow
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Behatsdaa-site-ai-mesh (Production Web App) β
β - Vite + Tailwind CSS + MiniSearch β
β - Card & Table views across all 3 tabs β
β - Automated GitHub Pages build & deployment β
β π https://elonuziel.github.io/Behatsdaa-site-ai-mesh/ β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
When you update data locally and push to main in this repository:
git add data/
git commit -m "chore(data): refresh catalog"
git push origin main
The GitHub Action automatically syncs the datasets into Behatsdaa-site-ai-mesh, which rebuilds and deploys the live site.
Behatsdaa-stores-list/
βββ scraper.py # Unified Playwright scraper for cards & rotating deals
βββ scrape_beplus.py # High-speed API scraper for Be-Plus billing discounts (10,600+ stores)
βββ extract_behatsdaa_deals.js # In-browser JS extractor for logged-in sessions (1,700+ deals)
βββ requirements.txt # Python dependencies (playwright, requests, etc.)
βββ data/ # Master production datasets (JSON & Excel CSV)
β βββ stores.json # 980+ store chains across 8 rechargeable wallets
β βββ stores.csv # Excel-compatible stores CSV
β βββ deals.json # 1,730+ rotating deals, vouchers & holiday specials
β βββ deals.csv # Excel-compatible deals CSV
β βββ billing_stores.json # 10,600+ Be-Plus statement discounts
β βββ billing_stores.csv # Excel-compatible billing discounts CSV
βββ tests/
β βββ test_scraper.py # Unit tests for Behatsdaa scraper logic & normalization
β βββ test_scrape_beplus.py # Unit tests for Be-Plus scraper & CRM API mapping
β βββ test_data_integrity.py # Schema validation & cross-linking integrity tests
βββ .github/workflows/
β βββ sync-data.yml # Automated sync workflow pushing data/ to site repo
βββ README.md # Documentation and scraper usage guide
[!NOTE] Why Scraping is Executed Locally / On-Demand:
- Behatsdaa WAF: Behatsdaaβs backend is protected by Imperva Incapsula bot mitigation, which blocks datacenter IP ranges (including GitHub Actions runners) and requires SMS / authenticated session verification.
- On-Demand Updates: Store catalogs and Be-Plus billing discounts do not change with every git commit, so running scrapers locally on demand prevents wasted runs and guarantees 100% data integrity.
- Freshness Transparency: Each tab on the live website independently displays the exact date it was last scraped.
Three extraction workflows are supported:
The simplest and most reliable method to capture the live catalog directly from your standard browser (Chrome, Edge, Brave). Because it runs inside your existing authenticated session, it requires zero browser setup or GUI in WSL and bypasses all Imperva WAF / Cloudflare protections instantly.
[!TIP] WSL Users & Downloads Folder: Files downloaded from Chrome/Edge in Windows land in your Windows Downloads folder, which is directly accessible in WSL at:
/mnt/c/Users/<YourWindowsUsername>/Downloads/cards_raw.json /mnt/c/Users/<YourWindowsUsername>/Downloads/deals_raw.jsonThe interactive menu (
./scraper.py) automatically searches/mnt/c/Users/*/Downloads/so you can just press Enter!
extract_behatsdaa_cards.js):F12 $\rightarrow$ Console.extract_behatsdaa_cards.js and press Enter to download cards_raw.json.python3 scraper.py --import-cards /mnt/c/Users/<YourUsername>/Downloads/cards_raw.json
extract_behatsdaa_deals.js):F12 $\rightarrow$ Console.extract_behatsdaa_deals.js and press Enter to download deals_raw.json.python3 scraper.py --import-deals /mnt/c/Users/<YourUsername>/Downloads/deals_raw.json
scraper.py) & Interactive Menu π€Launch the interactive CLI menu:
./scraper.py
(Or pass --menu)
==============================================================
π³ Behatsdaa - Scraper & Data Pipeline Suite
==============================================================
Select an action to perform:
[1] π³ Scrape Rechargeable Cards (Playwright headless - terminal login)
[2] π Scrape Rotating Deals (Playwright headless - terminal login)
[3] π Full Scrape: Cards + Deals (Playwright headless - terminal login)
[4] π₯οΈ Manual Browser Scrape (Playwright headful - enter in browser window)
[5] π·οΈ Scrape Be-Plus Billing Discounts (10,600+ stores, no browser)
[6] π₯ Import Cards from file (cards_raw.json)
[7] π₯ Import Deals from file (deals_raw.json)
[8] π§ Check & Install Requirements (Playwright & Chromium)
[0] β Exit
==============================================================
./scraper.py --check-deps or selecting Option [8] checks Python dependencies and Playwrightβs Chromium browser.playwright or the Chromium browser binary is missing when running any scraping option, the script prompts you ([Y/n]) and installs them automatically (including handling Ubuntu PEP 668 --break-system-packages), then immediately proceeds without failing!--headless):
Prompts for your Israeli ID (or reads --id) and the SMS OTP verification code directly in your terminal, enters them automatically, and saves cookies to ./behatsdaa_profile.--manual-login or Option 4):
Opens a visible Chrome/Edge browser window on screen, allows you to enter your ID and SMS code directly in the browser GUI, and resumes once you press Enter in the terminal.scrape_beplus.py) π³A standalone, high-performance scraper for the Be-Plus network (10,600+ participating businesses giving direct statement discounts on Behatsdaa credit cards):
# Run the complete Be-Plus scraper across all 445 pages (10,600+ businesses):
python scrape_beplus.py --output-dir data
# Run a quick test on the first 3 pages:
python scrape_beplus.py --max-pages 3
/index.php?option=com_crm&task=products.getItems) with category tree mapping.data/billing_stores.json and data/billing_stores.csv.Run the full Python test suite to verify scrapers, data parsers, and dataset schema integrity:
python -m unittest discover tests
Created for the benefit of Israeli reserve soldiers (Miluim) and Behatsdaa club beneficiaries. Brand names, logos, and terms are property of Behatsdaa and Be-Plus.