Joomla → BroadcastAV migration guide

The big picture

This is a one-way ETL job: Extract from Joomla's MySQL, Transform each row into our shape, Load into MongoDB. We never touch the live Joomla site and never write straight to production until the script has been proven on a throwaway database.

0 Identify where Joomla keeps the listings

Joomla has no built-in "products" table — the data lives in whichever extension the old site used. Everything later depends on this answer, so do it first.

Run in phpMyAdmin on the old site (replace prefix with the real table prefix, e.g. jos_ or a random one like x7k2p_):

SHOW TABLES;

SELECT name, element
FROM   prefix_extensions
WHERE  type = 'component' AND enabled = 1;

Users are always in prefix_users regardless of the extension.

1 Take a copy — never work against the live site

1. Database — cPanel → phpMyAdmin → Export → SQL format, or on the server:

   mysqldump -u USER -p DBNAME > joomla.sql

2. Images — download as a zip via FTP / File Manager. Usually /images/ plus the extension's own folder, e.g. /components/com_djclassifieds/images/ or /media/djclassifieds/.
3. Old URLs — save sitemap.xml or export pages from Google Search Console. Needed for redirects in step 7.

2 Restore the dump locally

Run MySQL in Docker on port 3307 so it doesn't clash with anything else:

docker run -d --name joomla-mysql -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=joomla -p 3307:3306 mysql:8
docker exec -i joomla-mysql mysql -uroot -proot joomla < joomla.sql

Open it with DBeaver / HeidiSQL / MySQL Workbench (host localhost, port 3307) and note the columns of the item, category, image and user tables:

SHOW CREATE TABLE prefix_djcf_items;
SELECT COUNT(*) FROM prefix_djcf_items;
SELECT published, COUNT(*) FROM prefix_djcf_items GROUP BY published;

3 Field mapping — decide before writing code

This is where most of the real decisions live. Column names below assume DJ-Classifieds (the typical case) — confirm them against your own dump from step 2.

4 One small schema change: legacyId

Add this field to Category, User and Listing:

legacyId: { type: Number, index: true, sparse: true },

The script then upserts by legacyId instead of blindly inserting. That gives you:

• Safe re-runs — running twice updates, never duplicates.
• Fix-and-retry — find a mapping bug, fix it, run again.
• Relationships — a listing finds its category / seller's new _id via the old id.
• Redirects — old URL ids map straight to new slugs (step 7).

5 The migration script

Lives in scripts/migrate-joomla/, follows the pattern of scripts/create-admin.ts, runs with tsx, and uses mysql2 (dev dependency) to read. It imports our real Mongoose models so schema validation and the image path setters still apply.

Load order (parents before children)

1. Categories — sorted by depth so parents exist first; build a legacyId → _id map.
2. Users — bulkWrite upserts with rewritten hashes (never .save()).
3. Listings — resolve categoryId / sellerId from the maps; set createdAt from Joomla's date.
4. Images — copy files, set paths, cap at 5.
5. Recompute Category.totalProducts with one aggregation at the end — not the per-insert $inc helpers, which would double-count on re-runs.
6. Report — rows read / inserted / updated / skipped with reason. Skipped rows go to a CSV for the client. Nothing is dropped silently.

Shape of the script

// scripts/migrate-joomla/index.ts  (sketch)
const DRY_RUN = process.argv.includes('--dry-run');

const mysql = await createConnection(process.env.JOOMLA_MYSQL_URL!);
await mongoose.connect(process.env.MONGODB_URL!);

const report = new MigrationReport();              // counts + skipped rows → CSV

const categoryMap = await migrateCategories(mysql, report, DRY_RUN);
const userMap     = await migrateUsers(mysql, report, DRY_RUN);
await migrateListings(mysql, { categoryMap, userMap }, report, DRY_RUN);
await recountCategoryTotals(DRY_RUN);

report.print();
report.writeSkippedCsv('migration-skipped.csv');

# run
npx tsx scripts/migrate-joomla/index.ts --dry-run   # everything except writes
npx tsx scripts/migrate-joomla/index.ts             # for real

6 Test on a throwaway database first

Point MONGODB_URL at a local or staging copy — never production for the first runs. Then verify:

7 Keep old links working (easy to forget)

Google has indexed the Joomla URLs. If they 404 on launch day, search ranking and traffic drop.

1. From the migrated data, generate a JSON map old URL → new URL (you know both via legacyId → slug).
2. Add them as 301 permanent redirects in the Next.js frontend — redirects() in next.config for a small list, or middleware reading the JSON map for a large one.
3. After launch, watch 404 logs and add anything missed.

/component/djclassifieds/?view=item&id=531   →  /new-ads/3-microphones/531-sony-ecm-w3
/classifieds/microphones/531-sony-ecm-w3      →  /new-ads/3-microphones/531-sony-ecm-w3

8 Cutover day

1. Announce a freeze; put Joomla in offline / read-only mode so no new data appears mid-migration.
2. Take the final dump + images (repeat step 1).
3. Back up production Mongo first: mongodump --uri "$MONGODB_URL" --out backup-before-migration.
4. Run the script against production; review the report.
5. Switch DNS to the new platform.
6. Watch 404 logs and error logs for the first days.
7. Archive the Joomla dump for a few months, then delete it (it holds personal data).

Progress checklist

• Identified the Joomla extension and table prefix
• Downloaded SQL dump, images zip and old URL list
• Restored dump in local MySQL (Docker)
• Shared table structures + sample rows (personal data removed)
• Mapping sheet agreed, including all "Decision" items
• Added legacyId to Category, User, Listing
• Migration script written with --dry-run and report
• Dry run clean; full run on staging verified (step 6)
• 301 redirects map generated and added to frontend
• Production Mongo backed up
• Final migration run, DNS switched, logs monitored