HidroScrapper

HidroScrapper uses a monorepo architecture with separate frontend, backend, and shared TypeScript packages. The frontend is built with Next.js 16, React 19, TypeScript, Tailwind CSS v4, and App Router, while the backend uses Express.js 5, MongoDB, Mongoose, and Apify REST API integration. MongoDB works as the single source of truth. All media profiles, posts, jobs, reports, users, and configuration values are stored persistently in MongoDB, avoiding in-memory state or JSON file storage. The system is designed around authenticated dashboard access, social media profile management, background scraping jobs, engagement analytics, report generation, and PDF exports.

Frontend Stack

The frontend was built with Next.js 16, React 19, TypeScript, Tailwind CSS v4, lucide-react, jspdf, html2canvas, clsx, and tailwind-merge. The application uses the App Router with protected dashboard routes, including Dashboard, Media, Media Detail, Reports, and Configuration sections. The UI includes reusable components such as MetricCard, JobsPanel, MediaCard, ScrapeModal, CompareModal, ReportsHistoryTable, ConsolidatedMetrics, and ReportViewer. The frontend communicates with the backend through service modules using NEXT_PUBLIC_API_URL as the API base URL.

Backend Stack

The backend was built with Express.js 5, MongoDB, Mongoose, TypeScript, dotenv, and Apify REST API integration. It exposes REST endpoints for authentication, media management, posts, scraping, reports, jobs, configuration, and system health checks. The backend starts immediately to avoid proxy 502 errors and connects to MongoDB in the background with retry logic. API routes are blocked with a 503 response until MongoDB is connected.

Database Stack

MongoDB is the persistent database and the single source of truth for the platform. The system stores media profiles, users, scraped posts, scraping jobs, generated reports, and global configuration. Posts include engagement metrics such as likes, comments, views, shares, total interactions, publication date, platform, URL, title, and content. Reports are persisted as serialized JSON with metrics, conclusions, and recommendations.

Automation Stack

HidroScrapper automates social media data collection through background scraping jobs. When a user starts a scrape, the backend marks the media profile as processing, creates a job record, responds immediately with a job ID, and continues the scraping process in the background. The backend calls the Apify API, retrieves dataset items, normalizes posts, filters results by date range, performs idempotent upserts by media ID and post URL, updates the media sync status, and marks the job as completed or failed.

AI Stack

The platform includes configuration support for LLM providers such as OpenAI, Anthropic, and Google. This makes the system extendable for AI-assisted analysis, report conclusions, recommendations, and future social media intelligence features. The current architecture is prepared to connect scraping data, engagement metrics, and LLM-based insights inside the reporting workflow.

Deployment Stack

The project is structured for Docker-based deployment. Each app includes its own Dockerfile, with separate build processes for the backend, frontend, and shared package. The backend compiles TypeScript into a dist directory and runs with Node.js, while the frontend uses next build to generate an optimized Next.js bundle. The backend is designed to start immediately and connect to MongoDB in the background to prevent reverse proxy errors.