BettaFish Public Opinion Research
Research public opinion across social media, the web, and private data, then produce interactive reports.
The evidence shows some security awareness: a database wrapper is described as read-only, dependencies are centralized, and the maintenance workflow uses scoped permissions, ephemeral credentials, path allowlists, commit checks, and manual merging. Deductions apply because these controls mostly protect auxiliary automation, while the core product contacts search providers, LLMs, databases, crawlers, and many platforms without a complete data-flow map, retention or redaction policy, per-action confirmation, product rollback mechanism, or clear external-write boundary. The database example exposes fixed weak credentials, many dependencies use unbounded >= constraints, and no lockfile, vulnerability scan, or supply-chain mitigation is evidenced. The repository name and an email provide limited attribution, but publisher identity and the accountable maintainer remain unverified.
The architecture, tree, workflows, and fixtures show modular processing, retries, dependency checks, JSON repair, and designed error logging. Error samples distinguish parse failures, repair failures, and general exceptions, supporting a relatively good but incomplete failure-message score. Deductions reflect inconsistent coverage claims such as 30+ versus 10+ platforms, a tests/run_tests.py script that says it runs all tests but manually invokes only TestLogMonitor, and substantial availability exposure to external APIs, browsers, databases, models, and system PDF dependencies. No deduction was made merely for execution-dependent properties excluded by the rubric.
The README identifies public-opinion users and developers and describes conversational research, standalone Agent apps, adaptation to finance, OpenAI-compatible models, PostgreSQL/MySQL, Docker, and Windows/Linux/macOS, providing solid ordinary environment-fit evidence. Deductions apply because capability boundaries are largely promotional: supported platform functions, data accessibility, prediction conditions, and private-data constraints are not precisely defined. Natural-language queries are the stated trigger, but ambiguity handling, refusal conditions, sensitive-use controls, and high-impact decision rules are absent.
Bilingual READMEs, an architecture workflow, a detailed source tree, quick-start material, source-start guidance, sample reports, a video, and contribution documents provide strong information architecture. Deductions apply because the supplied source-start excerpt is incomplete and contains a stray “c,” while FAQ coverage and consolidated limitations are weak. BettaFish, WeiYu, and the linked MiroFish pipeline create some naming and product-boundary instability. Most importantly, LICENSE concatenates GPL-2.0, a non-commercial learning license, and Apache-2.0 terms with incompatible grants, so it does not establish a clear single project license. A v1.2.1 badge exists without supplied changelog evidence. Issues, contribution files, and one email give an update path, but accountable maintenance ownership and support commitments remain unclear.
The system produces interactive HTML, PDF, Markdown, IR, intermediate chapters, and multi-Agent synthesized reports, with a sample report and a documented workflow; these outputs appear usable for reading, review, and regeneration. Combining search, private-domain analysis, multimodal processing, forum collaboration, and templated reporting offers plausible marginal value over a single chat response. Deductions apply because quality and coverage advantages are primarily self-asserted rather than supported by systematic comparisons in the supplied material. Continuous crawling, multiple iterative LLM agents, search APIs, databases, browser automation, ML models, and PDF tooling imply meaningful cost and operational overhead, with no budgets, rate limits, resource estimates, or graceful degradation guidance.
The repository structure and fixtures offer some cross-evidence for log parsing, report validation, and automation, and sample output occasionally names broad source categories. The major deduction is the absence of item-level traceable citations. One fixture explicitly generates detailed claims and visual inferences when search results are empty, relying on unspecified public information and general knowledge. Another lists many precise financial, litigation, rating, and market figures without machine-resolvable source identifiers. No method for reconciling cross-source conflicts is shown, and observations, model inference, forecasts, and imagined visual descriptions are blended rather than clearly labeled.
- Resolve the LICENSE manually before adoption: GPL-2.0, non-commercial learning terms, and Apache-2.0 are concatenated with materially conflicting grants.
- Do not use precise figures, quotations, forecasts, or visual descriptions from generated reports without independent checking; fixtures show specific claims can be produced even when search results are empty and without item-level citations.
- Before supplying real secrets or private databases, replace the example database credentials and document what third-party LLM, search, and scraping services receive, log, retain, and delete.
- Review platform terms, privacy, copyright, and applicable law before crawling social services; the supplied evidence does not establish compliant and stable access to every claimed platform.
- For production, lock dependencies, scan packages and images, and establish controls for API failure, spending limits, rate limits, and data recovery.
What does this agent do, and when should you use it?
BettaFish is a pure-Python, from-scratch multi-agent system for public-opinion research, organized around QueryEngine, MediaEngine, InsightEngine, ForumEngine, and ReportEngine. Its Flask application accepts a natural-language research question, runs web search, multimodal analysis, and private-database analysis in parallel, and coordinates further investigation through a moderated forum loop. MindSpider can separately collect social-media topics, posts, and comments, while SentimentAnalysisModel includes multilingual, BERT, GPT-2, Qwen3, and conventional machine-learning options. ReportEngine binds the collected findings into a validated Document IR and renders an interactive HTML report, with PDF and Markdown output also supported. The system can be self-hosted with Docker Compose or a local Python environment, uses OpenAI-compatible model endpoints, and is documented primarily with PostgreSQL while allowing MySQL configuration. It is best suited to research or engineering teams prepared to operate model APIs, databases, crawlers, and compliance controls rather than buyers seeking a managed analytics service.
The end-to-end path starts when app.py receives a question through the Flask application and launches Query Agent, Media Agent, and Insight Agent concurrently. QueryEngine calls domestic and international web and news search tools; MediaEngine examines video, images, and structured search-result cards; InsightEngine uses asynchronous read-only SQLAlchemy queries, keyword_optimizer.py, and sentiment_analyzer.py to retrieve topics, comments, and sentiment signals from a private opinion database. During iterative research, ForumEngine/monitor.py watches agent messages, llm_host.py produces moderator guidance, and the agents read the discussion through utils/forum_reader.py to reflect and redirect their searches. ReportEngine then selects a Markdown template, designs the layout, assigns word budgets, generates and validates chapter JSON, binds it with stitcher.py into a Document IR, and renders interactive HTML with optional PDF and Markdown exports. MindSpider can also run independently for broad-topic extraction or deep sentiment crawling through its --broad-topic, --deep-sentiment, and --complete commands.
- A university or brand-research team needs to combine web coverage, social posts, and large comment sets into a reputation or event report.
- An analytics group with an internal feedback database wants to study private operational data alongside public discussion.
- A media researcher needs to inspect text, images, short-form video, and structured search cards rather than limiting analysis to text.
- A data-science team wants runnable implementations of multilingual, BERT, GPT-2, Qwen3, and classical machine-learning sentiment methods for Weibo-oriented experiments.
- A developer studying agent architecture wants a Python implementation that does not depend on an existing agent framework and exposes its search, forum, and reporting components.
- A report engineer already has Markdown outputs from one or more analysis engines and wants to retry synthesis or rerender HTML, PDF, or Markdown without rerunning research.
What are this agent's strengths and limitations?
- Separate engines handle search, multimodal material, private databases, and reporting, while ForumEngine adds moderated, multi-round coordination between them.
- ReportEngine uses a validated Document IR and clearly separates template selection, document layout, word budgeting, chapter generation, and HTML/PDF/Markdown rendering.
- Each model-backed component can be configured with its own API_KEY, BASE_URL, and MODEL_NAME through an OpenAI-compatible interface instead of one named model vendor.
- The repository offers Docker Compose deployment, a Flask interface, standalone Streamlit apps, crawler commands, and a headless report-generation CLI.
- Sentiment support includes multilingual, BERT LoRA, GPT-2 LoRA, small Qwen3, and traditional machine-learning implementations rather than relying exclusively on general-purpose language models.
- Operators must supply and maintain model endpoints, a database, and a Playwright browser; the full stack is not a zero-configuration executable or hosted service.
- Crawler use carries compliance obligations for robots.txt, site terms, and applicable law, and the project assigns resulting legal responsibility to the user.
- The project disclaimer restricts use to learning, academic research, and education, explicitly prohibiting commercial use and commercial decision-making from its analysis.
- PDF export requires WeasyPrint and additional operating-system packages; without them, installation or PDF generation may fail.
- The source claims broad social coverage and large comment volumes but provides no quantitative evidence here for current per-platform availability, crawl reliability, rate limits, or analytical accuracy.
- Full research depends on network access, external model services, and configured data sources, so outages, exhausted quotas, or missing engine outputs can reduce or stop the workflow.
How do you install or deploy this agent?
Docker path: copy .env.example to .env, configure the database and every required model endpoint, then run docker compose up -d from the repository root and open http://localhost:5000. The documented Docker database defaults are DB_HOST=db, DB_PORT=5432, DB_USER=bettafish, DB_PASSWORD=bettafish, and DB_NAME=bettafish.
Source path: prepare Windows, Linux, or macOS with Python 3.9+, PostgreSQL or MySQL, and at least 2 GB of recommended memory. Create an environment with conda create -n your_conda_name python=3.11 followed by conda activate your_conda_name, or use uv venv --python 3.11. Install packages with pip install -r requirements.txt or uv pip install -r requirements.txt, then install the crawler browser with playwright install chromium. Copy .env.example to .env; configure DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, DB_NAME, DB_CHARSET, and DB_DIALECT, plus the API_KEY, BASE_URL, and MODEL_NAME values required by Insight, Media, and the other model-backed components. Model endpoints must implement the OpenAI-compatible request format. PDF export is optional and requires separate WeasyPrint system dependencies.
How do you use this agent?
For the first full-system invocation, activate the environment, run python app.py from the repository root, visit http://localhost:5000, and submit an analysis request conversationally. Individual engines can instead be launched with streamlit run SingleEngineApp/query_engine_streamlit_app.py --server.port 8503, streamlit run SingleEngineApp/media_engine_streamlit_app.py --server.port 8502, or streamlit run SingleEngineApp/insight_engine_streamlit_app.py --server.port 8501.
For crawling, run cd MindSpider and python main.py --setup. Use python main.py --broad-topic for topic extraction, python main.py --complete --date 2024-01-20 for the complete dated flow, or python main.py --deep-sentiment --platforms xhs dy wb for platform-scoped deep crawling.
When analysis logs already exist, run python report_engine_only.py, or supply a subject with python report_engine_only.py --query "Civil engineering industry analysis". The CLI also supports --skip-pdf, --skip-markdown, and --verbose. At least one of insight_engine_streamlit_reports, media_engine_streamlit_reports, or query_engine_streamlit_reports must contain a .md report.
FAQ
Does BettaFish require an official OpenAI model?
Can it be adopted for commercial monitoring or business decisions?
Must MindSpider run before a report can be generated?
Which databases are supported?
What happens if PDF dependencies are unavailable?
--skip-pdf. Markdown is also generated by default unless --skip-markdown is supplied.