BettaFish Public Opinion Research
Research public opinion across social media, the web, and private data, then produce interactive reports.
- Source repo
- 666ghj/BettaFish
- Stars
- ★ 42k
- Last updated
- 17d ago
- License
- GPL-2.0
- Primary language
- Python
- FA score
- 38/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformOpenAI API
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A university or brand-research team needs to combine web coverage, social posts, and large comment sets into a reputation or event report.
- Not a fit if
- Teams wanting a managed analytics service without running their own database and model APIs
- Anyone needing commercial or profit-making use, which the project explicitly prohibits
- Teams unwilling to own crawler compliance duties or install a Playwright browser
- Source review
- 38/100 · Major gaps 2 safety controls not found
What does this agent do, and when should you use it?
BettaFish is a pure-Python, from-scratch multi-agent system for public-opinion research, organized around QueryEngine, MediaEngine, InsightEngine, ForumEngine, and ReportEngine. Its Flask application accepts a natural-language research question, runs web search, multimodal analysis, and private-database analysis in parallel, and coordinates further investigation through a moderated forum loop. MindSpider can separately collect social-media topics, posts, and comments, while SentimentAnalysisModel includes multilingual, BERT, GPT-2, Qwen3, and conventional machine-learning options. ReportEngine binds the collected findings into a validated Document IR and renders an interactive HTML report, with PDF and Markdown output also supported. The system can be self-hosted with Docker Compose or a local Python environment, uses OpenAI-compatible model endpoints, and is documented primarily with PostgreSQL while allowing MySQL configuration. It is best suited to research or engineering teams prepared to operate model APIs, databases, crawlers, and compliance controls rather than buyers seeking a managed analytics service.
The end-to-end path starts when app.py receives a question through the Flask application and launches Query Agent, Media Agent, and Insight Agent concurrently. QueryEngine calls domestic and international web and news search tools; MediaEngine examines video, images, and structured search-result cards; InsightEngine uses asynchronous read-only SQLAlchemy queries, keyword_optimizer.py, and sentiment_analyzer.py to retrieve topics, comments, and sentiment signals from a private opinion database. During iterative research, ForumEngine/monitor.py watches agent messages, llm_host.py produces moderator guidance, and the agents read the discussion through utils/forum_reader.py to reflect and redirect their searches. ReportEngine then selects a Markdown template, designs the layout, assigns word budgets, generates and validates chapter JSON, binds it with stitcher.py into a Document IR, and renders interactive HTML with optional PDF and Markdown exports. MindSpider can also run independently for broad-topic extraction or deep sentiment crawling through its --broad-topic, --deep-sentiment, and --complete commands.
- A university or brand-research team needs to combine web coverage, social posts, and large comment sets into a reputation or event report.
- An analytics group with an internal feedback database wants to study private operational data alongside public discussion.
- A media researcher needs to inspect text, images, short-form video, and structured search cards rather than limiting analysis to text.
- A data-science team wants runnable implementations of multilingual, BERT, GPT-2, Qwen3, and classical machine-learning sentiment methods for Weibo-oriented experiments.
- A developer studying agent architecture wants a Python implementation that does not depend on an existing agent framework and exposes its search, forum, and reporting components.
- A report engineer already has Markdown outputs from one or more analysis engines and wants to retry synthesis or rerender HTML, PDF, or Markdown without rerunning research.
How do you install or deploy this agent?
Docker path: copy .env.example to .env, configure the database and every required model endpoint, then run docker compose up -d from the repository root and open http://localhost:5000. The documented Docker database defaults are DB_HOST=db, DB_PORT=5432, DB_USER=bettafish, DB_PASSWORD=bettafish, and DB_NAME=bettafish.
Source path: prepare Windows, Linux, or macOS with Python 3.9+, PostgreSQL or MySQL, and at least 2 GB of recommended memory. Create an environment with conda create -n your_conda_name python=3.11 followed by conda activate your_conda_name, or use uv venv --python 3.11. Install packages with pip install -r requirements.txt or uv pip install -r requirements.txt, then install the crawler browser with playwright install chromium. Copy .env.example to .env; configure DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, DB_NAME, DB_CHARSET, and DB_DIALECT, plus the API_KEY, BASE_URL, and MODEL_NAME values required by Insight, Media, and the other model-backed components. Model endpoints must implement the OpenAI-compatible request format. PDF export is optional and requires separate WeasyPrint system dependencies.
How do you use this agent?
For the first full-system invocation, activate the environment, run python app.py from the repository root, visit http://localhost:5000, and submit an analysis request conversationally. Individual engines can instead be launched with streamlit run SingleEngineApp/query_engine_streamlit_app.py --server.port 8503, streamlit run SingleEngineApp/media_engine_streamlit_app.py --server.port 8502, or streamlit run SingleEngineApp/insight_engine_streamlit_app.py --server.port 8501.
For crawling, run cd MindSpider and python main.py --setup. Use python main.py --broad-topic for topic extraction, python main.py --complete --date 2024-01-20 for the complete dated flow, or python main.py --deep-sentiment --platforms xhs dy wb for platform-scoped deep crawling.
When analysis logs already exist, run python report_engine_only.py, or supply a subject with python report_engine_only.py --query "Civil engineering industry analysis". The CLI also supports --skip-pdf, --skip-markdown, and --verbose. At least one of insight_engine_streamlit_reports, media_engine_streamlit_reports, or query_engine_streamlit_reports must contain a .md report.
What are this agent's strengths and limitations?
- Separate engines handle search, multimodal material, private databases, and reporting, while ForumEngine adds moderated, multi-round coordination between them.
- ReportEngine uses a validated Document IR and clearly separates template selection, document layout, word budgeting, chapter generation, and HTML/PDF/Markdown rendering.
- Each model-backed component can be configured with its own API_KEY, BASE_URL, and MODEL_NAME through an OpenAI-compatible interface instead of one named model vendor.
- The repository offers Docker Compose deployment, a Flask interface, standalone Streamlit apps, crawler commands, and a headless report-generation CLI.
- Sentiment support includes multilingual, BERT LoRA, GPT-2 LoRA, small Qwen3, and traditional machine-learning implementations rather than relying exclusively on general-purpose language models.
- Operators must supply and maintain model endpoints, a database, and a Playwright browser; the full stack is not a zero-configuration executable or hosted service.
- Crawler use carries compliance obligations for robots.txt, site terms, and applicable law, and the project assigns resulting legal responsibility to the user.
- The project disclaimer restricts use to learning, academic research, and education, explicitly prohibiting commercial use and commercial decision-making from its analysis.
- PDF export requires WeasyPrint and additional operating-system packages; without them, installation or PDF generation may fail.
- The source claims broad social coverage and large comment volumes but provides no quantitative evidence here for current per-platform availability, crawl reliability, rate limits, or analytical accuracy.
- Full research depends on network access, external model services, and configured data sources, so outages, exhausted quotas, or missing engine outputs can reduce or stop the workflow.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| BettaFish Public Opinion Research This agent | 38 · Major gaps | Self-hosted serviceFree + model costs | ★ 42k | 17d ago | Python | OpenAI API |
| Business Analytics Agent | 60 · Some gaps | Desktop appFree + model costs | ★ 2.6k | 10d ago | JavaScript | OpenAI API |
| FinRobot | 51 · Major gaps | CLIFree + model costs | ★ 8.2k | 20d ago | Python | OpenAI API |
| Maxun | 48 · Major gaps | Hosted serviceFreemium | ★ 18k | today | TypeScript | — |
How does FollowAgents rate this agent?
Why each dimension lost points
The evidence shows some security awareness: a database wrapper is described as read-only, dependencies are centralized, and the maintenance workflow uses scoped permissions, ephemeral credentials, path allowlists, commit checks, and manual merging. Deductions apply because these controls mostly protect auxiliary automation, while the core product contacts search providers, LLMs, databases, crawlers, and many platforms without a complete data-flow map, retention or redaction policy, per-action confirmation, product rollback mechanism, or clear external-write boundary. The database example exposes fixed weak credentials, many dependencies use unbounded >= constraints, and no lockfile, vulnerability scan, or supply-chain mitigation is evidenced. The repository name and an email provide limited attribution, but publisher identity and the accountable maintainer remain unverified.
The architecture, tree, workflows, and fixtures show modular processing, retries, dependency checks, JSON repair, and designed error logging. Error samples distinguish parse failures, repair failures, and general exceptions, supporting a relatively good but incomplete failure-message score. Deductions reflect inconsistent coverage claims such as 30+ versus 10+ platforms, a tests/run_tests.py script that says it runs all tests but manually invokes only TestLogMonitor, and substantial availability exposure to external APIs, browsers, databases, models, and system PDF dependencies. No deduction was made merely for execution-dependent properties excluded by the rubric.
The README identifies public-opinion users and developers and describes conversational research, standalone Agent apps, adaptation to finance, OpenAI-compatible models, PostgreSQL/MySQL, Docker, and Windows/Linux/macOS, providing solid ordinary environment-fit evidence. Deductions apply because capability boundaries are largely promotional: supported platform functions, data accessibility, prediction conditions, and private-data constraints are not precisely defined. Natural-language queries are the stated trigger, but ambiguity handling, refusal conditions, sensitive-use controls, and high-impact decision rules are absent.
Bilingual READMEs, an architecture workflow, a detailed source tree, quick-start material, source-start guidance, sample reports, a video, and contribution documents provide strong information architecture. Deductions apply because the supplied source-start excerpt is incomplete and contains a stray “c,” while FAQ coverage and consolidated limitations are weak. BettaFish, WeiYu, and the linked MiroFish pipeline create some naming and product-boundary instability. Most importantly, LICENSE concatenates GPL-2.0, a non-commercial learning license, and Apache-2.0 terms with incompatible grants, so it does not establish a clear single project license. A v1.2.1 badge exists without supplied changelog evidence. Issues, contribution files, and one email give an update path, but accountable maintenance ownership and support commitments remain unclear.
The system produces interactive HTML, PDF, Markdown, IR, intermediate chapters, and multi-Agent synthesized reports, with a sample report and a documented workflow; these outputs appear usable for reading, review, and regeneration. Combining search, private-domain analysis, multimodal processing, forum collaboration, and templated reporting offers plausible marginal value over a single chat response. Deductions apply because quality and coverage advantages are primarily self-asserted rather than supported by systematic comparisons in the supplied material. Continuous crawling, multiple iterative LLM agents, search APIs, databases, browser automation, ML models, and PDF tooling imply meaningful cost and operational overhead, with no budgets, rate limits, resource estimates, or graceful degradation guidance.
The repository structure and fixtures offer some cross-evidence for log parsing, report validation, and automation, and sample output occasionally names broad source categories. The major deduction is the absence of item-level traceable citations. One fixture explicitly generates detailed claims and visual inferences when search results are empty, relying on unspecified public information and general knowledge. Another lists many precise financial, litigation, rating, and market figures without machine-resolvable source identifiers. No method for reconciling cross-source conflicts is shown, and observations, model inference, forecasts, and imagined visual descriptions are blended rather than clearly labeled.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Resolve the LICENSE manually before adoption: GPL-2.0, non-commercial learning terms, and Apache-2.0 are concatenated with materially conflicting grants.
- Do not use precise figures, quotations, forecasts, or visual descriptions from generated reports without independent checking; fixtures show specific claims can be produced even when search results are empty and without item-level citations.
- Before supplying real secrets or private databases, replace the example database credentials and document what third-party LLM, search, and scraping services receive, log, retain, and delete.
- Review platform terms, privacy, copyright, and applicable law before crawling social services; the supplied evidence does not establish compliant and stable access to every claimed platform.
- For production, lock dependencies, scan packages and images, and establish controls for API failure, spending limits, rate limits, and data recovery.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v3.0.0 微舆v3.0.0
Release notes excerpt
预测引擎 MiroFish-v0.1.0 发布🎉 查看我们最新发布的预测引擎：[MiroFish-简洁通用的群体智能引擎，预测万物](https://github.com/666ghj/MiroFish) <div align="center"> <img src="https://github.com/user-attachments/assets/fba934e7-3d8e-4cf3-b81b-c4538acd3e56" alt="banner" width="500"> </div> “数据分析三板斧”全线贯通：我们激动的宣布 MiroFish 正式发布！随着最后一块版图补齐，我们构建了从 BettaFish（数据收集与分析）到 MiroFish（全景预测）的完整链路。至此，从原始数据到智能决策的闭环已成，让预见未来成为可能！ <img width="1502" height="990" alt="04fb364a3122e3ec7dfab8929897745a" src="https://github.com/user-attachments/assets/c5a987b2-3ce5-460f-b671-f2f700db1377" />View on GitHub ↗ -
v2.1.0 微舆v2.1.0
Release notes excerpt
What's Changed * feat(insight_agent): Insight Agent新增了DeepSearchAgent._cluster_and_sample_results()方法，该方法对搜索结果进行文本编码，使用 KMeans 聚类并按热度排序采样，然后从每个聚类中选取代表性语义返回 by @ghmark675 in https://github.com/666ghj/BettaFish/pull/350 * feat(xhs): 集成xhshow库优化签名生成与请求参数 by @ptghb in https://github.com/666ghj/BettaFish/pull/330 * Feat: 同样的效果更经济的搜索选择，新增安思派AI搜索支持 (#457) by @666ghj in https://github.com/666ghj/BettaFish/pull/461 * Shutdown only clean 增加安全关闭系统的功能 by @HKLHaoBin in https://github.com/666ghj/BettaFish/pull/441 * Update Via main by @MaYiding in https://github.com/666ghj/BettaFish/pull/463 …View on GitHub ↗ -
v2.0.0 微舆 v2.0.0
Release notes excerpt
版本概述 微舆BettaFish 是一个基于多智能体的舆情分析系统，致力于打破信息茧房并提供权威、深度的舆情洞察。v2.0.0 是 BettaFish 一次里程碑式的大版本，我们全面重构了报告引擎，增强了弱模型适配与长文稳定性，实现了原子级可控与标准化交付。修复大量BUG的同时针对性能瓶颈进行了专项优化，系统运行内存占用已降至 1GB 以内。 微舆 v2.0.0 在稳定性、可配置性及部署便利性上迈出了重要一步。我们希望此版本能为你带来更高效、更可靠的体验。 ✨ Report Engine V2 标准化生成流水线 现在的 Report Agent 遵循以下七阶段严谨流程，确保不同llm生成的报告的样式、内容统一与可控，提高了导出pdf的美观： 1. 多源情报聚合：自动汇总多 Agent 的中间产物与论坛记录，构建全景信息底座。 2. 智能模板匹配：内置多套专业舆情分析模板，系统根据内容特征自动择优匹配。 3. 架构与切片设计：由 LLM 顶层设计报告标题、目录结构、Hero 模块及整体视觉风格，并选定适配的 HTML 模板。 4. 章节任务规划：自动拆解篇章结构，生成细粒度的章节编写任务清单。 …View on GitHub ↗
FAQ
Does BettaFish require an official OpenAI model?
Can it be adopted for commercial monitoring or business decisions?
Must MindSpider run before a report can be generated?
Which databases are supported?
What happens if PDF dependencies are unavailable?
--skip-pdf. Markdown is also generated by default unless --skip-markdown is supplied.