Commit Graph

861 Commits

Author SHA1 Message Date
Fringg 784616b349 refactor: replace universal_migration.py with Alembic
Remove the 7,791-line universal_migration.py and 16 incomplete individual
Alembic migrations. Replace with a single initial schema migration using
Base.metadata.create_all(checkfirst=True).

Changes:
- Add programmatic Alembic runner (app/database/migrations.py) with
  auto-stamp logic for existing databases transitioning from
  universal_migration
- Extract ensure_default_web_api_token() to web_api_token_service.py
- Extract sync_postgres_sequences() to database.py with SQL injection
  prevention via _quote_ident()
- Add HMAC token hashing support with backward-compatible dual-hash
  fallback and automatic rehashing
- Remove dead init_db() function and unused imports
- Add Makefile targets: migrate, migration, migrate-stamp, migrate-history
- Fix fileConfig() destroying structlog config (disable_existing_loggers)
- Remove duplicate migrations/alembic/alembic.ini with credentials
- Add script.py.mako template for future migration generation
- Update startup flow: alembic upgrade → sync sequences → ensure token
- Harden database.py: ParamSpec for retry decorator, safe URL logging,
  echo='debug' mode, execute_with_retry validation
- Update documentation references

31 files changed, 302 insertions(+), 9,226 deletions(-)
2026-02-18 08:10:20 +03:00
Fringg 7883efc3d6 fix: return zeroed stats dict when withdrawal is disabled
can_request_withdrawal returned empty dict {} when withdrawal feature
was disabled, causing KeyError on 'total_earned' in withdrawal route.
2026-02-18 05:37:32 +03:00
Fringg 90278f1f5f style: fix ruff formatting in broadcast_service and tests 2026-02-17 18:50:25 +03:00
Fringg df5b1a072d fix: handle YooKassa NotFoundError gracefully in get_payment_info
Catch NotFoundError (404) separately from generic exceptions.
Old/expired payments return 404 from YooKassa API — this is expected
and should be logged as WARNING without traceback, not ERROR.
2026-02-17 18:46:32 +03:00
Fringg 10e231e52e feat: blocked user detection during broadcasts, filter blocked from all notifications
- Broadcast tri-state return: 'sent'/'blocked'/'failed' with blocked_count tracking
- Background cleanup: mark blocked users + disable their subscriptions + Remnawave
- blocked_count in BroadcastHistory model, schemas, API responses, admin UI
- Filter User.status==ACTIVE in subscription queries: get_expiring_subscriptions,
  get_expired_subscriptions, get_subscriptions_for_autopay,
  get_daily_subscriptions_for_charge, get_disabled_daily_subscriptions_for_resume
- Guard in notification_delivery_service.send_notification for BLOCKED/DELETED users
- Fix subscription tariff switch: preserve remaining days with total_seconds()
- Fix redundant local UTC imports across 16 files
- Fix test mocks: add **kwargs, correct assertion, remove dead expression
2026-02-17 18:37:25 +03:00
Fringg 7c20fde4e8 fix: medium-priority fixes for partner system
- replace unsafe referral code generator with unique DB-checked version
- remove dead code in get_global_partner_stats
- validate status filter params with Literal types in admin routes
2026-02-17 12:42:40 +03:00
Fringg fcf3a2c806 fix: resolve HIGH-priority performance and security issues in partner system
- fix N+1 query in money laundering analysis with GROUP BY batch query
- fix N+1 query in cabinet referral earnings with batch user fetch
- eliminate double balance stats computation in withdrawal flow
- replace in-memory referral counting with SQL COUNT/CASE aggregation
- fix HTML injection in admin Telegram notifications via html.escape()
- standardize return types for reject/complete withdrawal methods
2026-02-17 12:38:25 +03:00
Fringg 88997492c3 fix: critical security and data integrity fixes for partner system
- Add SELECT FOR UPDATE locking on all financial state transitions
  (withdrawal approve/reject/complete/create, partner approve/reject)
- Add html.escape() on all user-controlled values in email templates
- Wrap sync SMTP send_email in asyncio.to_thread to avoid blocking event loop
- Add missing database indexes on referral_earnings(user_id, referral_id),
  users(referred_by_id, partner_status), withdrawal_requests(user_id, status),
  advertising_campaigns(partner_user_id)
2026-02-17 12:28:30 +03:00
Fringg 327d4f4d15 feat: notify users on partner/withdrawal approve/reject
4 notification types via NotificationDeliveryService:
- Partner application approved/rejected
- Withdrawal request approved/rejected

Telegram + email + WebSocket routing handled automatically.
Email templates in ru/en/zh/ua.
2026-02-17 12:04:23 +03:00
Fringg cf7cc5a84e feat: add admin notifications for partner applications and withdrawals
Send notifications to admin chat when a partner application is submitted
or a withdrawal request is created, following existing notification pattern.
2026-02-17 11:48:38 +03:00
Fringg d39063b22f fix: unassign all campaigns when revoking partner status
Previously revoke_partner only changed partner_status and commission,
leaving campaigns orphaned with invalid partner_user_id. Now sets
partner_user_id=NULL on all campaigns belonging to the revoked partner.
2026-02-17 11:11:25 +03:00
Fringg 58bfaeaddb feat: add partner system and withdrawal management to cabinet
- Partner application flow: user applies, admin reviews/approves/rejects
- Individual commission % per partner with admin management
- Campaign assignment/unassignment to partners
- Withdrawal system: balance check, create request, cancel
- Admin withdrawal management with risk scoring and fraud analysis
- Database migration: partner_applications table, user partner fields, campaign partner_user_id
- Pydantic schemas with proper validation bounds
- Batch user fetching to prevent N+1 queries
- Row locking on cancel to prevent race conditions
2026-02-17 09:51:36 +03:00
Fringg 6fa49485d9 fix: add naive datetime guards to fromisoformat() in Redis cache readers
Old Redis entries saved before utcnow→now(UTC) migration lack timezone
info, causing TypeError on subtraction with aware datetimes.
2026-02-17 07:52:26 +03:00
Fringg 5dc4b0ec15 chore: ruff format oauth.py, auth schemas, admin_notification_service 2026-02-17 06:57:30 +03:00
Fringg d9552799c1 feat: add web campaign links with bonus processing in auth flow
- Add web_link generation for campaigns (uses MINIAPP_CUSTOM_URL)
- Process campaign_slug in all auth endpoints (telegram, widget, email, oauth)
- Apply campaign bonus (balance/subscription/tariff) with SELECT FOR UPDATE lock
- Add rollback + user refresh on campaign bonus failure
- Fix N+1 query in campaign registrations (batch subscription check)
- Remove duplicate queries in get_campaign_statistics (~60 lines dead code)
- Simplify _store_refresh_token (remove TOCTOU pre-check, keep IntegrityError)
- Remove dead expression in campaign_service.py
- Align start_parameter max_length to 64 (matches DB column)
- Remove unused campaign_slug from EmailRegisterStandaloneRequest
2026-02-17 06:44:03 +03:00
Fringg c75ec0b22a fix: AttributeError in withdrawal admin notification (send_to_admins → send_admin_notification) 2026-02-17 05:23:49 +03:00
Fringg eb18994b7d fix: complete datetime.utcnow() → datetime.now(UTC) migration
- Migrate 660+ datetime.utcnow() across 153 files to datetime.now(UTC)
- Migrate 30+ datetime.now() without UTC to datetime.now(UTC)
- Convert all 170 DateTime columns to DateTime(timezone=True)
- Add migrate_datetime_to_timestamptz() in universal_migration with SET LOCAL timezone='UTC' safety
- Remove 70+ .replace(tzinfo=None) workarounds
- Fix utcfromtimestamp → fromtimestamp(..., tz=UTC)
- Fix fromtimestamp() without tz= (system_logs, backup_service, referral_diagnostics)
- Fix fromisoformat/isoparse to ensure aware output (platega, yookassa, wata, miniapp, nalogo)
- Fix strptime() to add .replace(tzinfo=UTC) (backup_service, referral_diagnostics)
- Fix datetime.combine() to include tzinfo=UTC (remnawave_sync, traffic_monitoring)
- Fix datetime.max/datetime.min sentinels with .replace(tzinfo=UTC)
- Rename panel_datetime_to_naive_utc → panel_datetime_to_utc
- Remove DTZ003 from ruff ignore list
2026-02-17 04:45:40 +03:00
Fringg 4247981c98 fix: normalize transaction amount signs across all aggregations
SUBSCRIPTION_PAYMENT transactions have inconsistent signs in DB
(some negative, some positive). Add func.abs()/abs() to all SUM
queries and display code to ensure correct totals regardless of sign.

Affected: admin statistics, referral contest stats, tariff revenue,
campaign stats, reporting service, admin renewal notifications.
2026-02-17 03:40:37 +03:00
Fringg 491a7e1c42 fix: remove unused PaymentService from MonitoringService init
MonitoringService instantiated PaymentService() at module level during
import, triggering a debug log before structlog/logging were configured.
This caused [debug    ] with padded spaces (structlog default pad_level)
and appeared 7 seconds before the startup banner. The payment_service
attribute was never used in MonitoringService.
2026-02-16 19:02:57 +03:00
Fringg 0807a9ff19 fix: sync SUPPORT_SYSTEM_MODE between SystemSettings and SupportSettings
When changing SUPPORT_SYSTEM_MODE via system settings admin panel, the
SupportSettingsService JSON cache was not updated, causing the old value
to take priority. Now both services stay in sync bidirectionally.
2026-02-16 18:22:44 +03:00
Fringg 1f0fef114b refactor: complete structlog migration with contextvars, kwargs, and logging hardening
- Add ContextVarsMiddleware for automatic user_id/chat_id/username binding
  via structlog contextvars (aiogram) and http_method/http_path (FastAPI)
- Use bound_contextvars() context manager instead of clear_contextvars()
  to safely restore previous state instead of wiping all context
- Register ContextVarsMiddleware as outermost middleware (before GlobalError)
  so all error logs include user context
- Replace structlog.get_logger() with structlog.get_logger(__name__) across
  270 calls in 265 files for meaningful logger names
- Switch wrapper_class from BoundLogger to make_filtering_bound_logger()
  for pre-processor level filtering (performance optimization)
- Migrate 1411 %-style positional arg logger calls to structlog kwargs
  style across 161 files via AST script
- Migrate log_rotation_service.py from stdlib logging to structlog
- Add payment module prefixes to TelegramNotifierProcessor.IGNORED_LOGGER_PREFIXES
  and ExcludePaymentFilter.PAYMENT_MODULES to prevent payment data leaking
  to Telegram notifications and general log files
- Fix LoggingMiddleware: add from_user null-safety for channel posts,
  switch time.time() to time.monotonic() for duration measurement
- Remove duplicate logger assignments in purchase.py, config.py,
  inline.py, and admin/payments.py
2026-02-16 09:18:12 +03:00
Fringg 97ec39aa80 fix: add promo code anti-abuse protections
- Rate-limit on brute-force: 5 failed attempts per 5 min blocks user
- Daily stacking limit: max 5 promo activations per 24h (in-memory + DB)
- Format validation: only alphanumeric/hyphen/underscore, 3-50 chars
2026-02-16 06:52:45 +03:00
Fringg d712ab8301 fix: remove redundant trial inactivity monitoring checks
Remnawave already sends user.not_connected webhooks, making the
monitoring service's 1h/24h trial inactivity checks redundant.
The monitoring checks caused false positives because they relied on
traffic_used_gb which may not be synced in real-time.

Removed:
- _check_trial_inactivity_notifications from monitoring cycle
- _send_trial_inactive_notification method
- trial_inactive_1h / trial_inactive_24h notification settings
- Admin UI toggles and preview buttons for these notifications
2026-02-16 00:58:24 +03:00
Fringg 1e2a7e3096 fix: webhook notification 'My Subscription' button uses unregistered callback_data
Changed callback_data from 'subscription' (no handler) to 'menu_subscription'
(registered handler) in _get_subscription_keyboard and _get_traffic_keyboard.
In cabinet mode the button opens a WebApp URL so the bug was invisible,
but in default MAIN_MENU_MODE the callback went unhandled.
2026-02-16 00:30:17 +03:00
Fringg bf2b2f1c56 feat: add button style and emoji support for cabinet mode (Bot API 9.4)
- Upgrade aiogram to 3.25.0 for style/icon_custom_emoji_id support
- Add CABINET_BUTTON_STYLE config for global color override
- Per-section default styles: subscription (green), balance (blue),
  referral (green), admin (red), home (blue)
- Style priority: explicit > CABINET_BUTTON_STYLE > per-section default
- Add icon_custom_emoji_id pass-through for Premium bot owners
- Admin panel setting for button style with color picker
2026-02-12 22:34:38 +03:00
Fringg ad87c5fb5e feat: rename MAIN_MENU_MODE=text to cabinet with deep-linking to frontend sections
- Rename mode from 'text' to 'cabinet' (text/text_only/minimal kept as aliases)
- Add build_cabinet_url() for joining MINIAPP_CUSTOM_URL with section paths
- Cabinet main menu now has section-specific buttons: subscription, balance,
  referral, support, info — each opens the corresponding cabinet page
- Add CALLBACK_TO_CABINET_PATH mapping for automatic deep-linking from
  callback_data to cabinet routes (/subscription, /balance, /referral, etc.)
- Unmapped callback_data gracefully falls back to regular Telegram callbacks
- Add startup validation warning when cabinet mode is active without MINIAPP_CUSTOM_URL
- Update admin broadcast buttons with section-specific routing
- Backward compatible: is_text_main_menu_mode() kept as alias for is_cabinet_mode()
2026-02-12 22:21:08 +03:00
Fringg 760c833b74 fix: ticket creation crash and webhook PendingRollbackError
- tickets.py: remove ENABLE_LOGO_MODE branches that used edit_message_caption
  on text messages (prompt is always text, not photo with caption)
- webhook_service: add db.rollback() before retrying DB ops in _handle_user_deleted
  when subscription was cascade-deleted, catch PendingRollbackError alongside StaleDataError
2026-02-12 20:32:52 +03:00
Fringg 454b83138e fix: flood control handling in pinned messages and XSS hardening in HTML sanitizer
- Add retry loop with backoff to _unpin_message_for_user (max 3 attempts)
- Add TelegramRetryAfter handling in _send_and_pin_message (unpin + send phases)
- Fix missing failed_count increment when all broadcast retries exhaust (for/else)
- Remove dead code in unpin_active_pinned_message (unreachable TelegramRetryAfter catch)
- Harden sanitize_html: allowlist URI schemes (http/https/tg/mailto/tel), whitelist
  tag attributes, strip all attrs from tags without explicit whitelist, full HTML
  entity decoding via html.unescape
2026-02-12 19:13:40 +03:00
Fringg 57dc1ff47f fix: resolve deadlock on server_squads counter updates and add webhook notification toggles
- Fix deadlock: enforce sorted lock ordering in add_user_to_servers/remove_user_from_servers
- Fix cross-call deadlock: add update_server_user_counts() for atomic add+remove in one sorted pass
- Fix deadlock in squad migration: use sorted dict iteration for counter updates
- Fix broken "Buy traffic" button: subscription_add_traffic → buy_traffic callback_data
- Add 12 webhook notification toggle settings (WEBHOOK_NOTIFY_*) with master toggle
- Add admin UI category "Уведомления от вебхуков" with hints in BotConfigurationService
- Add toggle check in _notify_user() respecting master and per-event settings
2026-02-12 06:47:26 +03:00
Fringg fc42916b10 fix: harden backup create/restore against serialization and constraint errors
- Backup creation: handle Decimal, float NaN/Inf, fallback for JSON column dumps
- Restore users: savepoint per INSERT to survive duplicate telegram_id/email/referral_code
- Restore associations: savepoint per INSERT to survive FK or duplicate constraint violations
- Restore table records: savepoint already added in prior commit
2026-02-12 03:41:24 +03:00
Fringg 5893874776 fix: handle unique constraint conflicts during backup restore without clear_existing 2026-02-12 03:37:36 +03:00
Fringg 27365b3c75 fix: handle time/date types in backup JSON serialization 2026-02-12 02:51:20 +03:00
Fringg 3dac332a9f chore: ruff format 7 files 2026-02-11 21:50:49 +03:00
Fringg c5124b97b6 fix: payment race conditions, balance atomicity, renewal rollback safety
- YooKassa: SELECT FOR UPDATE on payment row to prevent concurrent double-processing
- subtract_user_balance: row locking to prevent concurrent balance race conditions
- subtract_user_balance: transaction creation before commit for atomicity
- subscription renewal: compensating refund if extend_subscription fails after charge
- StaleDataError: use savepoint instead of full rollback to protect parent transaction
2026-02-11 21:49:37 +03:00
Fringg ee2e79db31 refactor: remove modem functionality from classic subscriptions
Remove all modem purchase/management code:
- Delete modem handler, service, and tests
- Remove modem button from keyboards and admin panel
- Remove modem pricing from calculations
- Remove modem REST API endpoint and schemas
- Remove modem decorator, config settings, and notification formatting
- Keep DB column and migration for backwards compatibility
2026-02-11 21:14:08 +03:00
Fringg c30c2feee1 fix: handle StaleDataError in webhook user.deleted server counter decrement
When a user is deleted from the panel, the subscription may already be
cascade-deleted by the time the webhook handler tries to decrement
server counters. This caused StaleDataError followed by
PendingRollbackError when accessing subscription.id in the error handler.

- Save subscription.id before DB operations to avoid lazy load after rollback
- Catch StaleDataError explicitly and rollback the session
- Re-fetch subscription/user after potential rollback in _handle_user_deleted
- Skip subscription cleanup if it was already cascade-deleted
2026-02-11 18:35:36 +03:00
Fringg 7d9ced8f4f fix: delete subscription_servers before subscription to prevent FK violation
reset_user_subscription and reset_trial endpoints did not clean up
subscription_servers rows before deleting the subscription, causing
ForeignKeyViolationError on subscription_servers.subscription_id_fkey.

Also fixed the same missing cleanup in user_service.hard_delete_user.
2026-02-11 18:25:42 +03:00
Fringg b5998ea9d2 fix: use traffic topup config and add WATA 429 retry
- Cabinet API: use get_traffic_topup_packages() instead of
  get_traffic_packages() in classic mode endpoints (lines 622, 727, 2410)
  to prevent infinite free traffic exploit via initial-purchase packages
- WATA service: add retry logic for 429 rate limit responses with
  Retry-After parsing from header and response body, up to 2 retries,
  downgrade 429 from error to warning log level
2026-02-11 18:20:30 +03:00
Fringg 2094886990 fix: address review issues in backup, updates, and webhook handlers
- backup: add DATE column parsing in restore, use is_file() in delete_backup
- updates: add missing callback.answer() in show_updates_menu early return
- webhook: add server counter decrement and SubscriptionServer cleanup on user deletion, use single commit
2026-02-11 04:09:39 +03:00
Fringg b0fd38d60c fix: clear subscription data when user deleted from Remnawave panel
Previously only status was set to expired and remnawave_uuid cleared.
Now also clears subscription_url, subscription_crypto_link,
remnawave_short_uuid, and connected_squads so the bot correctly
shows no active subscription after panel deletion.
2026-02-11 04:02:09 +03:00
Fringg 02e40bd6f7 fix: expand backup coverage to all 68 models and harden restore
- Add 37 missing models to backup (payment providers, polls, contests,
  wheel, FAQ, promo offers, webhooks, configs, menu buttons, etc.)
- Add tariff_promo_groups and payment_method_promo_groups association tables
- Replace hardcoded association restore with generic handler
- Fix transaction atomicity: flush instead of commit in inner methods,
  remove inner rollback calls, single commit/rollback in outer handler
- Fix composite PK support for UserPromoGroup (was only detecting first PK)
- Fix duplicate insert bug when clear_existing=True and record already exists
- Add cabinet_refresh_tokens to clear list, fix support_audit_logs deletion order
- Add Time column parsing for ReferralContest.daily_summary_time
- Security: tarfile filter='data', path traversal protection in _restore_files
  and delete_backup, os.sep in startswith checks
2026-02-11 03:35:16 +03:00
Fringg a1ffd5bda6 fix: prevent cascading greenlet errors after sync rollback
After db.rollback() all ORM objects expire. Subsequent attribute access
triggers lazy load in async context causing greenlet_spawn errors for
every remaining user. Break the sync loop after rollback instead of
continuing with a corrupted session.

Also downgrade TelegramNetworkError to warning in channel_checker.
2026-02-11 01:39:39 +03:00
Fringg d58a80f3ea fix: handle StaleDataError in webhook when user already deleted
When a user is deleted via cabinet, RemnaWave sends user.disabled webhook
but the subscription row is already cascade-deleted. This caused
StaleDataError on commit + PendingRollbackError when logging user.id.

Save user_id before handler call and catch StaleDataError as warning.
2026-02-11 01:18:58 +03:00
Fringg e43a8d6ce4 fix: downgrade Telegram timeout errors to warning in monitoring service
Add TelegramNetworkError handling before generic Exception catch in all
notification methods to prevent timeout errors from generating error
reports in chat. Timeouts are transient network issues, not bugs.
2026-02-10 23:11:48 +03:00
Fringg 5156d635f0 fix: sync subscription status from panel in user.modified webhook
When subscription was extended in panel, webhook updated end_date but
left status as expired. Now syncs ACTIVE/DISABLED status from panel
payload when end_date is in the future.
2026-02-10 07:49:10 +03:00
Fringg f77922522a fix: allow non-HTTP deep links in crypto link webhook updates
_is_valid_url only accepted http(s), silently dropping valid deep links
like happ://, vless://, ss:// from revoked webhook payloads.
Added _is_valid_link that accepts any URI scheme.
2026-02-10 07:41:32 +03:00
Fringg 0db00a8f90 style: format cryptobot.py with ruff 2026-02-10 07:33:42 +03:00
Fringg fe5f5ded96 feat: add MULENPAY_WEBSITE_URL setting for post-payment redirect
Previously website_url was hardcoded to WEBHOOK_URL, redirecting users
to the webhook endpoint after payment. Now configurable via env var.
2026-02-10 07:25:58 +03:00
Fringg 2cb6d731e9 fix: stop CryptoBot webhook retry loop and save cabinet payments to DB
Cabinet was calling CryptoBotService.create_invoice() directly without
saving CryptoBotPayment to DB. When webhook arrived, payment lookup
failed and returned HTTP 400, causing infinite retries.

Now cabinet uses PaymentService.create_cryptobot_payment() (same as
miniapp) with proper USD conversion via currency_converter.

Also return HTTP 200 for unknown invoice_ids to stop retry spam.
2026-02-10 07:25:54 +03:00
Fringg 184c52d4ea feat: webhook protection — prevent sync/monitoring from overwriting webhook data
Add last_webhook_update_at timestamp to Subscription model. When a webhook
handler modifies a subscription, it stamps this field. Auto-sync, monitoring,
and force-check services skip subscriptions updated by webhook within the
last 60 seconds, preventing stale panel data from overwriting fresh
real-time changes.

- Add last_webhook_update_at column + migration
- Stamp all 8 webhook handlers with commit in every code path
- Add is_recently_updated_by_webhook() guard in 12 sync/monitoring paths
- Add REMNAWAVE_WEBHOOK_* variables to .env.example
- Add webhook setup documentation to README with Caddy/nginx examples
- Fix pre-existing yookassa webhook test (mock AsyncSessionLocal)
2026-02-10 07:16:22 +03:00