bumpy-croc-ai-trading-bot-636-637
Results
The bug report the agent sees
The live trading engine has several reliability defects. All of the following behaviour is required.
Abnormal loop death must be recorded and surfaced as a non-zero exit. `LiveTradingEngine` gains a `_loop_crashed` flag, false on construction. The trading loop is run through a wrapper (`_run_trading_loop`) that catches anything escaping `_trading_loop` and sets `_loop_crashed = True` instead of letting it propagate; exhausting the consecutive-error budget also sets it, while a clean stop such as reaching `max_steps` leaves it false. A new `_exit_if_loop_crashed(exit_on_crash: bool)` raises `SystemExit(1)` when `_loop_crashed` is true and `exit_on_crash` is true, and returns `None` otherwise, both when the loop did not crash and when the caller did not opt in. `start(...)` takes a new keyword `exit_on_crash` defaulting to `False`, routes the loop through the crash-catching wrapper, and calls that helper at the end, so `start("BTCUSDT", "1h", exit_on_crash=True)` on a loop that raises exits with code 1 while the same crash without the opt-in simply returns with `_loop_crashed` true. A transient database error inside the loop is still ridden out: the loop keeps iterating, `consecutive_errors` returns to 0, `db_unreachable_since` is set, and `_loop_crashed` stays false.
Every Binance REST call needs a finite timeout. The provider reads the timeout with `get_config().get_float("BINANCE_REST_TIMEOUT_SECONDS", <default>)` and passes it to the `Client` constructor as `requests_params={"timeout": <value>}`, in every branch that builds a client (authenticated, the `binanceus` tld variant, and unauthenticated), keeping the arguments those branches already pass. A test that patches the provider's `Client` and makes `get_float` return 15.0 asserts `call_args.kwargs.get("requests_params") == {"timeout": 15.0}`. The default lives beside the other timeout constants as `DEFAULT_BINANCE_REST_TIMEOUT = 15.0`. The disconnect-recovery paths must absorb a timeout rather than propagate it: a kline reconnect that raises marks the provider degraded and leaves `_ws_kline_active` false, and a user-stream resync that raises still marks the exchange degraded.
A stop-loss fill is deferred to the trading loop instead of closed on the poll thread. The engine gains `self._pending_fill_exits`, a `queue.SimpleQueue`. When `_handle_order_fill(order_id, symbol, qty, price)` is called with an order id that is a tracked position's stop-loss order, it enqueues exactly the tuple `(position_order_id, price)` and must not call `_execute_exit`; a fill for any other order enqueues nothing. A new `_drain_pending_fill_exits()` runs on the loop, takes each queued item and calls `_execute_exit` for the position with `skip_live_close=True` and `reason="stop_loss"`, leaving the queue empty. The drain looks each position up with `self.live_position_tracker.get_position(position_order_id)` and treats a `None` result as already closed: that item is dropped without calling `_execute_exit`, and the drain carries on with the rest of the queue. Do not detect a closed position any other way, for example through the tracker's `positions` mapping, because a caller may supply a tracker whose only reliable answer is `get_position`. The exit for a live position passes the fill price as both `limit_price` and `current_price`. One failing exit must not abort the drain: with two items queued and the first raising, `_execute_exit` is still called twice, the queue ends empty, and a CRITICAL log containing the text `draining deferred stop-loss exit` is emitted.
The websocket health monitor gets a watchdog. `_ensure_ws_health_monitor_alive()` restarts the monitor by calling `_start_ws_health_monitor()` exactly once when the engine is running, a stream is being watched, and `_ws_health_thread` is absent or its `is_alive()` is false. It must do nothing when that thread is alive, when no stream is configured (`_ws_kline_active` false and no user-data processor), or when the engine is shutting down (`stop_event` set).
IMPORTANT: Work in the repository at /app and commit your changes when you are done. Your submission is the diff from the starting commit.
Hidden tests · 5 fail-to-pass, 87 pass-to-passtests from both pull requests, run after the agent submits
Test patch · 388 lines
diff --git a/tests/unit/data_providers/test_binance_provider.py b/tests/unit/data_providers/test_binance_provider.py
index b2012d9..a95288c 100644
--- a/tests/unit/data_providers/test_binance_provider.py
+++ b/tests/unit/data_providers/test_binance_provider.py
@@ -34,6 +34,25 @@ class TestBinanceDataProvider:
provider = BinanceProvider()
assert provider is not None
+ @pytest.mark.data_provider
+ @patch("src.data_providers.binance_provider.Client")
+ def test_client_constructed_with_rest_timeout(self, mock_client_class):
+ """Every Binance REST call gets a socket timeout via requests_params (#631).
+
+ A timeout-less client lets a half-open TCP socket hang order polling,
+ reconciliation, and the WS disconnect-recovery path indefinitely.
+ """
+ mock_client_class.return_value = Mock()
+ with patch("src.data_providers.binance_provider.get_config") as mock_config:
+ mock_config_obj = Mock()
+ mock_config_obj.get_required.return_value = "fake_key"
+ mock_config_obj.get_float.return_value = 15.0
+ mock_config.return_value = mock_config_obj
+ BinanceProvider()
+
+ assert mock_client_class.called
+ assert mock_client_class.call_args.kwargs.get("requests_params") == {"timeout": 15.0}
+
@pytest.mark.data_provider
@patch("src.data_providers.binance_provider.Client")
def test_binance_historical_data_success(self, mock_client_class):
diff --git a/tests/unit/live/test_order_execution.py b/tests/unit/live/test_order_execution.py
index ac3c3f7..fb56222 100644
--- a/tests/unit/live/test_order_execution.py
+++ b/tests/unit/live/test_order_execution.py
@@ -429,20 +429,31 @@ class TestHandleOrderFill:
assert mock_log.call_args.args[0] == "order_filled"
def test_handle_fill_detects_stop_loss_fill(self, engine_with_exchange, sample_position):
- """Fill callback detects and handles stop-loss fills."""
+ """A stop-loss fill is queued for the trading loop, which then closes it (#631).
+
+ The OrderTracker poll thread must not run the close inline (it blocked all
+ polling and could force-remove the filled order); it only enqueues, and
+ the loop's drain performs the actual close.
+ """
sample_position.stop_loss_order_id = "sl_order_456"
engine_with_exchange.live_position_tracker.track_recovered_position(
sample_position, db_id=None
)
with patch.object(engine_with_exchange, "_execute_exit") as mock_close:
+ # Poll-thread callback: enqueue only, no inline close.
engine_with_exchange._handle_order_fill("sl_order_456", "BTCUSDT", 0.02, 48000.0)
+ mock_close.assert_not_called()
+
+ # Trading-loop drain: perform the deferred close.
+ engine_with_exchange._drain_pending_fill_exits()
mock_close.assert_called_once()
call_args = mock_close.call_args
assert call_args.args[0] == sample_position
assert call_args.kwargs["reason"] == "stop_loss"
assert call_args.kwargs["limit_price"] == 48000.0
+ assert call_args.kwargs["skip_live_close"] is True
def test_handle_fill_skips_if_position_already_closed(
self, engine_with_exchange, sample_position
diff --git a/tests/unit/test_db_resilience.py b/tests/unit/test_db_resilience.py
index 9bd50df..d8eee7a 100644
--- a/tests/unit/test_db_resilience.py
+++ b/tests/unit/test_db_resilience.py
@@ -11,6 +11,9 @@ Covers:
- Loop behaviour: transient DB errors are ridden out (not counted toward
``max_consecutive_errors``); permanent/unrelated errors still trigger
shutdown; a prolonged outage escalates to close-only mode.
+- An abnormal loop death (unhandled crash or error exhaustion) is recorded so
+ ``start()`` exits non-zero for an orchestrator restart instead of exiting 0
+ and leaving the bot silently dead (#630).
"""
from __future__ import annotations
@@ -155,6 +158,8 @@ def test_transient_db_error_in_loop_is_ridden_out() -> None:
assert engine._get_latest_data.call_count == 3
assert engine.consecutive_errors == 0
assert engine.db_unreachable_since is not None
+ # A clean max_steps stop is not a crash — start() would exit 0 here.
+ assert engine._loop_crashed is False
@pytest.mark.fast
@@ -171,6 +176,8 @@ def test_non_transient_error_in_loop_still_shuts_down() -> None:
# Shut down after the first counted error — did not run all five iterations.
assert engine._get_latest_data.call_count == 1
assert engine.is_running is False
+ # Error exhaustion is an abnormal stop: start() must exit non-zero (#630).
+ assert engine._loop_crashed is True
@pytest.mark.fast
@@ -189,3 +196,288 @@ def test_prolonged_db_outage_enters_close_only() -> None:
engine._trading_loop("BTCUSDT", "1h", max_steps=1)
assert engine._close_only_mode is True
+
+
+# --------------------------------------------------------------------------- #
+# Loop-death -> non-zero exit so the orchestrator restarts the process (#630)
+# --------------------------------------------------------------------------- #
+
+
+@pytest.mark.fast
+def test_run_trading_loop_records_unhandled_crash() -> None:
+ """A crash escaping the loop is caught and recorded, not silently swallowed."""
+ engine = _make_engine()
+ engine._trading_loop = Mock(side_effect=RuntimeError("loop blew up"))
+
+ # The thread target must not propagate — it records the crash instead.
+ engine._run_trading_loop("BTCUSDT", "1h")
+
+ assert engine._loop_crashed is True
+
+
+@pytest.mark.fast
+def test_exit_if_loop_crashed_exits_nonzero_when_opted_in() -> None:
+ """An abnormal loop death makes start() exit 1 so Railway ON_FAILURE restarts."""
+ engine = _make_engine()
+ engine._loop_crashed = True
+ engine.main_thread = None # nothing to join in this unit
+
+ with pytest.raises(SystemExit) as exc_info:
+ engine._exit_if_loop_crashed(exit_on_crash=True)
+
+ assert exc_info.value.code == 1
+
+
+@pytest.mark.fast
+def test_exit_if_loop_crashed_is_noop_after_clean_stop() -> None:
+ """A clean stop leaves the process to exit 0 (no SystemExit raised)."""
+ engine = _make_engine()
+ engine._loop_crashed = False
+
+ # Must return normally — start() then exits 0 as before.
+ assert engine._exit_if_loop_crashed(exit_on_crash=True) is None
+
+
+@pytest.mark.fast
+def test_exit_if_loop_crashed_returns_when_not_opted_in() -> None:
+ """A crash must NOT kill library callers (e.g. the migration baseline tool)
+ that drive start() and read results afterward — only the runner opts in."""
+ engine = _make_engine()
+ engine._loop_crashed = True
+ engine.main_thread = None
+
+ # exit_on_crash defaults False: returns instead of calling sys.exit().
+ assert engine._exit_if_loop_crashed(exit_on_crash=False) is None
+
+
+@pytest.mark.fast
+def test_start_exits_nonzero_on_loop_crash_when_opted_in() -> None:
+ """End-to-end wiring: start(exit_on_crash=True) routes the loop through the
+ crash-catching wrapper and exits 1 when it dies (locks the thread target)."""
+ engine = _make_engine()
+ engine.resume_from_last_balance = False
+ engine._start_websocket_streams = Mock() # no real WS in this unit
+ engine._trading_loop = Mock(side_effect=RuntimeError("loop blew up"))
+
+ with pytest.raises(SystemExit) as exc_info:
+ engine.start("BTCUSDT", "1h", exit_on_crash=True)
+
+ assert exc_info.value.code == 1
+ assert engine._loop_crashed is True
+
+
+@pytest.mark.fast
+def test_start_returns_on_loop_crash_when_not_opted_in() -> None:
+ """End-to-end wiring: without opt-in, start() returns even if the loop crashes."""
+ engine = _make_engine()
+ engine.resume_from_last_balance = False
+ engine._start_websocket_streams = Mock()
+ engine._trading_loop = Mo
… [7547 more characters]Reference fix · 4 files, +246 −56used only for grading calibration
Both upstream fixes combined. The agent could not see this: one commit in the repository and no network.
diff --git a/src/engines/live/runner.py b/src/engines/live/runner.py
index c7197247e..499224af3 100644
--- a/src/engines/live/runner.py
+++ b/src/engines/live/runner.py
@@ -287,7 +287,10 @@ def main():
# Start trading
logger.info(f"Starting trading engine for {args.symbol} on {args.timeframe}")
- engine.start(args.symbol, args.timeframe)
+ # Production entry point: exit non-zero on abnormal loop death so the
+ # orchestrator (Railway ON_FAILURE) restarts the process instead of
+ # leaving the bot silently dead (#630).
+ engine.start(args.symbol, args.timeframe, exit_on_crash=True)
except KeyboardInterrupt:
logger.info("🛑 Trading stopped by user")
diff --git a/src/engines/live/trading_engine.py b/src/engines/live/trading_engine.py
index 8aecd9cc3..4d20d62f7 100644
--- a/src/engines/live/trading_engine.py
+++ b/src/engines/live/trading_engine.py
@@ -532,6 +532,9 @@ def __init__(
# Threading
self.main_thread = None
+ # Set when the trading loop dies abnormally (unhandled crash or error
+ # exhaustion) so start() can exit non-zero for an orchestrator restart (#630).
+ self._loop_crashed = False
self.stop_event = threading.Event()
# Optional regime detector (feature-gated)
@@ -1156,8 +1159,20 @@ def _finalize_runtime(self) -> None:
self._runtime_dataset = None
self._runtime_warmup = 0
- def start(self, symbol: str, timeframe: str = "1h", max_steps: int | None = None) -> None:
- """Start the live trading engine"""
+ def start(
+ self,
+ symbol: str,
+ timeframe: str = "1h",
+ max_steps: int | None = None,
+ exit_on_crash: bool = False,
+ ) -> None:
+ """Start the live trading engine.
+
+ ``exit_on_crash`` makes an abnormal loop death exit the process non-zero
+ so an orchestrator restarts it (#630). It defaults to False so start()
+ stays a well-behaved library call for callers that read results after it
+ returns (e.g. the migration baseline tool); the production runner opts in.
+ """
if self.is_running:
logger.warning("Trading engine is already running")
return
@@ -1338,7 +1353,7 @@ def start(self, symbol: str, timeframe: str = "1h", max_steps: int | None = None
# Start main trading loop in separate thread
self.main_thread = threading.Thread(
- target=self._trading_loop, args=(symbol, timeframe, max_steps)
+ target=self._run_trading_loop, args=(symbol, timeframe, max_steps)
)
self.main_thread.daemon = True
self.main_thread.start()
@@ -1352,6 +1367,11 @@ def start(self, symbol: str, timeframe: str = "1h", max_steps: int | None = None
finally:
self.stop()
+ # After a clean stop this is a no-op; after an abnormal loop death it
+ # exits the process non-zero (when opted in) so the orchestrator
+ # restarts it (#630).
+ self._exit_if_loop_crashed(exit_on_crash)
+
def _enter_close_only_mode(self) -> None:
"""Enter close-only mode: no new entries, exits/stops/trailing still active."""
if not self._close_only_mode:
@@ -1657,6 +1677,48 @@ def _signal_handler(self, signum: int, frame: Any) -> None:
self.stop()
sys.exit(0)
+ def _run_trading_loop(self, symbol: str, timeframe: str, max_steps: int | None = None) -> None:
+ """Thread target so an unhandled exception can't kill the loop *silently*.
+
+ A bare daemon thread that raises just vanishes, leaving the process alive
+ but brain-dead (HTTP server up, loop gone) — the zombie that hid the
+ 2026-05-19 outage for 12 days. Catch any unhandled exception, record it,
+ and let start() turn it into a non-zero exit for an orchestrator restart (#630).
+ """
+ try:
+ self._trading_loop(symbol, timeframe, max_steps)
+ except Exception as e:
+ self._loop_crashed = True
+ logger.critical("Trading loop terminated unexpectedly: %s", e, exc_info=True)
+
+ def _exit_if_loop_crashed(self, exit_on_crash: bool) -> None:
+ """Exit the process non-zero if the trading loop died abnormally (#630).
+
+ A clean stop (signal, explicit stop, or max_steps) leaves this a no-op.
+ On an abnormal death (unhandled crash or consecutive-error exhaustion):
+ when *exit_on_crash* (the production runner), exit 1 so the orchestrator
+ restarts the process instead of leaving the bot silently dead; otherwise
+ return so library callers that read results after start() keep working.
+ """
+ if not (self._loop_crashed and exit_on_crash):
+ return
+ # The loop thread may still be unwinding its own shutdown (e.g. closing
+ # positions); wait so a daemon thread isn't killed mid-cleanup. We exit
+ # regardless of the join result (the daemon dies with the process), but
+ # surface a wedged cleanup so it's visible rather than silent.
+ if self.main_thread is not None and self.main_thread != threading.current_thread():
+ self.main_thread.join(timeout=30)
+ if self.main_thread.is_alive():
+ logger.error(
+ "Loop thread still alive after 30s join; exiting anyway — "
+ "shutdown cleanup may be incomplete."
+ )
+ logger.critical(
+ "Trading loop ended abnormally; exiting with code 1 to trigger an "
+ "orchestrator restart instead of running dead."
+ )
+ sys.exit(1)
+
def _trading_loop(self, symbol: str, timeframe: str, max_steps: int | None = None) -> None:
"""Main trading loop"""
from src.infrastructure import liveness # shared loop-liveness for /health (#627)
@@ -2020,6 +2082,8 @@ def _trading_loop(self, symbol: str, timeframe: str, max_steps: int | None = Non
f"Too many consecutive errors ({self.consecutive_errors}). Stopping engine.",
exc_info=True,
)
+ # Abnormal stop: signal start() to exit non-zero for a restart (#630).
+ self._loop_crashed = True
self.stop()
break
# Exponential backoff with adaptive intervals
diff --git a/src/config/constants.py b/src/config/constants.py
index 342b5d01a..bf4e158d8 100644
--- a/src/config/constants.py
+++ b/src/config/constants.py
@@ -279,10 +279,16 @@
# - INFERENCE_TIMEOUT_SECONDS
# - API_REQUEST_TIMEOUT_SECONDS
# - DATA_FETCH_TIMEOUT_SECONDS
+# - BINANCE_REST_TIMEOUT_SECONDS
DEFAULT_MODEL_LOAD_TIMEOUT = 60.0 # Timeout for loading ML models (ONNX, Keras)
DEFAULT_INFERENCE_TIMEOUT = 30.0 # Timeout for model inference
DEFAULT_API_REQUEST_TIMEOUT = 30.0 # Timeout for external API requests
DEFAULT_DATA_FETCH_TIMEOUT = 60.0 # Timeout for historical data fetches
+# Socket timeout (seconds) applied to every Binance REST call via the client's
+# requests_params. Bounds disconnect-path resync/reconnect and order polling so
+# a half-open TCP socket can't hang the WS health thread or loop (#631). Kept
+# below the WS health check interval so a stuck call returns before stalling.
+DEFAULT_BINANCE_REST_TIMEOUT = 15.0
# Numeric Precision Constants
DEFAULT_EPSILON = 1e-9 # Small value for floating point comparisons
diff --git a/src/data_providers/binance_provider.py b/src/data_providers/binance_provider.py
index 7182e30b1..364756952 100644
--- a/src/data_providers/binance_provider.py
+++ b/src/data_providers/binance_provider.py
@@ -24,6 +24,7 @@
from src.config import get_config
from src.config.constants import (
+ DEFAULT_BINANCE_REST_TIMEOUT,
DEFAULT_DATA_FETCH_TIMEOUT,
DEFAULT_STARTUP_BAN_MAX_RETRIES,
DEFAULT_STARTUP_BAN_MAX_WAIT,
@@
… [15203 more characters]DeepSeek V4.1 Flash · max · attempt 1 · solved · 85 steps · $0.083
Submitted diff · 429 lines
diff --git a/src/config/constants.py b/src/config/constants.py
index 342b5d0..0626b92 100644
--- a/src/config/constants.py
+++ b/src/config/constants.py
@@ -279,10 +279,12 @@ DEFAULT_MFE_MAE_LOG_LEVEL = "INFO"
# - INFERENCE_TIMEOUT_SECONDS
# - API_REQUEST_TIMEOUT_SECONDS
# - DATA_FETCH_TIMEOUT_SECONDS
+# - BINANCE_REST_TIMEOUT_SECONDS
DEFAULT_MODEL_LOAD_TIMEOUT = 60.0 # Timeout for loading ML models (ONNX, Keras)
DEFAULT_INFERENCE_TIMEOUT = 30.0 # Timeout for model inference
DEFAULT_API_REQUEST_TIMEOUT = 30.0 # Timeout for external API requests
DEFAULT_DATA_FETCH_TIMEOUT = 60.0 # Timeout for historical data fetches
+DEFAULT_BINANCE_REST_TIMEOUT = 15.0 # Timeout for Binance REST requests (seconds)
# Numeric Precision Constants
DEFAULT_EPSILON = 1e-9 # Small value for floating point comparisons
diff --git a/src/data_providers/binance_provider.py b/src/data_providers/binance_provider.py
index 7182e30..d314027 100644
--- a/src/data_providers/binance_provider.py
+++ b/src/data_providers/binance_provider.py
@@ -24,6 +24,7 @@ import pandas as pd
from src.config import get_config
from src.config.constants import (
+ DEFAULT_BINANCE_REST_TIMEOUT,
DEFAULT_DATA_FETCH_TIMEOUT,
DEFAULT_STARTUP_BAN_MAX_RETRIES,
DEFAULT_STARTUP_BAN_MAX_WAIT,
@@ -420,18 +421,34 @@ class BinanceProvider(DataProvider, ExchangeInterface):
self.testnet,
)
+ rest_timeout = get_config().get_float(
+ "BINANCE_REST_TIMEOUT_SECONDS", DEFAULT_BINANCE_REST_TIMEOUT
+ )
+ requests_params = {"timeout": rest_timeout}
+
if self.api_key and self.api_secret:
logger.debug("Creating authenticated %s client...", api_endpoint)
if api_endpoint == "binanceus":
- client = Client(self.api_key, self.api_secret, testnet=self.testnet, tld="us")
+ client = Client(
+ self.api_key,
+ self.api_secret,
+ testnet=self.testnet,
+ tld="us",
+ requests_params=requests_params,
+ )
else:
- client = Client(self.api_key, self.api_secret, testnet=self.testnet)
+ client = Client(
+ self.api_key,
+ self.api_secret,
+ testnet=self.testnet,
+ requests_params=requests_params,
+ )
else:
logger.debug("Creating public %s client...", api_endpoint)
if api_endpoint == "binanceus":
- client = Client(tld="us")
+ client = Client(tld="us", requests_params=requests_params)
else:
- client = Client()
+ client = Client(requests_params=requests_params)
auth_mode = "with credentials" if self.api_key and self.api_secret else "public mode"
logger.info(
diff --git a/src/engines/live/trading_engine.py b/src/engines/live/trading_engine.py
index 8aecd9c..2ea88ef 100644
--- a/src/engines/live/trading_engine.py
+++ b/src/engines/live/trading_engine.py
@@ -4,6 +4,7 @@ import json
import logging
import math
import os
+import queue
import signal
import sys
import threading
@@ -463,6 +464,9 @@ class LiveTradingEngine:
self.last_account_snapshot = None # Track when we last logged account state
self.timeframe: str | None = None # Will be set when trading starts
self._active_symbol: str | None = None
+ # Stop-loss fills arrive on the OrderTracker poll thread. Defer the
+ # actual position close to the trading loop via this queue.
+ self._pending_fill_exits: queue.SimpleQueue[tuple[str, float]] = queue.SimpleQueue()
# WebSocket stream state (populated during start() if provider supports it)
self._kline_buffer = None
@@ -486,6 +490,10 @@ class LiveTradingEngine:
# outage made Postgres unresolvable and shut the live bot down).
self.db_unreachable_since: float | None = None
self.error_cooldown = DEFAULT_ERROR_COOLDOWN
+ # Set when the trading loop dies abnormally (uncaught exception or
+ # exhausting the consecutive-error budget). start() can surface this
+ # as a non-zero process exit via exit_on_crash.
+ self._loop_crashed = False
# Time exit policy (construct from overrides if not provided)
self.time_exit_policy = time_exit_policy
@@ -1156,7 +1164,13 @@ class LiveTradingEngine:
self._runtime_dataset = None
self._runtime_warmup = 0
- def start(self, symbol: str, timeframe: str = "1h", max_steps: int | None = None) -> None:
+ def start(
+ self,
+ symbol: str,
+ timeframe: str = "1h",
+ max_steps: int | None = None,
+ exit_on_crash: bool = False,
+ ) -> None:
"""Start the live trading engine"""
if self.is_running:
logger.warning("Trading engine is already running")
@@ -1336,9 +1350,11 @@ class LiveTradingEngine:
# Try to start WebSocket streams for reduced API weight
self._start_websocket_streams(symbol, timeframe)
- # Start main trading loop in separate thread
+ # Start main trading loop in separate thread. The wrapper records an
+ # abnormal loop death instead of letting it propagate out of the
+ # worker thread (where Python would only print it to stderr).
self.main_thread = threading.Thread(
- target=self._trading_loop, args=(symbol, timeframe, max_steps)
+ target=self._run_trading_loop, args=(symbol, timeframe, max_steps)
)
self.main_thread.daemon = True
self.main_thread.start()
@@ -1352,6 +1368,28 @@ class LiveTradingEngine:
finally:
self.stop()
+ self._exit_if_loop_crashed(exit_on_crash)
+
+ def _run_trading_loop(
+ self, symbol: str, timeframe: str, max_steps: int | None
… [13863 more characters]step 1
$ cd /repo && ls -la && git log --oneline -5
output · 29 lines
total 396 drwxr-xr-x 1 root root 4096 Nov 5 09:39 . drwxr-xr-x 1 root root 4096 Nov 5 09:38 .. drwxr-xr-x 1 root root 4096 Nov 5 09:38 .agents drwxr-xr-x 1 root root 4096 Nov 5 09:38 .claude -rw-r--r-- 1 root root 230 Nov 5 09:38 .coveragerc drwxr-xr-x 1 root root 4096 Nov 5 09:38 .cursor -rw-r--r-- 1 root root 1450 Nov 5 09:38 .dockerignore -rw-r--r-- 1 root root 1014 Nov 5 09:38 .env.example drwxr-xr-x 1 root root 4096 Nov 5 09:38 .git drwxr-xr-x 1 root root 4096 Nov 5 09:38 .github -rw-r--r-- 1 root root 692 Nov 5 09:38 .gitignore
17 more lines
drwxr-xr-x 1 root root 4096 Nov 5 09:38 .specify -rw-r--r-- 1 root root 29389 Nov 5 09:38 AGENTS.md -rw-r--r-- 1 root root 9339 Nov 5 09:38 CLAUDE.md -rw-r--r-- 1 root root 19924 Nov 5 09:38 CODE.md -rw-r--r-- 1 root root 1399 Nov 5 09:38 Dockerfile -rw-r--r-- 1 root root 1056 Nov 5 09:38 LICENSE -rw-r--r …[1460 chars omitted]… quirements-server.txt -rw-r--r-- 1 root root 918 Nov 5 09:38 requirements.txt drwxr-xr-x 1 root root 4096 Nov 5 09:38 scripts -rw-r--r-- 1 root root 370 Nov 5 09:38 setup.cfg -rw-r--r-- 1 root root 2952 Nov 5 09:38 sitecustomize.py drwxr-xr-x 1 root root 4096 Nov 5 09:38 skills drwxr-xr-x 1 root root 4096 Nov 5 09:38 src drwxr-xr-x 1 root root 4096 Nov 5 09:38 tests 7f7e8ee base
step 2
$ cd /repo && find src -maxdepth 3 -type f | head -100 && echo "---" && grep -rln "class LiveTradingEngine" src tests | head
output · 39 lines
src/config/config_manager.py src/config/constants.py src/config/feature_flags.py src/config/paths.py src/config/providers/base.py src/config/providers/dotenv_provider.py src/config/providers/env_provider.py src/config/providers/railway_provider.py src/config/providers/README.md src/config/providers/__init__.py src/config/README.md src/config/__init__.py
27 more lines
src/dashboards/backtesting/dashboard.py src/dashboards/backtesting/README.md src/dashboards/backtesting/__init__.py src/dashboards/market_prediction/dashboard.py src/dashboards/market_prediction/README.md src/dashboards/market_prediction/__init__.py src/dashboards/monitoring/dashboard.py src/dashboards/monitoring/README.md src/dashboards/monitoring/__init__.py src/dashboards/README.md src/dashboards/__init__.py src/database/admin_ui/app.py src/database/admin_ui/README.md src/database/admin_ui/__init__.py src/database/manager.py src/dat …[2392 chars omitted]… rc/infrastructure/logging/config.py src/infrastructure/logging/context.py src/infrastructure/logging/decision_logger.py src/infrastructure/logging/events.py --- src/engines/live/trading_engine.py tests/integration/live_trading/test_engine_core.py tests/integration/live_trading/test_fallbacks.py tests/integration/live_trading/test_risk_integration.py tests/integration/live_trading/test_thread_safety.py
step 3
$ cd /repo && wc -l src/engines/live/trading_engine.py src/data_providers/binance_provider.py && grep -n "def start\|_trading_loop\|consecutive_errors\|db_unreachable_since\|_handle_order_fill\|_execute_exit\|_ws_health\|stop_event\|_pending\|def _start_ws\|_ws_kline_active\|user_data_processor\|max_steps\|_loop_crashed\|_run_trading_loop\|_drain_pending" src/engines/live/trading_engine.py
output · 24 lines
4606 src/engines/live/trading_engine.py 2102 src/data_providers/binance_provider.py 6708 total 174: max_consecutive_errors: int = 10, # Maximum consecutive errors before shutdown 399: on_fill=self._handle_order_fill, 469: self._user_data_processor = None 470: self._ws_kline_active = False 472: self._ws_health_thread = None 480: self.max_consecutive_errors = max_consecutive_errors 481: self.consecutive_errors = 0 485: # max_consecutive_errors (incident 2026-05-19: a Railway internal-DNS 487: self.db_unreachable_since: float | None = None
12 more lines
535: self.stop_event = threading.Event() 618: max_consecutive_errors=self.max_consecutive_errors, 1159: def start(self, symbol: str, timeframe: str = "1h", max_steps: int | None = None) -> None: 1251: self._pending_balance_correction = Fals …[4400 chars omitted]… 3313: def _execute_exit( 3831: if self.stop_event.is_set(): 3876: self._ws_kline_active 4176: # Run PositionReconciler regardless of position count — resolve_pending_orders 4198: results = reconciler.resolve_pending_orders() 4517: self._execute_exit( 4532: # Model update logic is handled in strategy_manager.apply_pending_update()
step 4
$ cd /repo && sed -n '440,560p' src/engines/live/trading_engine.py
output · 26 lines
strategy.strategy if isinstance(strategy, StrategyRuntime) else strategy
)
# Support component-based Strategy
if isinstance(managed_strategy, ComponentStrategy):
self.strategy_manager = StrategyManager()
self.strategy_manager.current_strategy = managed_strategy
self.strategy_manager.on_strategy_change = self._handle_strategy_change
self.strategy_manager.on_model_update = self._handle_model_update
logger.info("Hot swapping enabled for %s", managed_strategy.__class__.__name__)
else:
logger.info("Hot swapping disabled: provided strategy does not implement Strategy")
14 more lines
# Set up strategy logging if database is available
if self.db_manager:
if hasattr(self.strategy, "set_database_manager"):
…[4547 chars omitted]…
n modeling
self.execution_fill_policy = self._resolve_execution_fill_policy()
self.execution_model = ExecutionModel(self.execution_fill_policy)
# Initialize modular handlers (use injected or create defaults)
self._init_modular_handlers(
position_tracker=position_tracker,
execution_engine=execution_engine,
entry_handler=entry_handler,step 5
$ cd /repo && sed -n '1159,1470p' src/engines/live/trading_engine.py
output · 30 lines
def start(self, symbol: str, timeframe: str = "1h", max_steps: int | None = None) -> None:
"""Start the live trading engine"""
if self.is_running:
logger.warning("Trading engine is already running")
return
self.is_running = True
self._active_symbol = symbol
self.timeframe = timeframe # Store the trading timeframe
# Set base logging context for this engine run
set_context(
component="live_engine",18 more lines
strategy=getattr(self.strategy, "__class__", type("_", (), {})).__name__,
symbol=symbol,
timeframe=timeframe,
)
log_engine_event(
"engine_start",
initial_balance=self.current_balance,
max_position_size=self.max_position_size,
check_interval=self.check_interval,
mode="live" if self.enabl
…[8629 chars omitted]…
nterface
# Check for RESYNCING state (set by error callback) — needs recovery
if getattr(exchange, "_user_ws_state", None) == WebSocketState.RESYNCING:
logger.warning("User data stream in RESYNCING state — triggering recovery")
self._handle_user_stream_disconnect()
return
if getattr(exchange, "_user_ws_state", None) != WebSocketState.PRIMARY:step 6
$ cd /repo && sed -n '1300,1380p' src/engines/live/trading_engine.py
output · 28 lines
elif getattr(self, "_pending_balance_correction", False):
# Balance correction was pending but no session ID available
logger.warning(
"⚠️ Balance correction pending but no trading session ID available - skipping database update"
)
self._pending_balance_correction = False
self._pending_corrected_balance = None
# Set session ID on strategy for logging
if hasattr(self.strategy, "session_id"):
self.strategy.session_id = self.trading_session_id
16 more lines
# Start order tracker for monitoring order fills (live trading only)
if self.order_tracker and self.enable_live_trading:
self.order_tracker.start()
logger.info("📡 Order tracker started")
# Start periodic reconciler (live trading only, not paper mode)
if
…[2387 chars omitted]…
treaming requires credentials (live mode only).
Falls back gracefully if provider doesn't support WebSocket.
"""
from src.engines.live.kline_buffer import KlineBuffer
from src.engines.live.user_data_processor import UserDataProcessor
# Resolve the underlying BinanceProvider for kline streaming.
# CachedDataProvider wraps it; unwrap to access WS methods.step 7
$ cd /repo && sed -n '1650,1720p' src/engines/live/trading_engine.py && echo "=====END=====" && sed -n '1950,2070p' src/engines/live/trading_engine.py
output · 36 lines
)
logger.info("Trading engine stopped")
def _signal_handler(self, signum: int, frame: Any) -> None:
"""Handle shutdown signals"""
logger.info("Received signal %s", signum)
self.stop()
sys.exit(0)
def _trading_loop(self, symbol: str, timeframe: str, max_steps: int | None = None) -> None:
"""Main trading loop"""24 more lines
from src.infrastructure import liveness # shared loop-liveness for /health (#627)
logger.info("Trading loop started")
steps = 0
cfg = get_config()
self._active_symbol = symbol
try:
heartbeat_every = int(cfg.get("ENGINE_HEARTBEAT_STEPS", "60"))
except Exception:
heartbeat_every = 60
while self.is_running and not self.stop_event.is_set():
if max_steps is not None and steps >= max_steps:
logger.i
…[8561 chars omitted]…
"does not exist", # role/database missing — will not self-heal
)
transient_markers = (
"could not translate host name",
"temporary failure in name resolution",
"name or service not known",
"could not connect",
"connection refused",
"server closed the connection",
"connection already closed",step 8
$ cd /repo && sed -n '3100,3345p' src/engines/live/trading_engine.py
output · 35 lines
None,
None,
skip_live_close=False,
)
except Exception as e:
logger.error("Failed to open position: %s", e, exc_info=True)
if self.trading_session_id is not None:
self.db_manager.log_event(
event_type="ERROR",
message=f"Failed to open position: {str(e)}",
severity="error",23 more lines
component="LiveTradingEngine",
details={"stack_trace": str(e)},
session_id=self.trading_session_id,
)
else:
logger.warning("⚠️ Cannot log error to database - no trading session ID available")
def _handle_order_fill(
self, order_id: str, symbol: str, filled_qty: float, avg_price: float
) -> None:
…[8629 chars omitted]…
sition.entry_price <= 0:
logger.error(
"Invalid entry_price %s for position %s - cannot close position safely",
position.entry_price,
position.symbol,
)
return
metrics = self.live_position_tracker.mfe_mae_tracker.get_position_metrics(
position.order_id
)step 9
$ cd /repo && grep -rln "loop_crashed\|run_trading_loop\|exit_on_crash\|pending_fill_exits\|drain_pending_fill\|ws_health_monitor_alive\|BINANCE_REST_TIMEOUT\|requests_params" tests src | head -50
step 10
$ cd /repo && find tests -path '*live*' -o -name '*binance*' | head -100
output · 28 lines
tests/integration/live tests/integration/live/test_correlation_enforcement.py tests/integration/live/test_dynamic_risk_integration_live_engine.py tests/integration/live/test_mfe_mae_live_integration.py tests/integration/live/test_mfe_mae_persistence_throttle.py tests/integration/live/test_partial_operations_live_engine.py tests/integration/live/test_reconciliation_integration.py tests/integration/live/test_regime_integration_live_engine.py tests/integration/live/test_strategy_manager_integration.py tests/integration/live/test_time_exits_live_engine_integration.py tests/integration/live/__init__.py tests/integration/live_trading
16 more lines
tests/integration/live_trading/test_database_concurrency.py tests/integration/live_trading/test_database_error_paths.py tests/integration/live_trading/test_database_logging.py tests/integration/live_trading/test_data_validation.py tests/integration/live_trading …[1358 chars omitted]… cs.py tests/unit/live/test_position_tracker_quantity.py tests/unit/live/test_position_tracker_symbol_guard.py tests/unit/live/test_reconciliation.py tests/unit/live/test_strategy_manager_unit.py tests/unit/live/test_user_data_processor.py tests/unit/test_backtest_live_parity.py tests/unit/test_binance_provider_interest.py tests/unit/test_binance_provider_websocket.py tests/unit/test_health_liveness.py
step 11
$ cd /repo && grep -n "Client(\|DEFAULT_.*TIMEOUT\|get_float\|requests_params\|_handle_kline_disconnect\|_handle_user_stream_disconnect\|_ws_kline_active\|degraded\|_resync\|resync\|timeout" src/data_providers/binance_provider.py | head -100
output · 22 lines
27: DEFAULT_DATA_FETCH_TIMEOUT, 34:from src.infrastructure.timeout import TimeoutError as InfraTimeoutError 35:from src.infrastructure.timeout import run_with_timeout 217: RESYNCING = "resyncing" # Gap detected, running REST reconciliation 218: REST_DEGRADED = "degraded" # WS failed, using REST polling 426: client = Client(self.api_key, self.api_secret, testnet=self.testnet, tld="us") 428: client = Client(self.api_key, self.api_secret, testnet=self.testnet) 432: client = Client(tld="us") 434: client = Client() 618: return _OfflineClient() 645: # Wrap API call with timeout to prevent indefinite hangs 647: data_timeout = get_config().get_float(
10 more lines
648: "DATA_FETCH_TIMEOUT_SECONDS", DEFAULT_DATA_FETCH_TIMEOUT 651: klines = run_with_timeout( 654: …[269 chars omitted]… work/timeout errors (which return None). 1991: # health when it has been started AND is degraded/stale. 2015: def mark_kline_degraded(self) -> None: 2020: def mark_user_degraded(self) -> None: 2026: """Handle kline WebSocket disconnection. Sets kline state; engine handles resync.""" 2031: """Handle user data WebSocket disconnection. Sets user state; engine handles resync."""
step 12
$ cd /repo && sed -n '1,120p' src/data_providers/binance_provider.py && echo "===CLIENT===" && sed -n '390,450p' src/data_providers/binance_provider.py
output · 47 lines
""" Unified Binance Provider This module combines both data provider and exchange functionality for Binance, providing a single interface for all Binance operations including: - Historical and live data fetching - Order execution and management - Account synchronization - Position management """ import logging
35 more lines
import math
import re
import threading
import time
from collections.abc import Callable
from datetime import UTC, datetime, timedelta
from enum import Enum
from functools import wraps
from typing import Any, TypeVar
import pandas as pd
from src.config import get_config
from src.config.constants import (
DEFAULT_DATA_FETCH_TIMEOUT,
DEFAULT_STARTUP_BAN_MAX_RETRIES,
DEFAULT_STARTUP_BAN_MAX_WAIT,
DEFAULT_WS_KLINE_STALENESS_THRESHOLD,
DEFAULT_WS_RECONNECT_MAX_RETRIES,
DEFAULT_WS_USER_STALENESS_THRESHOLD,
)
from src.infrastructure.timeout import TimeoutErro
…[5059 chars omitted]…
api_endpoint.title(),
auth_mode,
self.testnet,
)
logger.debug("Testing client with server time request...")
test_response = client.get_server_time()
logger.debug("Server time test successful: %s", test_response)
# Only promote to self._client after all verification passes
self._client = client
if self._use_margin:step 13
$ cd /repo && grep -n "TIMEOUT" src/config/constants.py
output · 11 lines
173:DEFAULT_THREAD_JOIN_TIMEOUT = 30 # Timeout for main thread join in seconds 174:DEFAULT_ORDER_TRACKER_TIMEOUT = 10 # Timeout for order tracker thread join 175:DEFAULT_REQUEST_TIMEOUT = 10 # Timeout for HTTP requests (webhooks, etc.) 278:# - MODEL_LOAD_TIMEOUT_SECONDS 279:# - INFERENCE_TIMEOUT_SECONDS 280:# - API_REQUEST_TIMEOUT_SECONDS 281:# - DATA_FETCH_TIMEOUT_SECONDS 282:DEFAULT_MODEL_LOAD_TIMEOUT = 60.0 # Timeout for loading ML models (ONNX, Keras) 283:DEFAULT_INFERENCE_TIMEOUT = 30.0 # Timeout for model inference 284:DEFAULT_API_REQUEST_TIMEOUT = 30.0 # Timeout for external API requests 285:DEFAULT_DATA_FETCH_TIMEOUT = 60.0 # Timeout for historical data fetches
step 14
$ cd /repo && sed -n '160,180p' src/config/constants.py && sed -n '270,290p' src/config/constants.py
output · 23 lines
# CPU Optimization Constants DEFAULT_CHECK_INTERVAL = 60 # Base check interval in seconds DEFAULT_MIN_CHECK_INTERVAL = 30 # Minimum check interval (high activity) DEFAULT_MAX_CHECK_INTERVAL = 300 # Maximum check interval (low activity) DEFAULT_PERFORMANCE_MONITOR_INTERVAL = 30 # Performance monitoring interval DEFAULT_SLEEP_POLL_INTERVAL = 0.5 # Sleep polling interval (reduced from 0.1s) DEFAULT_ACCOUNT_SNAPSHOT_INTERVAL = 1800 # Account snapshot interval (30 minutes) DEFAULT_DATA_FRESHNESS_THRESHOLD = 120 # Skip processing if data is older than 2 minutes DEFAULT_MARKET_DATA_STALENESS_THRESHOLD = 3600 # Consider market data stale after 1 hour DEFAULT_MARKET_DATA_LIMIT = 500 # Default number of candles to fetch # Thread and Request Timeout Constants
11 more lines
DEFAULT_THREAD_JOIN_TIMEOUT = 30 # Timeout for main thread join in seconds
DEFAULT_ORDER_TRACKER_TIMEOUT = 10 # Timeout for o
…[901 chars omitted]…
RENCE_TIMEOUT = 30.0 # Timeout for model inference
DEFAULT_API_REQUEST_TIMEOUT = 30.0 # Timeout for external API requests
DEFAULT_DATA_FETCH_TIMEOUT = 60.0 # Timeout for historical data fetches
# Numeric Precision Constants
DEFAULT_EPSILON = 1e-9 # Small value for floating point comparisons
DEFAULT_NORMALIZATION_EPSILON = (
1e-8 # Epsilon for feature normalization to prevent division by zerostep 15
$ cd /repo && sed -n '1880,2102p' src/data_providers/binance_provider.py
output · 35 lines
return
self._last_kline_event_time = datetime.now(UTC)
self._kline_event_received = True
on_kline(msg)
self._kline_event_received = False # Reset until first event confirms
self._kline_socket_key = self._twm.start_kline_socket(
callback=_kline_callback, symbol=symbol, interval=timeframe
)
self._kline_ws_state = WebSocketState.PRIMARY
self._last_kline_event_time = datetime.now(UTC)
return True23 more lines
except Exception as e:
logger.error("Failed to start kline stream: %s", e)
return False
def start_user_stream(self, on_user_event: Callable[[dict], None]) -> bool:
"""Start user data stream. Requires credentials. Live mode only.
Args:
on_user_event: Callback receiving raw user
…[8210 chars omitted]…
X_RETRIES,
e,
)
if attempt < DEFAULT_WS_RECONNECT_MAX_RETRIES:
backoff = 2 ** (attempt - 1)
logger.info("Retrying user stream reconnect in %ds...", backoff)
time.sleep(backoff)
return False
# Aliases for backward compatibility
BinanceDataProvider = BinanceProvider
BinanceExchange = BinanceProviderstep 16
$ cd /repo && sed -n '1440,1615p' src/engines/live/trading_engine.py
output · 31 lines
# Grace period: skip the first check to let streams deliver initial events
self.stop_event.wait(DEFAULT_WS_HEALTH_CHECK_INTERVAL)
while self.is_running and not self.stop_event.is_set():
try:
self._check_kline_health()
self._check_user_stream_health()
except Exception as e:
logger.error("WS health check error: %s", e, exc_info=True)
self.stop_event.wait(DEFAULT_WS_HEALTH_CHECK_INTERVAL)
def _check_kline_health(self) -> None:19 more lines
"""Check kline stream health and reconnect if stale."""
if not self._ws_kline_provider or not self._ws_kline_active:
return
if not getattr(self._ws_kline_provider, "ws_healthy", True):
logger.warning("Kline stream unhealthy — attempting reconnect")
self._handle_kline_disconnect()
def _che
…[6766 chars omitted]…
d price %s. "
"Manual intervention required.",
position.symbol,
current_price,
)
continue
self._execute_exit(
position,
"Engine shutdown",
None,step 17
$ cd /repo && cat REGRESSION_TEST_ISSUE.md
output · 41 lines
# Regression Test Non-Determinism Issue ## Problem The regime regression test (`tests/integration/backtesting/test_regime_regression.py`) produces **different results on CI vs locally**: - **Local**: `total_return = -0.001737383983146934` - **CI**: `total_return = -0.0025730682591151854` - **Difference**: ~0.00084 (~8.4 basis points) This violates the fundamental requirement that **backtests must be deterministic and reproducible** across all environments.
29 more lines
## Root Cause Analysis ### 1. Test Infrastructure Bug The test attempts to stub regime detection using `StubRegimeStrategySwitcher`, but **the stub is not being used**: ```python # Test expects these regimes (from stub): trend_up:low_vol trend_down:high_vol range:low_vol # Actually detects (from real RegimeDetector): range:VolLabel.LOW trend_down:VolLabel.LOW range:VolLabel.LOW ``` **The monkeypatch is not working** - bo …[2080 chars omitted]… al calculations**: Demand bit-exact reproducibility 2. **For integration tests**: Use tolerance-based assertions with reasonable bounds 3. **For regression tests**: Test behavior/properties, not exact floating-point values ## References - Failing PR: #501 - Test file: `tests/integration/backtesting/test_regime_regression.py` - Snapshot: `tests/integration/backtesting/regime_regression_snapshot.json`
step 18
$ cd /repo && grep -rn "LiveTradingEngine(" tests/unit/live tests/integration/live_trading tests/unit/engines/live | head -40output · 16 lines
tests/unit/live/test_close_position_parity.py:49: engine = LiveTradingEngine( tests/unit/live/test_close_position_parity.py:104: engine = LiveTradingEngine( tests/unit/live/test_component_risk_binding.py:53: return LiveTradingEngine(strategy=strategy, **engine_kwargs) tests/unit/live/test_live_engine_comprehensive_safety.py:99: LiveTradingEngine( tests/unit/live/test_live_engine_comprehensive_safety.py:109: LiveTradingEngine( tests/unit/live/test_live_engine_comprehensive_safety.py:119: LiveTradingEngine( tests/unit/live/test_live_engine_comprehensive_safety.py:127: LiveTradingEngine( tests/unit/live/test_live_engine_comprehensive_safety.py:137: LiveTradingEngine( tests/unit/live/test_live_engine_comprehensive_safety.py:145: LiveTradingEngine( tests/unit/live/test_live_engine_comprehensive_safety.py:157: …[2634 chars omitted]…
4 more lines
tests/unit/live/test_live_engine_comprehensive_safety.py:763: engine = LiveTradingEngine( tests/unit/live/test_live_engine_comprehensive_safety.py:775: engine = LiveTradingEngine( tests/unit/live/test_live_engine_comprehensive_safety.py:788: engine = LiveTradingEngine( tests/unit/live/test_live_engine_comprehensive_safety.py:808: engine = LiveTradingEngine(
step 19
$ cd /repo && sed -n '1,130p' tests/unit/live/test_live_engine_comprehensive_safety.py && grep -rn "def engine\|fixture" tests/integration/live_trading/test_engine_core.py | head -30
output · 28 lines
""" Comprehensive safety and reliability tests for the Live Trading Engine. This test suite validates the live trading engine's safety guardrails and reliability mechanisms to ensure it operates safely in production. Test Categories: 1. Initialization & Configuration - Parameter validation, safe defaults 2. Safety Guardrails - Over-leveraging, max positions, balance protection 3. Error Handling & Recovery - API failures, network issues, exceptions 4. Position Management - Entry/exit logic, stop losses, take profits 5. Account Synchronization - Balance tracking, position reconciliation
16 more lines
6. Risk Management Integration - Dynamic risk, correlation control
7. Health Monitoring - Heartbeat, data freshness, error tracking
8. Graceful Shutdown - Signal handling, position cleanup
9. Database Integration - Session management, trade logging
10. Edge Cases - Zero balance, extreme volatility, rap
…[3381 chars omitted]…
data_provider=mock_data_provider,
max_position_size=1.5, # > 100%
enable_live_trading=False,
)
with pytest.raises(ValueError, match="Max position size must be between 0 and 1"):
LiveTradingEngine(
strategy=minimal_strategy,
data_provider=mock_data_provider,
max_position_size=0, # Zerostep 20
$ cd /repo && grep -n "live_position_tracker\|position_tracker" src/engines/live/trading_engine.py | head -40 && grep -n "def get_position\|def has_position\|def positions\|class .*PositionTracker" src/engines/live/*.py src/engines/shared/*.py | head -40
output · 19 lines
59:from src.engines.live.execution.position_tracker import ( 191: position_tracker: LivePositionTracker | None = None, 558: position_tracker=position_tracker, 574: return self.live_position_tracker._positions 579: self.live_position_tracker.reset() 583: self.live_position_tracker.track_recovered_position(position, db_id=None) 597: position_tracker: LivePositionTracker | None, 608: position_tracker: Position tracking handler. 641: self.live_position_tracker = position_tracker or LivePositionTracker( 680: position_tracker=self.live_position_tracker, 1091: for position in self.live_position_tracker.positions.values(): 1325: position_tracker=self.live_position_tracker,
7 more lines
1596: positions_snapshot = self.live_position_tracker.positions 1625: self.live_position …[2425 chars omitted]… gime_strategy_switcher.py:388: def get_position_size_multiplier(self, regime_label: str) -> float: src/engines/live/trading_engine.py:572: def positions(self) -> dict[str, Position]: src/engines/live/trading_engine.py:577: def positions(self, value: dict[str, Position]) -> None: src/engines/shared/side_utils.py:189:def get_position_side(position: HasSide | None, default: str = "long") -> str:
step 21
$ cd /repo && grep -rn "class LivePositionTracker\|def get_position\|def has_position\|def positions\|def open_position" src/engines/live/execution/position_tracker.py | head -50 && sed -n '1,100p' src/engines/live/execution/position_tracker.py
output · 38 lines
70:class LivePositionTracker: 124: def positions(self) -> dict[str, LivePosition]: 144: def has_position(self, order_id: str) -> bool: 149: def has_position_for_symbol(self, symbol: str) -> bool: 159: def get_position(self, order_id: str) -> LivePosition | None: 173: def open_position( 814: def get_position_state(self, order_id: str) -> dict | None: """LivePositionTracker manages active position state and MFE/MAE tracking. Centralizes multi-position lifecycle management including partial exits, scale-ins, and performance metric tracking for live trading. """
26 more lines
from __future__ import annotations
import logging
import math
import threading
from dataclasses import dataclass
from datetime import UTC, datetime
from typing import TYPE_CHECKING
from src.config.constants import (
DEFAULT_EPSILON,
DEFAULT_FEE_RATE,
DEFAULT_MFE_MAE_PRECISION_DECIMALS,
DEFAULT
…[2527 chars omitted]…
ECIMALS,
mfe_mae_update_frequency: float = DEFAULT_MFE_MAE_UPDATE_FREQUENCY_SECONDS,
fee_rate: float = DEFAULT_FEE_RATE,
slippage_rate: float = DEFAULT_SLIPPAGE_RATE,
) -> None:
"""Initialize position tracker.
Args:
db_manager: Database manager for persistence (optional).
mfe_mae_precision: Decimal precision for MFE/MAE calculations.step 22
$ cd /repo && grep -n "requests\.\|urlopen\|http" src/data_providers/binance_provider.py | head -50 && grep -n "Client(" src/data_providers/binance_provider.pyoutput · 5 lines
426: client = Client(self.api_key, self.api_secret, testnet=self.testnet, tld="us") 428: client = Client(self.api_key, self.api_secret, testnet=self.testnet) 432: client = Client(tld="us") 434: client = Client() 618: return _OfflineClient()
step 23
$ cd /repo && sed -n '130,250p' src/engines/live/trading_engine.py && sed -n '1,130p' src/engines/live/trading_engine.py
output · 40 lines
api_key = config.get("BINANCE_TESTNET_API_KEY")
api_secret = config.get("BINANCE_TESTNET_API_SECRET")
provider_name = "Binance Testnet"
else:
api_key = config.get("BINANCE_API_KEY")
api_secret = config.get("BINANCE_API_SECRET")
provider_name = "Binance"
if api_key and api_secret:
return BinanceProvider(api_key, api_secret, testnet=testnet), provider_name
else:
return None, f"{provider_name} (no credentials)"28 more lines
class LiveTradingEngine:
"""
Advanced live trading engine that executes strategies in real-time.
Features:
- Real-time data streaming
- Actual order execution
- Position management
- Risk management
- Sentiment integration
- Error handling & recovery
- Performance monitoring
- Graceful shutdown
"""
def __in
…[8629 chars omitted]…
ret = config.get("COINBASE_API_SECRET")
if api_key and api_secret:
return CoinbaseProvider(api_key, api_secret, testnet=testnet), "Coinbase"
else:
return None, "Coinbase (no credentials)"
else:
# Use testnet credentials if testnet mode is enabled, otherwise use production
if testnet:
api_key = config.get("BINANCE_TESTNET_API_KEY")step 24
$ cd /repo && sed -n '1,30p' src/engines/live/trading_engine.py && grep -n "^import\|^from" src/engines/live/trading_engine.py | head -30
output · 46 lines
from __future__ import annotations import json import logging import math import os import signal import sys import threading import time from dataclasses import asdict from datetime import UTC, datetime, timedelta
34 more lines
from typing import Any
import pandas as pd
from sqlalchemy.exc import DBAPIError, InterfaceError, OperationalError
from src.config import get_config
from src.config.constants import (
DEFAULT_ACCOUNT_SNAPSHOT_INTERVAL,
DEFAULT_CHECK_INTERVAL,
DEFAULT_DATA_FRESHNESS_THRESHOLD,
DEFAULT_DB_OUTAGE_CLOSE_ONLY_SECONDS,
DEFAULT_DYNAMIC_RISK_ENABLED,
DEFAULT_END_OF_DAY_FLAT,
DEFAULT_ERROR_COOLDOWN,
DEFAULT_EXECUTION_FILL_POLICY,
DEFAULT_FEE_RATE,
DEFAULT_INITIAL_BALANCE,
DEFAULT_MARKET_TIMEZONE,
1:from __future__ import annotations
3:import json
4:import logging
5:import math
6:import os
7:import signal
8:import sys
9:import threading
…[868 chars omitted]…
ta_handler import MarketDataHandler
56:from src.engines.live.execution.entry_handler import LiveEntryHandler, LiveEntrySignal
57:from src.engines.live.execution.execution_engine import LiveExecutionEngine
58:from src.engines.live.execution.exit_handler import LiveExitHandler
59:from src.engines.live.execution.position_tracker import (
63:from src.engines.live.health.health_monitor import HealthMonitorstep 25
$ cd /repo && grep -n "def _sleep_with_interrupt" -A20 src/engines/live/trading_engine.py && sed -n '160,360p' tests/integration/live_trading/test_engine_core.py
output · 28 lines
3826: def _sleep_with_interrupt(self, seconds: float): 3827- """Sleep in small increments to allow for interrupt and float seconds""" 3828- end_time = time.time() + seconds 3829- poll_interval = DEFAULT_SLEEP_POLL_INTERVAL # Use configurable interval instead of 0.1 3830- while time.time() < end_time: 3831- if self.stop_event.is_set(): 3832- break 3833- time.sleep(min(poll_interval, end_time - time.time())) 3834- 3835- def _calculate_adaptive_interval(self, current_price: float | None = None) -> int: 3836- """Calculate adaptive check interval based on recent trading activity and market conditions""" 3837- # Base interval from configuration
16 more lines
3838- interval = self.base_check_interval
3839-
3840- # Factor in recent trading activity — use naive UTC for comparison
3841- # since position
…[7852 chars omitted]…
size=1.0, # Use full position size so 1% price move = 1% sized PnL
entry_price=100.0,
entry_time=datetime.now(UTC),
stop_loss=99.0,
order_id="test_trail_001",
)
engine.live_position_tracker.track_recovered_position(position, db_id=None)
import pandas as pd
df = pd.DataFrame({"close": [100.0, 101.0, 102.0, 103.0], "atr": [1.0, 1.0, 1.0, 1.0]})step 26
$ cd /repo && grep -rn "\.start(" src cli tests | grep -i "engine\|live" | head -80output · 19 lines
src/engines/live/order_tracker.py:172: self._thread.start()
src/engines/live/README.md:31:engine.start("BTCUSDT", "1h")
src/engines/live/reconciliation.py:2339: self._thread.start()
src/engines/live/runner.py:290: engine.start(args.symbol, args.timeframe)
src/engines/live/trading_engine.py:1314: self.order_tracker.start()
src/engines/live/trading_engine.py:1331: self._periodic_reconciler.start()
src/engines/live/trading_engine.py:1344: self.main_thread.start()
src/engines/live/trading_engine.py:1413: self._user_data_processor.start()
src/engines/live/trading_engine.py:1434: self._ws_health_thread.start()
src/engines/live/trading_engine.py:1552: self._user_data_processor.start()
cli/commands/live_health.py:167: t.start()
cli/commands/migration.py:185: engine.start(symbol=strategy.ge7 more lines
…[1776 chars omitted]… sor.py:78: processor.start() tests/unit/live/test_user_data_processor.py:94: processor.start() tests/unit/live/test_user_data_processor.py:110: processor.start() tests/unit/live/test_user_data_processor.py:135: processor.start() tests/unit/live/test_user_data_processor.py:164: processor.start() tests/unit/live/test_user_data_processor.py:216: processor.start()
step 27
$ cd /repo && grep -n "def get_float\|def get(" -A20 src/config/config_manager.py | head -80output · 39 lines
54: def get(self, key: str, default: Any | None = None) -> str | None: 55- """ 56- Get a configuration value from the first available provider. 57- 58- Args: 59- key: Configuration key to retrieve 60- default: Default value if key not found in any provider 61- 62- Returns: 63- Configuration value or default 64- """ 65- for provider in self.providers:
27 more lines
66- if provider.is_available(): 67- value = provider.get(key) 68- if value is not None: 69- return value 70- 71- return default 72- 73- def get_required(self, key: str) -> str: 74- """ -- 104: def get_float(self, key: str, default: float = 0.0) -> float: 105- """Get configuration value as float""" 106- value = self.get(key) 107- if value is None: 108- …[192 chars omitted]… : bool = False) -> bool: 115- """Get configuration value as boolean""" 116- value = self.get(key) 117- if value is None: 118- return default 119- 120- # Validate value is string to prevent AttributeError from custom providers 121- if not isinstance(value, str): 122- # Convert to string first for robustness 123- value = str(value) 124-
step 28
$ cd /repo && sed -n '1,160p' tests/unit/data_providers/test_binance_provider.py && grep -n "Client\|_attempt_client_init\|reconnect_kline\|_handle_kline_disconnect\|_handle_user_stream_disconnect\|mark_kline_degraded\|mark_user_degraded" tests/unit/data_providers/test_binance_provider.py tests/unit/test_binance_provider_websocket.py | head -80
output · 37 lines
from datetime import UTC, datetime
from unittest.mock import Mock, patch
import pandas as pd
import pytest
pytestmark = pytest.mark.unit
try:
from src.data_providers.binance_provider import (
STOP_LOSS_LIMIT_SLIPPAGE_FACTOR,
BinanceProvider,25 more lines
with_rate_limit_retry,
)
from src.data_providers.exchange_interface import OrderSide
BINANCE_AVAILABLE = True
except ImportError:
BINANCE_AVAILABLE = False
BinanceProvider = Mock
with_rate_limit_retry = None
STOP_LOSS_LIMIT_SLIPPAGE_FACTOR = 0.005
OrderSide = Mock
@pytest.mark.skipif(not BINANCE_AVAILABLE, reason="Binance provider not available")
class TestBinanceDataProvider:
@pytest.mark.data_provider
def test_binance_provider_initialization(self):
with patch("src.data_providers.binance_provider.get_config") as mock_config:
mock_config_obj = Mock()
…[8629 chars omitted]…
nance_provider_websocket.py:388: result = provider.reconnect_kline()
tests/unit/test_binance_provider_websocket.py:395: def test_reconnect_kline_returns_false_on_failure(self, provider):
tests/unit/test_binance_provider_websocket.py:396: """reconnect_kline() returns False on exception."""
tests/unit/test_binance_provider_websocket.py:405: result = provider.reconnect_kline()step 29
$ cd /repo && sed -n '250,390p' src/data_providers/binance_provider.py
output · 34 lines
Args:
api_key: Binance API key (optional, will try to get from config)
api_secret: Binance API secret (optional, will try to get from config)
testnet: Whether to use testnet/sandbox mode
"""
# Initialize DataProvider
DataProvider.__init__(self)
config = get_config()
# Margin vs spot account routing (A1)
account_type = config.get("BINANCE_ACCOUNT_TYPE", "spot") or "spot"22 more lines
self._use_margin: bool = account_type.lower() == "margin"
trading_mode = config.get("TRADING_MODE", "paper") or "paper"
self._is_live: bool = trading_mode.lower() == "live"
self._margin_symbol_verified: set[str] = set()
# Get credentials from config if not provided
if api_key is None:
api_key = config.get("BINANCE_API_KEY")
if api_secret is None:
…[4726 chars omitted]…
api_endpoint = get_binance_api_endpoint()
is_us = is_us_location()
logger.info(
f"Geo-detection result: {'US location' if is_us else 'Non-US location'} - using {api_endpoint} API"
)
last_error = None
deadline = time.monotonic() + DEFAULT_STARTUP_BAN_MAX_WAIT
for attempt in range(DEFAULT_STARTUP_BAN_MAX_RETRIES + 1):
try:step 30
$ cd /repo && sed -n '110,180p' src/engines/live/execution/position_tracker.py && grep -n "def track_recovered_position\|def open_position" -A80 src/engines/live/execution/position_tracker.py | head -140
output · 39 lines
self.mfe_mae_tracker = MFEMAETracker(
precision_decimals=mfe_mae_precision,
fee_rate=fee_rate,
slippage_rate=slippage_rate,
)
self._mfe_mae_update_frequency = mfe_mae_update_frequency
self._last_mfe_mae_persist: datetime | None = None
# Shared executor for consistent partial exit calculations
self._partial_exit_executor = PartialExitExecutor(
fee_rate=fee_rate,
slippage_rate=slippage_rate,
)27 more lines
@property
def positions(self) -> dict[str, LivePosition]:
"""Get all active positions.
Returns a copy to prevent concurrent modification issues.
"""
with self._positions_lock:
return dict(self._positions)
@property
def position_count(self) -> int:
"""Get count of active positions."""
with self._positions_l
…[7523 chars omitted]…
"
305- with self._positions_lock:
306- position = self._positions.pop(order_id, None)
307- if position is not None:
308- self.mfe_mae_tracker.clear(order_id)
309- self._position_db_ids.pop(order_id, None)
310- return position
311-
312- def close_position(
313- self,
314- order_id: str,
315- exit_price: float,step 31
$ cd /repo && sed -n '2070,2160p' src/engines/live/trading_engine.py
output · 31 lines
"connection already closed",
"ssl connection has been closed",
"terminating connection",
"the database system is starting up",
"connection timed out",
"could not receive data from server",
"no route to host",
)
seen: set[int] = set()
cur: BaseException | None = exc
while cur is not None and id(cur) not in seen:
seen.add(id(cur))19 more lines
text = str(cur).lower()
if any(marker in text for marker in permanent_markers):
return False
if any(marker in text for marker in transient_markers):
return True
if isinstance(cur, OperationalError | InterfaceError):
return True
if isinstance(cur, DBAPIError) and getattr(cur, "connection_invalidated", False):
retu
…[2628 chars omitted]…
try:
# During resync in live mode, return None to trigger skip-cycle.
# Paper mode falls back to REST immediately (no exchange-side SL).
if (
self.enable_live_trading
and self._ws_kline_provider
and getattr(self._ws_kline_provider, "_kline_ws_state", None)
== WebSocketState.RESYNCING
):step 32
$ cd /repo && grep -rn "_ws_kline_active\|_ws_health_thread\|_start_ws_health_monitor\|_handle_kline_disconnect\|_handle_user_stream_disconnect" tests | head -100
output · 14 lines
tests/unit/test_trading_engine_websocket.py:139: mock_engine._ws_kline_active = True tests/unit/test_trading_engine_websocket.py:155: mock_engine._ws_kline_active = False tests/unit/test_trading_engine_ws_health.py:61: """Should call _handle_kline_disconnect when ws_healthy is False.""" tests/unit/test_trading_engine_ws_health.py:65: mock_engine._ws_kline_active = True tests/unit/test_trading_engine_ws_health.py:67: with patch.object(mock_engine, "_handle_kline_disconnect") as mock_disconnect: tests/unit/test_trading_engine_ws_health.py:77: mock_engine._ws_kline_active = True tests/unit/test_trading_engine_ws_health.py:79: with patch.object(mock_engine, "_handle_kline_disconnect") as mock_disconnect: tests/unit/test_trading_engine_ws_health.py:87: mock_engine._ws_kline_active = False tests/unit/test_trading_engine_ws_health.py:89 …[941 chars omitted]… tests/unit/test_trading_engine_ws_health.py:168: assert mock_engine._ws_kline_active is False tests/unit/test_trading_engine_ws_health.py:172: """Tests for _handle_user_stream_disconnect()."""
2 more lines
tests/unit/test_trading_engine_ws_health.py:189: mock_engine._handle_user_stream_disconnect() tests/unit/test_trading_engine_ws_health.py:212: mock_engine._handle_user_stream_disconnect()
step 33
$ cd /repo && cat tests/unit/test_trading_engine_ws_health.py && echo "===WEBSOCKET===" && sed -n '1,240p' tests/unit/test_trading_engine_websocket.py
output · 39 lines
"""Tests for WebSocket health monitoring in LiveTradingEngine.""" from datetime import UTC, datetime, timedelta from unittest.mock import MagicMock, patch import pandas as pd import pytest from src.data_providers.binance_provider import WebSocketState @pytest.fixture
27 more lines
def mock_engine():
"""Create a LiveTradingEngine with mocked dependencies for WS health testing."""
with patch("src.engines.live.trading_engine.DatabaseManager"):
from src.engines.live.trading_engine import LiveTradingEngine
mock_strategy = MagicMock()
mock_strategy.get_risk_overrides.return_value = {}
mock_strategy.__class__.__name__ = "MockStrategy"
mock_strategy.config = {}
mock_dp = MagicMock()
mock_dp.get_live_data.return_value = _make_df()
mock_dp.get_current_price.return_value = 50000.0
engine = LiveTradingEngine(
st
…[8629 chars omitted]…
top_streams.assert_called_once()
mock_udp.stop.assert_called_once()
@pytest.mark.fast
def test_stop_handles_no_ws_gracefully(self, mock_engine):
"""stop() works when no WS was set up."""
mock_engine.is_running = True
mock_engine.main_thread = MagicMock()
mock_engine.main_thread.is_alive.return_value = False
mock_engine.stop() # Should not raisestep 34
$ cd /repo && sed -n '130,230p' tests/unit/test_trading_engine_ws_health.py
output · 38 lines
class TestHandleKlineDisconnect:
"""Tests for _handle_kline_disconnect()."""
@pytest.mark.fast
def test_resyncs_buffer_and_reconnects(self, mock_engine):
"""Should resync from REST and reconnect kline stream."""
mock_provider = MagicMock()
mock_provider.reconnect_kline.return_value = True
mock_engine._ws_kline_provider = mock_provider
mock_engine._ws_kline_active = False
mock_buffer = MagicMock()26 more lines
mock_engine._kline_buffer = mock_buffer
mock_engine._handle_kline_disconnect()
mock_buffer.resync_from_rest.assert_called_once_with(
mock_engine.data_provider, "BTCUSDT", "1h"
)
mock_provider.reconnect_kline.assert_called_once()
assert mock_engine._ws_kline_active is True
@pytest.mark.fast
def test_sets_rest_degraded_on_reconnect_failure(self, mock_engine):
…[1839 chars omitted]…
mock_engine.exchange_interface = mock_exchange
mock_tracker = MagicMock()
mock_engine.order_tracker = mock_tracker
mock_reconciler = MagicMock()
mock_engine._periodic_reconciler = mock_reconciler
mock_engine._handle_user_stream_disconnect()
mock_exchange.mark_user_degraded.assert_called_once()
mock_tracker.enable_polling.assert_called_once()step 35
$ cd /repo && python - <<'PY'
output · 1 lines
286:DEFAULT_BINANCE_REST_TIMEOUT = 15.0 # Timeout for Binance REST requests (seconds)
step 36
$ cd /repo && python - <<'PY'
output · 37 lines
from functools import wraps
from typing import Any, TypeVar
import pandas as pd
from src.config import get_config
from src.config.constants import (
DEFAULT_BINANCE_REST_TIMEOUT,
DEFAULT_DATA_FETCH_TIMEOUT,
DEFAULT_STARTUP_BAN_MAX_RETRIES,
DEFAULT_STARTUP_BAN_MAX_WAIT,
DEFAULT_WS_KLINE_STALENESS_THRESHOLD,25 more lines
DEFAULT_WS_RECONNECT_MAX_RETRIES,
DEFAULT_WS_USER_STALENESS_THRESHOLD,
)
from src.infrastructure.timeout import TimeoutError as InfraTimeoutError
from src.infrastructure.timeout import run_with_timeout
from src.trading.symbols.factory import SymbolFactory
"Attempting to create %s client - has_credentials: %s, testnet: %s",
api_endpoint,
bool(self.api_key and self.api_secret),
self.testnet,
)
rest_timeout = get_config().get_float(
"BINANCE_REST_TIMEOUT_SECONDS", DEFAULT_BINANCE
…[792 chars omitted]…
..", api_endpoint)
if api_endpoint == "binanceus":
client = Client(tld="us", requests_params=requests_params)
else:
client = Client(requests_params=requests_params)
auth_mode = "with credentials" if self.api_key and self.api_secret else "public mode"
logger.info(
"%s client initialized successfully (%s, testnet: %s)",step 37
$ cd /repo && python - <<'PY'
output · 3 lines
7:import queue 469: self._pending_fill_exits: queue.SimpleQueue[tuple[str, float]] = queue.SimpleQueue() 496: self._loop_crashed = False
step 38
$ cd /repo && python - <<'PY'
output · 34 lines
timeframe: str = "1h",
max_steps: int | None = None,
exit_on_crash: bool = False,
) -> None:
"""Start the live trading engine"""
if self.is_running:
logger.warning("Trading engine is already running")
return
self.is_running = True
self._active_symbol = symbol
self.timeframe = timeframe # Store the trading timeframe22 more lines
# Set base logging context for this engine run
set_context(
component="live_engine",
strategy=getattr(self.strategy, "__class__", type("_", (), {})).__name__,
symbol=symbol,
timeframe=timeframe,
)
log_engine_event(
"engine_start",
initial_balance=self.current_balance,
max_position_size=self.max_position_size,
check_interval=self.check_interval,
…[2641 chars omitted]…
"""Initialize WebSocket streams for reduced API weight.
Kline streaming works in both paper and live mode.
User data streaming requires credentials (live mode only).
Falls back gracefully if provider doesn't support WebSocket.
"""
from src.engines.live.kline_buffer import KlineBuffer
from src.engines.live.user_data_processor import UserDataProcessorstep 39
$ cd /repo && python - <<'PY'
output · 19 lines
1475: def _ensure_ws_health_monitor_alive(self) -> None: 1476- """Restart the WebSocket health monitor if it died unexpectedly. 1477- 1478- Does nothing while the monitor is alive, when no stream is configured, 1479- or once the engine has begun shutting down. 1480- """ 1481- if not self.is_running or self.stop_event.is_set(): 1482- return 1483- if not self._ws_kline_active and not self._user_data_processor: 1484- return 1485- thread = getattr(self, "_ws_health_thread", None) 1486- if thread is not None and thread.is_alive():
7 more lines
1487- return 1488- self._start_ws_health_monitor() 1489- 1490- def _ws_health_loop(self) -> None: 1491- """Monitor WebSocket streams and trigger reconnection on failure.""" 1492- from src.config.constants import DEFAULT_WS_HEALTH_CHECK_INTERVAL 1493-
step 40
$ cd /repo && python - <<'PY'
output · 30 lines
# Resync kline history from REST
if self._kline_buffer:
try:
self._kline_buffer.resync_from_rest(
self.data_provider, self._active_symbol, self.timeframe
)
except Exception as e:
logger.error("Kline REST resync failed: %s", e)
# Attempt reconnect. A timeout (or any other error) during reconnect
# must fall back to REST polling rather than escape the health loop.
reconnected = False
if hasattr(self._ws_kline_provider, "reconnect_kline"):18 more lines
try:
reconnected = bool(self._ws_kline_provider.reconnect_kline())
except Exception as e:
logger.error("Kline reconnect failed with error: %s", e, exc_info=True)
if reconnected:
self._ws_kline_active = True
logger.info("Kline We
…[4505 chars omitted]…
ker
if self.order_tracker:
self.order_tracker.stop()
logger.info("📡 Order tracker stopped")
# Close or preserve open positions depending on trading mode
positions_snapshot = self.live_position_tracker.positions
if positions_snapshot:
if self.enable_live_trading:
# LIVE: close all positions on exchange before shutdown.step 41
$ cd /repo && python - <<'PY'
output · 35 lines
self._print_final_stats()
# End the trading session in database
if self.trading_session_id:
self.db_manager.end_trading_session(
session_id=self.trading_session_id, final_balance=self.current_balance
)
logger.info("Trading engine stopped")
def _signal_handler(self, signum: int, frame: Any) -> None:
"""Handle shutdown signals"""23 more lines
logger.info("Received signal %s", signum)
self.stop()
sys.exit(0)
def _trading_loop(self, symbol: str, timeframe: str, max_steps: int | None = None) -> None:
"""Main trading loop"""
from src.infrastructure import liveness # shared loop-liveness for /health (#627)
logger.info("Trading loop started")
steps = 0
cfg = get_config()
self._active_symbol = symbol
try:
heartbeat_every =
…[1481 chars omitted]…
t is an abnormal death, not a
# clean operator-requested stop.
self._loop_crashed = True
self.stop()
break
# Exponential backoff with adaptive intervals
sleep_time = min(self.error_cooldown, self.check_interval * self.consecutive_errors)
self._sleep_with_interrupt(sleep_time)step 42
$ cd /repo && sed -n '1728,1760p' src/engines/live/trading_engine.py
output · 28 lines
from src.infrastructure import liveness # shared loop-liveness for /health (#627)
logger.info("Trading loop started")
steps = 0
cfg = get_config()
self._active_symbol = symbol
try:
heartbeat_every = int(cfg.get("ENGINE_HEARTBEAT_STEPS", "60"))
except Exception:
heartbeat_every = 60
while self.is_running and not self.stop_event.is_set():
if max_steps is not None and steps >= max_steps:16 more lines
logger.info("Reached max_steps=%s, stopping engine for test.", max_steps)
self.stop()
break
steps += 1
liveness.beat() # record loop liveness for the /health endpoint (#627)
# Process fills deferred from the OrderTracker poll thread, and
# make sure the WebSocket health monitor has not silently died.
…[381 chars omitted]…
self.data_provider.update_live_data(symbol, timeframe)
except Exception as e:
logger.debug("update_live_data failed: %s", e)
# Fetch latest market data
df = self._get_latest_data(symbol, timeframe)
if df is None or df.empty:
log_data_event("no_data", reason="empty_frame")step 43
$ cd /repo && python - <<'PY'
output · 38 lines
None,
emergency_price,
None,
None,
None,
skip_live_close=False,
)
except Exception as e:
logger.error("Failed to open position: %s", e, exc_info=True)
if self.trading_session_id is not None:
self.db_manager.log_event(26 more lines
event_type="ERROR",
message=f"Failed to open position: {str(e)}",
severity="error",
component="LiveTradingEngine",
details={"stack_trace": str(e)},
session_id=self.trading_session_id,
)
else:
logger.warning("⚠️ Cannot log error to database - no trading session ID available")
def _handle_order_fil
…[2459 chars omitted]…
)
continue
try:
self._execute_exit(
position,
reason="stop_loss",
limit_price=price,
current_price=price,
candle_high=None,
candle_low=None,
candle=None,
skip_live_close=True,
)step 44
$ cd /repo && python -m py_compile src/engines/live/trading_engine.py src/data_providers/binance_provider.py src/config/constants.py && git diff --stat && git diff -- src/config/constants.py src/data_providers/binance_provider.py | head -120
output · 28 lines
src/config/constants.py | 1 + src/data_providers/binance_provider.py | 25 +++- src/engines/live/trading_engine.py | 256 +++++++++++++++++++++++---------- 3 files changed, 201 insertions(+), 81 deletions(-) diff --git a/src/config/constants.py b/src/config/constants.py index 342b5d0..7a4ce70 100644 --- a/src/config/constants.py +++ b/src/config/constants.py @@ -283,6 +283,7 @@ DEFAULT_MODEL_LOAD_TIMEOUT = 60.0 # Timeout for loading ML models (ONNX, Keras) DEFAULT_INFERENCE_TIMEOUT = 30.0 # Timeout for model inference DEFAULT_API_REQUEST_TIMEOUT = 30.0 # Timeout for external API requests DEFAULT_DATA_FETCH_TIMEOUT = 60.0 # Timeout for historical data fetches
16 more lines
+DEFAULT_BINANCE_REST_TIMEOUT = 15.0 # Timeout for Binance REST requests (seconds)
# Numeric Precision Constants
DEFAULT_EPSILON = 1e-9 # Small value for floating point comparisons
diff --git a
…[1692 chars omitted]…
)
if api_endpoint == "binanceus":
- client = Client(tld="us")
+ client = Client(tld="us", requests_params=requests_params)
else:
- client = Client()
+ client = Client(requests_params=requests_params)
auth_mode = "with credentials" if self.api_key and self.api_secret else "public mode"
logger.info(step 45
$ cd /repo && cat > /tmp/repro_reliability.py <<'PY'
output · 28 lines
Trading loop crashed unexpectedly: boom
Traceback (most recent call last):
File "/repo/src/engines/live/trading_engine.py", line 1378, in _run_trading_loop
self._trading_loop(symbol, timeframe, max_steps)
File "/usr/local/lib/python3.12/unittest/mock.py", line 1139, in __call__
return self._mock_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/unittest/mock.py", line 1143, in _mock_call
return self._execute_mock_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/unittest/mock.py", line 1198, in _execute_mock_call
raise effect16 more lines
RuntimeError: boom
Trading loop crashed — exiting with status 1
⚠️ PAPER TRADING MODE - No real orders will be executed
Trading loop crashed unexpectedly: boom
Traceback (most recent call last):
File "/repo/src/engines/live/trading_en
…[2660 chars omitted]…
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/unittest/mock.py", line 1143, in _mock_call
return self._execute_mock_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/unittest/mock.py", line 1198, in _execute_mock_call
raise effect
TimeoutError: timeout
Configuration providers available: ['Environment Variables']
all repro checks passedstep 46
$ cd /repo && cat > /tmp/repro_provider_timeout.py <<'PY'
output · 5 lines
binance True call('aaaaaaaaaaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbbbbbbbb', testnet=False, requests_params={'timeout': 15.0})
binance False call(requests_params={'timeout': 15.0})
binanceus True call('aaaaaaaaaaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbbbbbbbb', testnet=False, tld='us', requests_params={'timeout': 15.0})
binanceus False call(tld='us', requests_params={'timeout': 15.0})
provider timeout checks passedstep 47
$ cd /repo && cat > /tmp/repro_loop.py <<'PY'
output · 25 lines
Database temporarily unreachable in trading loop (Exception); backing off 30s and retrying — not counted toward shutdown (unreachable for 0s).
Database temporarily unreachable in trading loop (Exception); backing off 30s and retrying — not counted toward shutdown (unreachable for 0s).
Database temporarily unreachable in trading loop (Exception); backing off 30s and retrying — not counted toward shutdown (unreachable for 0s).
Error in trading loop (#1): bad
Traceback (most recent call last):
File "/repo/src/engines/live/trading_engine.py", line 1758, in _trading_loop
df = self._get_latest_data(symbol, timeframe)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/unittest/mock.py", line 1139, in __call__
return self._mock_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/unittest/mock.py", line 13 more lines
…[712 chars omitted]…
est/mock.py", line 1143, in _mock_call
return self._execute_mock_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/unittest/mock.py", line 1198, in _execute_mock_call
raise effect
RuntimeError: bad
No market data received
No market data received
Configuration providers available: ['Environment Variables']
transient ok
budget ok
clean okstep 48
$ cd /repo && python -m pytest tests/unit/test_trading_engine_ws_health.py tests/unit/test_trading_engine_websocket.py -q 2>&1 | tail -40
output · 8 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [100%][0m ============================= slowest 10 durations ============================= 1.71s setup tests/unit/test_trading_engine_ws_health.py::TestHandleUserStreamDisconnect::test_enables_polling_on_reconnect_failure 0.06s call tests/unit/test_trading_engine_websocket.py::TestGetLatestDataWithWebSocket::test_falls_back_to_rest_during_resyncing_paper_mode 0.01s call tests/unit/test_trading_engine_ws_health.py::TestHandleUserStreamDisconnect::test_enables_polling_on_reconnect_failure (7 durations < 0.005s hidden. Use -vv to show these durations.) [32m[32m[1m18 passed[0m[32m in 2.37s[0m[0m
step 49
$ cd /repo && python -m pytest tests/unit/data_providers/test_binance_provider.py tests/unit/test_binance_provider_websocket.py tests/unit/test_binance_provider_interest.py -q 2>&1 | tail -50
output · 7 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [ 80%] [0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m …[1152 chars omitted]… ers/test_binance_provider.py::TestBinanceDataProvider::test_binance_provider_initialization 3.01s call tests/unit/data_providers/test_binance_provider.py::TestMarginStartupChecks::test_margin_startup_checks_pass 3.01s call tests/unit/data_providers/test_binance_provider.py::TestPlaceStopLossOrder::test_generic_exception_returns_none [32m[32m[1m90 passed[0m[32m in 136.72s (0:02:16)[0m[0m
step 50
$ cd /repo && python -m pytest tests/unit/live/test_live_engine_comprehensive_safety.py tests/unit/live/test_handler_bug_fixes.py tests/unit/engines/live/test_exit_conditions.py -q 2>&1 | tail -50
output · 8 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [ 96%] [0m[32m.[0m[32m.[0m[32m.[0m[32m [100%][0m ============================= slowest 10 dur …[1023 chars omitted]… _live_engine_comprehensive_safety.py::TestEdgeCasesAndStress::test_regime_detector_optional 0.72s setup tests/unit/live/test_live_engine_comprehensive_safety.py::TestEdgeCasesAndStress::test_partial_operations_configuration 0.72s setup tests/unit/live/test_live_engine_comprehensive_safety.py::TestEdgeCasesAndStress::test_very_small_position_size [32m[32m[1m75 passed[0m[32m in 42.19s[0m[0m
step 51
$ cd /repo && python -m pytest tests/integration/live_trading/test_engine_core.py tests/integration/live_trading/test_database_concurrency.py -q 2>&1 | tail -60
output · 13 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [100%][0m ============================= slowest 10 durations ============================= 1.23s call tests/integration/live_trading/test_database_concurrency.py::test_concurrent_open_close_with_database 0.59s call tests/integration/live_trading/test_engine_core.py::test_stop_loss_update_with_component_strategy 0.59s call tests/integration/live_trading/test_engine_core.py::test_stop_loss_from_component_strategy 0.59s call tests/integration/live_trading/test_engine_core.py::test_position_management_with_trading_decision 0.58s call tests/integration/live_trading/test_engine_core.py::test_order_execution_with_component_strategy 0.58s call …[217 chars omitted]… 0.04s call tests/integration/live_trading/test_engine_core.py::TestLiveTradingEngine::test_maximum_position_limits 0.03s call tests/integration/live_trading/test_engine_core.py::test_position_sizing_from_trading_decision 0.03s call tests/integration/live_trading/test_engine_core.py::TestLiveTradingEngine::test_position_opening_paper_trading
1 more lines
[32m[32m[1m17 passed[0m[32m in 6.77s[0m[0m
step 52
$ cd /repo && git diff -- src/engines/live/trading_engine.py | sed -n '1,260p'
output · 29 lines
diff --git a/src/engines/live/trading_engine.py b/src/engines/live/trading_engine.py index 8aecd9c..c5bdbef 100644 --- a/src/engines/live/trading_engine.py +++ b/src/engines/live/trading_engine.py @@ -4,6 +4,7 @@ import json import logging import math import os +import queue import signal import sys import threading
17 more lines
@@ -463,6 +464,9 @@ class LiveTradingEngine:
self.last_account_snapshot = None # Track when we last logged account state
self.timeframe: str | None = None # Will be set when trading starts
self._active_symbol: str | None = None
+ # Stop-loss fills arrive on the OrderTracker poll thread. Defer the
+ # actual position close to the trading loop via this queue.
+ self._pending_fill_exits: queue.SimpleQueue[tuple[str, float]] = queue.SimpleQueue()
# WebSocket stream state (populated during start() if provider su
…[8629 chars omitted]…
self._ensure_ws_health_monitor_alive()
try:
# For mock and real providers, update live data if supported.
# Skip when WS kline cache is active (no REST needed).
@@ -2020,6 +2090,9 @@ class LiveTradingEngine:
f"Too many consecutive errors ({self.consecutive_errors}). Stopping engine.",
exc_info=True,step 53
$ cd /repo && git diff -- src/engines/live/trading_engine.py | sed -n '260,520p'
output · 30 lines
exc_info=True,
)
+ # Exhausting the error budget is an abnormal death, not a
+ # clean operator-requested stop.
+ self._loop_crashed = True
self.stop()
break
# Exponential backoff with adaptive intervals
@@ -3142,34 +3215,63 @@ class LiveTradingEngine:
average_price=avg_price,
)
18 more lines
- # Check if this is a stop-loss order fill - need to close the position.
- # positions property returns a thread-safe copy. _execute_exit has its
- # own defensive has_position() check to handle the race where another
- # thread closes the position between our lookup and the exit call.
- position_to_close: Position | None = None
+ # Check if this is a stop-loss order fill. The fill arr
…[2852 chars omitted]…
ueued fills
+ # from being processed.
+ logger.critical(
+ "Error draining deferred stop-loss exit for position %s: %s",
+ position_order_id,
+ e,
+ exc_info=True,
+ )
def _handle_partial_fill(
self, order_id: str, symbol: str, new_filled_qty: float, avg_price: floatstep 54
$ cd /repo && cat pyproject.toml && cat setup.cfg && cat pytest.ini | head -80
output · 51 lines
[tool.black] line-length = 100 [tool.ruff] line-length = 100 [tool.ruff.lint] select = ["E","F","I","B","UP"] ignore = ["E203","E501","E402"] [tool.ruff.lint.isort] known-first-party = ["src","config","data_providers","database","live","monitoring","prediction","risk","strategies","utils","backtesting"]
39 more lines
[build-system]
requires = ["setuptools>=67", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "ai-trading-bot"
version = "0.1.0"
description = "AI Trading Bot CLI and tools"
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
"python-binance>=1.0.36",
"nest_asyncio>=1.6.0",
"pandas>=2.1.4",
"python-dotenv>=1.0.0",
"numpy>=1.26.2",
"scikit-learn>=1.3.2",
"requests>=2.31.0",
"textblob>=0.17.1",
"nltk>=3.9.0",
"ccxt>=4.4.85",
"tqdm>=4.66.1",
"sqlalchemy>=2.0.23",
"psycopg2-binary>=2.9.9",
"psutil>=5
…[2715 chars omitted]…
ilterwarnings =
ignore::DeprecationWarning
ignore::PendingDeprecationWarning
ignore::ResourceWarning:testcontainers
# Test collection optimizations
# Note: collect_ignore_glob is not a valid pytest.ini option
# Use pytest.ini addopts with --ignore instead if needed
# Removed coverage configuration; moved to .coveragerc
# Removed custom [pytest:ini] test_groups; not a valid pytest optionstep 55
$ cd /repo && (ruff check src/engines/live/trading_engine.py src/data_providers/binance_provider.py src/config/constants.py || true) && (black --check src/engines/live/trading_engine.py src/data_providers/binance_provider.py src/config/constants.py || true)
output · 2 lines
bash: line 942: ruff: command not found bash: line 942: black: command not found
step 56
$ cd /repo && grep -rn "_handle_order_fill\|pending_fill\|stop_loss_order_id" tests | head -100
output · 14 lines
tests/integration/live/test_reconciliation_integration.py:169: stop_loss_order_id: str | None = "sl_orig_001", tests/integration/live/test_reconciliation_integration.py:191: if stop_loss_order_id and db_id: tests/integration/live/test_reconciliation_integration.py:192: db_manager.update_position(position_id=db_id, stop_loss_order_id=stop_loss_order_id) tests/integration/live/test_reconciliation_integration.py:208: stop_loss_order_id=stop_loss_order_id, tests/integration/live/test_reconciliation_integration.py:395: assert position.stop_loss_order_id == "sl_new_002" tests/integration/live/test_reconciliation_integration.py:486: stop_loss_order_id=None, tests/integration/live/test_reconciliation_integration.py:508: assert position.stop_loss_order_id == "sl_new_003" tests/integration/live/test_reconciliation_integration.py:556: …[5899 chars omitted]… 77: assert pos.stop_loss_order_id == "new_sl_default" tests/unit/live/test_reconciliation.py:2488: stop_loss_order_id=None, tests/unit/live/test_reconciliation.py:2521: assert pos.stop_loss_order_id is None
2 more lines
tests/unit/live/test_reconciliation.py:2906: stop_loss_order_id="sl_777", tests/unit/test_reconciliation_interest.py:44: stop_loss_order_id: str | None = None
step 57
$ cd /repo && sed -n '1,180p' tests/unit/live/test_order_execution.py && sed -n '400,550p' tests/unit/live/test_order_execution.py
output · 39 lines
""" Tests for live trading order execution methods. This module tests the following components: - LiveExecutionEngine entry/exit order placement - _handle_order_fill: Handles order fill callbacks (including stop-loss detection) - _handle_partial_fill: Handles partial fill callbacks (with SL warning) - _handle_order_cancel: Handles order cancellation (with phantom position removal) - _reconcile_positions_with_exchange: Reconciles local state with exchange on startup """ from __future__ import annotations
27 more lines
from datetime import UTC, datetime
from decimal import Decimal
from unittest.mock import MagicMock, Mock, patch
import pytest
from src.data_providers.exchange_interface import (
OrderSide,
OrderType,
)
from src.data_providers.exchange_interface import (
OrderStatus as ExchangeOrderStatus,
)
from src.engines.live.execution.execution_engine import LiveExecutionEngine
fro
…[8629 chars omitted]…
filled_quantity=0.01,
average_price=48000.0,
)
def test_handle_partial_fill_no_matching_position(self, engine_with_exchange):
"""Partial fill with no matching position should not error."""
engine_with_exchange._handle_partial_fill("unknown", "BTCUSDT", 0.5, 50000.0)
# ============================================================================step 58
$ cd /repo && sed -n '416,490p' tests/unit/live/test_order_execution.py
output · 32 lines
# Tests for _handle_order_fill
# ============================================================================
class TestHandleOrderFill:
"""Tests for the _handle_order_fill method."""
def test_handle_fill_logs_event(self, engine_with_exchange):
"""Fill callback logs the event."""
with patch("src.engines.live.trading_engine.log_order_event") as mock_log:
engine_with_exchange._handle_order_fill("order123", "BTCUSDT", 1.5, 50000.0)
20 more lines
mock_log.assert_called_once()
assert mock_log.call_args.args[0] == "order_filled"
def test_handle_fill_detects_stop_loss_fill(self, engine_with_exchange, sample_position):
"""Fill callback detects and handles stop-loss fills."""
sample_position.stop_loss_order_id = "sl_order_456"
engine_with_exchange.live_position_tracker.track_recovered_position(
sample_po
…[2220 chars omitted]…
02, 50000.0)
mock_close.assert_not_called()
def test_handle_fill_no_matching_position(self, engine_with_exchange):
"""Fill callback handles fill with no matching position."""
engine_with_exchange._handle_order_fill("unknown_order", "BTCUSDT", 1.0, 50000.0)
# ============================================================================
# Tests for _handle_partial_fillstep 59
$ cd /repo && python -m pytest tests/unit/live/test_order_execution.py -q 2>&1 | tail -60
output · 12 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[31mF[0m[32m.[0m[32m.[0m[32m.[0m[31m [100%][0m
=================================== FAILURES ===================================
[31m[1m_________ TestHandleOrderFill.test_handle_fill_detects_stop_loss_fill __________[0m
[1m[31mtests/unit/live/test_order_execution.py[0m:441: in test_handle_fill_detects_stop_loss_fill
[0mmock_close.assert_called_once()[90m[39;49;00m
[1m[31mE AssertionError: Expected '_execute_exit' to have been called once. Called 0 times.[0m
============================= slow
…[203 chars omitted]…
Use -vv to show these durations.)
[36m[1m=========================== short test summary info ============================[0m
[31mFAILED[0m tests/unit/live/test_order_execution.py::[1mTestHandleOrderFill::test_handle_fill_detects_stop_loss_fill[0m - AssertionError: Expected '_execute_exit' to have been called once. Called 0...
[31m[31m[1m1 failed[0m, [32m37 passed[0m[31m in 3.48s[0m[0mstep 60
$ cd /repo && grep -rn "_trading_loop\|main_thread" src/engines/live/trading_engine.py tests/unit/test_trading_engine_websocket.py tests/integration/live_trading | head -60
output · 16 lines
src/engines/live/trading_engine.py:542: self.main_thread = None src/engines/live/trading_engine.py:554: if threading.current_thread() is threading.main_thread(): src/engines/live/trading_engine.py:1356: self.main_thread = threading.Thread( src/engines/live/trading_engine.py:1357: target=self._run_trading_loop, args=(symbol, timeframe, max_steps) src/engines/live/trading_engine.py:1359: self.main_thread.daemon = True src/engines/live/trading_engine.py:1360: self.main_thread.start() src/engines/live/trading_engine.py:1364: while self.is_running and self.main_thread.is_alive(): src/engines/live/trading_engine.py:1373: def _run_trading_loop( src/engines/live/trading_engine.py:1378: self._trading_loop(symbol, timeframe, max_steps) src/engines/live/trading_engine.py:1703: self.main_thread src/engines/live/trad …[606 chars omitted]…
4 more lines
t_trading_engine_websocket.py:190: mock_engine.main_thread = MagicMock()
tests/unit/test_trading_engine_websocket.py:191: mock_engine.main_thread.is_alive.return_value = False
tests/integration/live_trading/test_thread_safety.py:95: if hasattr(engine, "_trading_loop"):
tests/integration/live_trading/test_thread_safety.py:96: engine._trading_loop("BTCUSDT", "1h")step 61
$ cd /repo && sed -n '1680,1725p' src/engines/live/trading_engine.py
output · 31 lines
"Engine shutdown",
None,
float(current_price),
None,
None,
None,
)
except Exception as e:
logger.error(
"Failed to close position %s: %s", position.order_id, e, exc_info=True
)
self.live_position_tracker.remove_position(position.order_id)19 more lines
else:
# PAPER: preserve open positions in DB so they survive restart.
# _recover_active_positions() will reload them on next start().
# The first candle evaluation after recovery will check SL/TP.
logger.info(
"Paper mode: preserving %s open position
…[504 chars omitted]…
g_session_id:
self.db_manager.end_trading_session(
session_id=self.trading_session_id, final_balance=self.current_balance
)
logger.info("Trading engine stopped")
def _signal_handler(self, signum: int, frame: Any) -> None:
"""Handle shutdown signals"""
logger.info("Received signal %s", signum)
self.stop()
sys.exit(0)step 62
$ cd /repo && python -m pytest tests/unit/live -q --ignore=tests/unit/live/test_order_execution.py 2>&1 | tail -60
output · 7 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [ 24%] [0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m …[3089 chars omitted]… _manager_unit.py::TestStrategyManagerThreadSafety::test_update_lock_behavior 0.65s setup tests/unit/live/test_live_engine_comprehensive_safety.py::TestErrorHandlingRecovery::test_network_timeout_recovery 0.65s setup tests/unit/live/test_live_engine_comprehensive_safety.py::TestErrorHandlingRecovery::test_max_consecutive_errors_triggers_shutdown [32m[32m[1m290 passed[0m[32m in 46.26s[0m[0m
step 63
$ cd /repo && python -m pytest tests/unit/test_health_liveness.py tests/unit/engines/live -q 2>&1 | tail -40
output · 9 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [100%][0m ============================= slowest 10 durations ============================= 3.01s call tests/unit/engines/live/test_margin_side_effect.py::test_spot_get_balance …[684 chars omitted]… itions 0.63s call tests/unit/engines/live/test_exit_conditions.py::test_check_exit_conditions_with_multiple_positions 0.59s call tests/unit/engines/live/test_exit_conditions.py::test_check_exit_conditions_passes_candle_to_execute_exit 0.59s call tests/unit/engines/live/test_session_recovery.py::test_does_not_recover_session_from_8_days_ago [32m[32m[1m71 passed[0m[32m in 29.35s[0m[0m
step 64
$ cd /repo && python - <<'PY'
step 65
$ cd /repo && git status --short
output · 3 lines
M src/config/constants.py M src/data_providers/binance_provider.py M src/engines/live/trading_engine.py
step 66
$ cd /repo && grep -n "__all__" src/config/constants.py | head
step 67
$ cd /repo && grep -rln "draining deferred stop-loss exit\|exit_on_crash\|_loop_crashed\|BINANCE_REST_TIMEOUT_SECONDS" /repo /tmp 2>/dev/null | head -20
output · 6 lines
/repo/src/data_providers/binance_provider.py /repo/src/data_providers/__pycache__/binance_provider.cpython-312.pyc /repo/src/engines/live/trading_engine.py /repo/src/engines/live/__pycache__/trading_engine.cpython-312.pyc /tmp/repro_loop.py /tmp/repro_reliability.py
step 68
$ cd /repo && python - <<'PY'
output · 1 lines
_queue.SimpleQueue[tuple[str, float]]
step 69
$ cd /repo && grep -rn "from binance.client import Client\|binance.client import\|Client(" src | grep -v __pycache__ | head -50output · 6 lines
src/data_providers/binance_provider.py:60: from binance.client import Client src/data_providers/binance_provider.py:432: client = Client( src/data_providers/binance_provider.py:440: client = Client( src/data_providers/binance_provider.py:449: client = Client(tld="us", requests_params=requests_params) src/data_providers/binance_provider.py:451: client = Client(requests_params=requests_params) src/data_providers/binance_provider.py:635: return _OfflineClient()
step 70
$ cd /repo && python -m pytest tests/unit/cli/test_live.py tests/unit/cli/test_live_health.py -q 2>&1 | tail -30
output · 8 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [100%][0m ============================= slowest 10 durations ============================= 3.10s call tests/unit/cli/test_live_health.py::TestHealthCheckHandler::test_status_endpoint_returns_200 0.02s setup tests/unit/cli/test_live_health.py::TestHandleLiveHealth::test_filters_out_port_argument 0.01s call tests/unit/cli/test_live_health.py::TestHealthCheckHandler::test_health_endpoint_returns_json (7 durations < 0.005s hidden. Use -vv to show these durations.) [32m[32m[1m35 passed[0m[32m in 6.33s[0m[0m
step 71
$ cd /repo && python -m pytest tests/integration/live_trading -q 2>&1 | tail -60
output · 11 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [100%][0m ============================= slowest 10 durations ============================= 1.12s call tests/integration/live_trading/test_database_concurrency.py::test_concurrent_open_close_with_database 0.65s call tests/integration/live_trading/test_fallbacks.py::TestLiveTradingFallbacks::test_strategy_execution_logging 0.64s call tests/integration/live_trading/test_database_logging.py::T …[546 chars omitted]… ction_logged_correctly 0.63s call tests/integration/live_trading/test_engine_core.py::test_stop_loss_from_component_strategy 0.63s call tests/integration/live_trading/test_engine_core.py::test_position_exit_with_should_exit_position 0.63s call tests/integration/live_trading/test_engine_core.py::test_stop_loss_update_with_component_strategy [32m[32m[1m46 passed[0m[32m in 12.10s[0m[0m
step 72
$ cd /repo && sed -n '250,330p' src/engines/live/runner.py && grep -n "exit_on_crash\|start(" src/engines/live/runner.py cli/commands/migration.pyoutput · 36 lines
logger.info("Continuing without sentiment analysis...")
# Set up risk parameters
risk_params = RiskParameters(
base_risk_per_trade=args.risk_per_trade,
max_risk_per_trade=args.max_risk_per_trade,
max_drawdown=args.max_drawdown,
)
# Create trading engine
logger.info("Creating live trading engine...")
engine = LiveTradingEngine(24 more lines
strategy=strategy,
data_provider=data_provider,
sentiment_provider=sentiment_provider,
risk_parameters=risk_params,
check_interval=args.check_interval,
initial_balance=args.balance,
max_position_size=args.max_position,
enable_live_trading=args.live_trading,
log_trades=args.log_trades,
alert_webhook_url=args.webhook_url,
account_snapshot
…[763 chars omitted]…
user")
except Exception as e:
logger.error(f"❌ Fatal error: {e}")
sys.exit(1)
finally:
logger.info("🏁 Trading session ended")
if __name__ == "__main__":
main()
src/engines/live/runner.py:290: engine.start(args.symbol, args.timeframe)
cli/commands/migration.py:185: engine.start(symbol=strategy.get_trading_pair(), timeframe=timeframe, max_steps=steps)step 73
$ cd /repo && grep -rn "engine.start\|runner" tests/unit tests/integration | grep -i "runner\|assert_called.*start" | head -40
output · 14 lines
grep: tests/unit/cli/__pycache__/test_live.cpython-312-pytest-9.2.0.pyc: binary file matches
grep: tests/unit/cli/__pycache__/test_live_health.cpython-312-pytest-9.2.0.pyc: binary file matches
tests/unit/cli/test_live.py:187: def test_forwards_to_live_runner(self):
tests/unit/cli/test_live.py:188: """Test that live command forwards to runner module."""
tests/unit/cli/test_live.py:201: "src.engines.live.runner", ["ml_basic", "--paper-trading"]
tests/unit/cli/test_live.py:217: mock_forward.assert_called_once_with("src.engines.live.runner", [])
tests/unit/cli/test_live_health.py:267: def test_forwards_args_to_live_runner(self):
tests/unit/cli/test_live_health.py:268: """Test that arguments are forwarded to live runner."""
tests/unit/cli/test_live_health.py:286: "src.engines.live.runner", ["ml_basic", "--paper-trading", "--symbo
…[3150 chars omitted]…
ests/unit/experiments/test_codex_fixes.py:159: strategy = runner._load_strategy("ml_basic")
tests/unit/experiments/test_codex_fixes.py:174: runner._apply_parameter_overrides(strategy, cfg)2 more lines
tests/unit/experiments/test_codex_fixes.py:176: runner._validate_post_override_invariants(strategy) tests/unit/experiments/test_codex_fixes.py:356: from src.experiments.runner import ExperimentRunner
step 74
$ cd /repo && grep -rn "DEFAULT_DATA_FETCH_TIMEOUT\|BINANCE_REST_TIMEOUT" --exclude-dir=.git --exclude='*.pyc' . | head -30
output · 8 lines
./src/config/constants.py:285:DEFAULT_DATA_FETCH_TIMEOUT = 60.0 # Timeout for historical data fetches
./src/config/constants.py:286:DEFAULT_BINANCE_REST_TIMEOUT = 15.0 # Timeout for Binance REST requests (seconds)
./src/data_providers/binance_provider.py:27: DEFAULT_BINANCE_REST_TIMEOUT,
./src/data_providers/binance_provider.py:28: DEFAULT_DATA_FETCH_TIMEOUT,
./src/data_providers/binance_provider.py:425: "BINANCE_REST_TIMEOUT_SECONDS", DEFAULT_BINANCE_REST_TIMEOUT
./src/data_providers/binance_provider.py:665: "DATA_FETCH_TIMEOUT_SECONDS", DEFAULT_DATA_FETCH_TIMEOUT
./src/data_providers/coingecko_provider.py:25:from src.config.constants import DEFAULT_DATA_FETCH_TIMEOUT
./src/data_providers/coingecko_provider.py:99: self._timeout = float(config.get("DATA_FETCH_TIMEOUT_SECONDS", DEFAULT_DATA_FETCH_TIMEOUT))step 75
$ cd /repo && git diff --check
step 76
$ cd /repo && python - <<'PY'
output · 2 lines
Configuration providers available: ['Environment Variables'] drain get_position only check passed
step 77
$ cd /repo && python - <<'PY'
output · 2 lines
Configuration providers available: ['Environment Variables'] absent attribute check passed
step 78
$ cd /repo && grep -n "class TimeoutError" -A5 src/infrastructure/timeout.py
step 79
$ cd /repo && sed -n '1,120p' src/infrastructure/timeout.py
output · 52 lines
"""Timeout utilities for protecting against blocking operations. Provides timeout decorators and context managers for operations that may block indefinitely (file I/O, model loading, network calls). """ import builtins import logging import math import signal import threading import time
40 more lines
from collections.abc import Callable
from contextlib import contextmanager
from functools import wraps
from typing import Any, TypeVar
logger = logging.getLogger(__name__)
T = TypeVar("T")
class OperationTimeoutError(builtins.TimeoutError):
"""Raised when an operation exceeds its timeout.
Inherits from the built-in TimeoutError to maintain compatibility
with code that catches the built-in exception.
"""
pass
# Backwards compatibility alias
TimeoutError = OperationTimeoutError # noqa: A001
def _timeout_handler(signum: int, _frame: Any) -> None:
"""Signal handler for
…[2285 chars omitted]…
or(f"timeout seconds must be a positive finite number, got {seconds}")
def decorator(func: Callable[..., T]) -> Callable[..., T]:
op_name = operation_name or func.__name__
@wraps(func)
def wrapper(*args: Any, **kwargs: Any) -> T:
with timeout_context(seconds, op_name):
return func(*args, **kwargs)
return wrapper
return decoratorstep 80
$ cd /repo && python - <<'PY'
output · 14 lines
DEFAULT_MFE_MAE_LOG_LEVEL = "INFO" # Timeout Configuration Defaults # These values can be overridden via environment variables: # - MODEL_LOAD_TIMEOUT_SECONDS # - INFERENCE_TIMEOUT_SECONDS # - API_REQUEST_TIMEOUT_SECONDS # - DATA_FETCH_TIMEOUT_SECONDS # - BINANCE_REST_TIMEOUT_SECONDS DEFAULT_MODEL_LOAD_TIMEOUT = 60.0 # Timeout for loading ML models (ONNX, Keras) DEFAULT_INFERENCE_TIMEOUT = 30.0 # Timeout for model inference DEFAULT_API_REQUEST_TIMEOUT = 30.0 # Timeout for external API requests
2 more lines
DEFAULT_DATA_FETCH_TIMEOUT = 60.0 # Timeout for historical data fetches DEFAULT_BINANCE_REST_TIMEOUT = 15.0 # Timeout for Binance REST requests (seconds)
step 81
$ cd /repo && python -m py_compile src/engines/live/trading_engine.py src/data_providers/binance_provider.py src/config/constants.py && python /tmp/repro_reliability.py >/tmp/repro_reliability.out 2>&1 && tail -1 /tmp/repro_reliability.out && python /tmp/repro_provider_timeout.py >/tmp/repro_provider.out 2>&1 && tail -1 /tmp/repro_provider.out && python /tmp/repro_loop.py >/tmp/repro_loop.out 2>&1 && tail -3 /tmp/repro_loop.out && git diff --check
output · 5 lines
all repro checks passed provider timeout checks passed transient ok budget ok clean ok
step 82
$ cd /repo && python -m pytest tests/unit/test_trading_engine_ws_health.py tests/unit/test_trading_engine_websocket.py tests/integration/live_trading/test_engine_core.py -q 2>&1 | tail -20
output · 11 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [100%][0m ============================= slowest 10 durations ============================= 0.62s call tests/integration/live_trading/test_engine_core.py::test_order_execution_with_component_strategy 0.62s call tests/integration/live_trading/test_engine_core.py::test_position_management_with_trading_decision 0.62s call tests/integration/live_trading/test_engine_core.py::test_stop_loss_from_component_strategy 0.62s call tests/integration/live_trading/test_engine_core.py::test_stop_loss_update_ …[413 chars omitted]… egration/live_trading/test_engine_core.py::test_position_sizing_from_trading_decision 0.03s call tests/integration/live_trading/test_engine_core.py::TestLiveTradingEngine::test_position_opening_paper_trading 0.03s call tests/integration/live_trading/test_engine_core.py::TestLiveTradingEngine::test_engine_initialization_with_live_trading_enabled [32m[32m[1m34 passed[0m[32m in 6.20s[0m[0m
step 83
$ cd /repo && git diff --stat && git status --short && git diff -- src/config/constants.py src/data_providers/binance_provider.py
output · 30 lines
src/config/constants.py | 2 + src/data_providers/binance_provider.py | 25 +++- src/engines/live/trading_engine.py | 258 +++++++++++++++++++++++---------- 3 files changed, 203 insertions(+), 82 deletions(-) M src/config/constants.py M src/data_providers/binance_provider.py M src/engines/live/trading_engine.py diff --git a/src/config/constants.py b/src/config/constants.py index 342b5d0..0626b92 100644 --- a/src/config/constants.py +++ b/src/config/constants.py @@ -279,10 +279,12 @@ DEFAULT_MFE_MAE_LOG_LEVEL = "INFO"
18 more lines
# - INFERENCE_TIMEOUT_SECONDS
# - API_REQUEST_TIMEOUT_SECONDS
# - DATA_FETCH_TIMEOUT_SECONDS
+# - BINANCE_REST_TIMEOUT_SECONDS
DEFAULT_MODEL_LOAD_TIMEOUT = 60.0 # Timeout for loading ML models (ONNX, Keras)
DEFAULT_INFERENCE_TIMEOUT = 30.0 # Timeout for model inference
DEFAULT_API_REQUEST_TIMEOUT = 30.0 # Timeout for external API request
…[1967 chars omitted]…
)
if api_endpoint == "binanceus":
- client = Client(tld="us")
+ client = Client(tld="us", requests_params=requests_params)
else:
- client = Client()
+ client = Client(requests_params=requests_params)
auth_mode = "with credentials" if self.api_key and self.api_secret else "public mode"
logger.info(step 84
$ cd /repo && grep -rn "_handle_order_fill(" --exclude='*.pyc' src tests | head -30output · 6 lines
src/engines/live/trading_engine.py:3192: def _handle_order_fill(
tests/unit/live/test_order_execution.py:426: engine_with_exchange._handle_order_fill("order123", "BTCUSDT", 1.5, 50000.0)
tests/unit/live/test_order_execution.py:439: engine_with_exchange._handle_order_fill("sl_order_456", "BTCUSDT", 0.02, 48000.0)
tests/unit/live/test_order_execution.py:468: engine_with_exchange._handle_order_fill("sl_order_456", "BTCUSDT", 0.02, 48000.0)
tests/unit/live/test_order_execution.py:480: engine_with_exchange._handle_order_fill("entry_order_123", "BTCUSDT", 0.02, 50000.0)
tests/unit/live/test_order_execution.py:486: engine_with_exchange._handle_order_fill("unknown_order", "BTCUSDT", 1.0, 50000.0)step 85
$ done: {"notes": "Implemented the reliability fixes:\n- Added `_loop_crashed`, `_run_trading_loop`, `_exit_if_loop_crashed`, and `start(..., exit_on_crash=False)`; consecutive-error exhaustion marks the crash flag while clean max_steps stops do not.\n- Added `DEFAULT_BINANCE_REST_TIMEOUT = 15.0` and passed `requests_params={\"timeout\": .output · 3 lines
…[537 chars omitted]… test_handle_fill_detects_stop_loss_fill` still asserts the old immediate `_execute_exit` behavior that this issue explicitly replaces; all other affected tests pass."} submitted.
GPT-6 Luna · max · attempt 1 · solved · 66 steps · $0.161
Submitted diff · 447 lines
diff --git a/src/config/constants.py b/src/config/constants.py
index 342b5d0..4483275 100644
--- a/src/config/constants.py
+++ b/src/config/constants.py
@@ -279,10 +279,12 @@ DEFAULT_MFE_MAE_LOG_LEVEL = "INFO"
# - INFERENCE_TIMEOUT_SECONDS
# - API_REQUEST_TIMEOUT_SECONDS
# - DATA_FETCH_TIMEOUT_SECONDS
+# - BINANCE_REST_TIMEOUT_SECONDS
DEFAULT_MODEL_LOAD_TIMEOUT = 60.0 # Timeout for loading ML models (ONNX, Keras)
DEFAULT_INFERENCE_TIMEOUT = 30.0 # Timeout for model inference
DEFAULT_API_REQUEST_TIMEOUT = 30.0 # Timeout for external API requests
DEFAULT_DATA_FETCH_TIMEOUT = 60.0 # Timeout for historical data fetches
+DEFAULT_BINANCE_REST_TIMEOUT = 15.0 # Timeout for Binance REST API requests
# Numeric Precision Constants
DEFAULT_EPSILON = 1e-9 # Small value for floating point comparisons
diff --git a/src/data_providers/binance_provider.py b/src/data_providers/binance_provider.py
index 7182e30..d7bc377 100644
--- a/src/data_providers/binance_provider.py
+++ b/src/data_providers/binance_provider.py
@@ -24,6 +24,7 @@ import pandas as pd
from src.config import get_config
from src.config.constants import (
+ DEFAULT_BINANCE_REST_TIMEOUT,
DEFAULT_DATA_FETCH_TIMEOUT,
DEFAULT_STARTUP_BAN_MAX_RETRIES,
DEFAULT_STARTUP_BAN_MAX_WAIT,
@@ -420,18 +421,46 @@ class BinanceProvider(DataProvider, ExchangeInterface):
self.testnet,
)
+ try:
+ rest_timeout = float(
+ get_config().get_float(
+ "BINANCE_REST_TIMEOUT_SECONDS", DEFAULT_BINANCE_REST_TIMEOUT
+ )
+ )
+ except (TypeError, ValueError, OverflowError):
+ rest_timeout = DEFAULT_BINANCE_REST_TIMEOUT
+ if not math.isfinite(rest_timeout) or rest_timeout <= 0:
+ logger.warning(
+ "Invalid Binance REST timeout %r; using %.1fs",
+ rest_timeout,
+ DEFAULT_BINANCE_REST_TIMEOUT,
+ )
+ rest_timeout = DEFAULT_BINANCE_REST_TIMEOUT
+ requests_params = {"timeout": rest_timeout}
+
if self.api_key and self.api_secret:
logger.debug("Creating authenticated %s client...", api_endpoint)
if api_endpoint == "binanceus":
- client = Client(self.api_key, self.api_secret, testnet=self.testnet, tld="us")
+ client = Client(
+ self.api_key,
+ self.api_secret,
+ testnet=self.testnet,
+ tld="us",
+ requests_params=requests_params,
+ )
else:
- client = Client(self.api_key, self.api_secret, testnet=self.testnet)
+ client = Client(
+ self.api_key,
+ self.api_secret,
+ testnet=self.testnet,
+ requests_params=requests_params,
+ )
else:
logger.debug("Creating public %s client...", api_endpoint)
if api_endpoint == "binanceus":
- client = Client(tld="us")
+ client = Client(tld="us", requests_params=requests_params)
else:
- client = Client()
+ client = Client(requests_params=requests_params)
auth_mode = "with credentials" if self.api_key and self.api_secret else "public mode"
logger.info(
diff --git a/src/engines/live/trading_engine.py b/src/engines/live/trading_engine.py
index 8aecd9c..8374c81 100644
--- a/src/engines/live/trading_engine.py
+++ b/src/engines/live/trading_engine.py
@@ -4,6 +4,7 @@ import json
import logging
import math
import os
+import queue
import signal
import sys
import threading
@@ -470,6 +471,11 @@ class LiveTradingEngine:
self._ws_kline_active = False
self._ws_kline_provider = None
self._ws_health_thread = None
+ self._ws_health_monitor_lock = threading.Lock()
+
+ # Filled stop-loss callbacks may arrive on the order-polling thread.
+ # Defer position updates to the trading loop to avoid racing it.
+ self._pending_fill_exits: queue.SimpleQueue[tuple[str, float]] = queue.SimpleQueue()
# Performance tracker (unified with backtest engine)
from src.performance.tracker import PerformanceTracker
@@ -479,6 +485,7 @@ class LiveTradingEngine:
# Error handling
self.max_consecutive_errors = max_consecutive_errors
self.consecutive_errors = 0
+ self._loop_crashed = False
# Monotonic timestamp marking when the database first became unreachable
# inside the trading loop (None while reachable). Lets the engine ride
# out transient DB outages instead of counting them toward
@@ -1156,12 +1163,24 @@ class LiveTradingEngine:
self._runtime_dataset = None
self._runtime_warmup = 0
- def start(self, symbol: str, timeframe: str = "1h", max_steps: int | None = None) -> None:
- """Start the live trading engine"""
+ def start(
+ self,
+ symbol: str,
+ timeframe: str = "1h",
+ max_steps: int | None = None,
+ *,
+ exit_on_crash: bool = False,
+ ) -> None:
+ """Start the live trading engine.
+
+ When ``exit_on_crash`` is enabled, an unexpected trading-loop failure
+ is surfaced to the caller as a non-zero process exit.
+ """
if self.is_running:
logger.warning("Trading engine is already running")
return
+ self._loop_crashed = False
self.is_running = True
self._active_symbol = symbol
self.timeframe = timeframe # Store the trading timeframe
@@ -1338,7 +1357,7 @@ class LiveTradingEngine:
# Start main trading loop in separate thread
self.main_thread = threading.Thread(
- target=self._trading_loop, args=(symbol, timeframe, max_steps)
+ target=self._run_trading_l
… [14310 more characters]step 1
$ pwd && ls -la && find . -maxdepth 2 -type f | sort | sed -n '1,200p'
output · 41 lines
/repo total 396 drwxr-xr-x 1 root root 4096 Nov 2 14:37 . drwxr-xr-x 1 root root 4096 Nov 2 14:36 .. drwxr-xr-x 1 root root 4096 Nov 2 14:35 .agents drwxr-xr-x 1 root root 4096 Nov 2 14:36 .claude -rw-r--r-- 1 root root 230 Nov 2 14:35 .coveragerc drwxr-xr-x 1 root root 4096 Nov 2 14:35 .cursor -rw-r--r-- 1 root root 1450 Nov 2 14:35 .dockerignore -rw-r--r-- 1 root root 1014 Nov 2 14:35 .env.example drwxr-xr-x 1 root root 4096 Nov 2 14:36 .git drwxr-xr-x 1 root root 4096 Nov 2 14:36 .github
29 more lines
-rw-r--r-- 1 root root 692 Nov 2 14:36 .gitignore drwxr-xr-x 1 root root 4096 Nov 2 14:35 .specify -rw-r--r-- 1 root root 29389 Nov 2 14:35 AGENTS.md -rw-r--r-- 1 root root 9339 Nov 2 14:36 CLAUDE.md -rw-r--r-- 1 root root 19924 Nov 2 14:36 CODE.md -rw-r--r-- 1 root root 1399 Nov 2 14:36 Dockerfile -rw-r--r-- 1 root root 1056 Nov 2 14:35 LICENSE -r …[4399 chars omitted]… .py ./migrations/script.py.mako ./mypy.ini ./package-lock.json ./pyproject.toml ./pytest.ini ./railway.json ./requirements-github.txt ./requirements-server.txt ./requirements.txt ./scripts/chaos_smoke_test.py ./scripts/check_heartbeat.py ./scripts/phase1_benchmark.py ./scripts/validate_models.py ./setup.cfg ./sitecustomize.py ./src/__init__.py ./tests/README.md ./tests/conftest.py ./tests/run_tests.py
step 2
$ rg -n "class LiveTradingEngine|_trading_loop|_handle_order_fill|_execute_exit|ws_health|BINANCE_REST|DEFAULT_.*TIMEOUT|class.*Provider" src tests | head -300
output · 1 lines
bash: line 8: rg: command not found
step 3
$ grep -RInE "class LiveTradingEngine|_trading_loop|_handle_order_fill|_execute_exit|ws_health|BINANCE_REST|DEFAULT_.*TIMEOUT|class .*Provider" src tests | head -300
output · 16 lines
src/config/constants.py:173:DEFAULT_THREAD_JOIN_TIMEOUT = 30 # Timeout for main thread join in seconds src/config/constants.py:174:DEFAULT_ORDER_TRACKER_TIMEOUT = 10 # Timeout for order tracker thread join src/config/constants.py:175:DEFAULT_REQUEST_TIMEOUT = 10 # Timeout for HTTP requests (webhooks, etc.) src/config/constants.py:282:DEFAULT_MODEL_LOAD_TIMEOUT = 60.0 # Timeout for loading ML models (ONNX, Keras) src/config/constants.py:283:DEFAULT_INFERENCE_TIMEOUT = 30.0 # Timeout for model inference src/config/constants.py:284:DEFAULT_API_REQUEST_TIMEOUT = 30.0 # Timeout for external API requests src/config/constants.py:285:DEFAULT_DATA_FETCH_TIMEOUT = 60.0 # Timeout for historical data fetches src/config/providers/base.py:11:class ConfigProvider(ABC): src/config/providers/dotenv_provider.py:13:class DotEnvProvider(ConfigProvider): src/config/providers/env_provider.py:13:clas …[8630 chars omitted]… sconnect when ws_healthy is False."""
4 more lines
tests/unit/test_trading_engine_ws_health.py:63: mock_provider.ws_healthy = False tests/unit/test_trading_engine_ws_health.py:73: """Should not trigger disconnect when ws_healthy is True.""" tests/unit/test_trading_engine_ws_health.py:75: mock_provider.ws_healthy = True tests/unit/trading/test_shared_sentiment.py:8:class DummySentimentProvider:
step 4
$ find src/engines/live -maxdepth 2 -type f -print && sed -n '1,230p' src/engines/live/trading_engine.py && sed -n '350,520p' src/engines/live/trading_engine.py && sed -n '1280,1740p' src/engines/live/trading_engine.py
output · 34 lines
src/engines/live/account_sync.py src/engines/live/data/market_data_handler.py src/engines/live/data/__init__.py src/engines/live/event_deduplicator.py src/engines/live/execution/entry_handler.py src/engines/live/execution/execution_engine.py src/engines/live/execution/exit_handler.py src/engines/live/execution/position_tracker.py src/engines/live/execution/__init__.py src/engines/live/health/health_monitor.py src/engines/live/health/__init__.py src/engines/live/kline_buffer.py
22 more lines
src/engines/live/logging/event_logger.py
src/engines/live/logging/__init__.py
src/engines/live/margin_interest_tracker.py
src/engines/live/order_tracker.py
src/engines/live/pnl.py
src/engines/live/README.md
src/engines/live/reconciliation.py
src/engines/live/regime_strategy_switcher.py
src/engines/live/runner.py
src/engines/live/strategy_manager.py
src/engines/live/trading_engine.py
src/engines/live/user_data_pr
…[8630 chars omitted]…
, "volume"]
df = df.dropna(subset=essential_columns)
except Exception:
# Fallback to conservative behavior if subset fails for any reason
df = df.dropna()
# Context readiness gating
ready, reason = self._is_context_ready(df)
safety_mode = not ready
if safety_mode:step 5
$ grep -nE "^class LiveTradingEngine|^ def __init__|^ def start\(|^ def stop\(|^ def _trading_loop|^ def _execute_exit|^ def _handle_order_fill|^ def _start_ws_health_monitor|^ def _ws_health_loop|_ws_health_thread|consecutive_errors|db_unreachable_since|def _handle_kline_disconnect|resync|reconnect|Client\(" src/engines/live/trading_engine.py src/data_providers/binance_provider.py src/config/constants.pyoutput · 15 lines
src/engines/live/trading_engine.py:144:class LiveTradingEngine:
src/engines/live/trading_engine.py:159: def __init__(
src/engines/live/trading_engine.py:174: max_consecutive_errors: int = 10, # Maximum consecutive errors before shutdown
src/engines/live/trading_engine.py:472: self._ws_health_thread = None
src/engines/live/trading_engine.py:480: self.max_consecutive_errors = max_consecutive_errors
src/engines/live/trading_engine.py:481: self.consecutive_errors = 0
src/engines/live/trading_engine.py:485: # max_consecutive_errors (incident 2026-05-19: a Railway internal-DNS
src/engines/live/trading_engine.py:487: self.db_unreachable_since: float | None = None
src/engines/live/trading_engine.py:618: max_consecutive_errors=self.max_consecutive_errors,
src/engines/live/trading_engine.py:1159: def start(self, symbol: str, timeframe:
…[7298 chars omitted]…
True if reconnect succeeded, False otherwise.3 more lines
src/data_providers/binance_provider.py:2088: "User stream reconnect attempt %d/%d failed: %s",
src/data_providers/binance_provider.py:2095: logger.info("Retrying user stream reconnect in %ds...", backoff)
src/config/constants.py:350:DEFAULT_WS_RECONNECT_MAX_RETRIES = 3 # Maximum reconnect attempts before REST_DEGRADEDstep 6
$ sed -n '135,220p' src/engines/live/trading_engine.py; sed -n '430,510p' src/engines/live/trading_engine.py; sed -n '1120,1585p' src/engines/live/trading_engine.py; sed -n '1660,2055p' src/engines/live/trading_engine.py; sed -n '3070,3210p' src/engines/live/trading_engine.py; sed -n '3290,3415p' src/engines/live/trading_engine.py
output · 39 lines
api_secret = config.get("BINANCE_API_SECRET")
provider_name = "Binance"
if api_key and api_secret:
return BinanceProvider(api_key, api_secret, testnet=testnet), provider_name
else:
return None, f"{provider_name} (no credentials)"
class LiveTradingEngine:
"""
Advanced live trading engine that executes strategies in real-time.27 more lines
Features:
- Real-time data streaming
- Actual order execution
- Position management
- Risk management
- Sentiment integration
- Error handling & recovery
- Performance monitoring
- Graceful shutdown
"""
def __init__(
self,
strategy: ComponentStrategy | StrategyRuntime,
data_provider: DataProvider,
sentiment_provider: SentimentDataProvider | None = None,
risk_parameters: RiskParameters | None = None,
…[8630 chars omitted]…
)
# Convert from base asset units to USDT using exit price
interest_cost = interest_base * float(exit_result.exit_price)
if interest_cost > 0:
realized_pnl -= interest_cost
logger.info(
"Deducted margin interest $%.4f (%.8f %s @ %.2f) from PnL for %s",step 7
$ sed -n '280,510p' src/engines/live/trading_engine.py; sed -n '1120,1435p' src/engines/live/trading_engine.py
output · 29 lines
self._component_risk_context_cache: dict[str, Any] | None = None
# Timing configuration
self.base_check_interval = check_interval
self.check_interval = check_interval
self.min_check_interval = DEFAULT_MIN_CHECK_INTERVAL
self.max_check_interval = DEFAULT_MAX_CHECK_INTERVAL
self.data_freshness_threshold = DEFAULT_DATA_FRESHNESS_THRESHOLD
self.last_data_timestamp = None
self.initial_balance = initial_balance
self.current_balance = initial_balance # Will be updated during startup
self._balance_lock = threading.Lock() # Protect concurrent balance modifications17 more lines
self.max_position_size = max_position_size
self.enable_live_trading = enable_live_trading
# Execution realism (parity with backtest)
self.fee_rate = fee_rate
self.slippage_rate = slippage_rate
sel
…[8630 chars omitted]…
ocessor:
self._start_ws_health_monitor()
def _start_ws_health_monitor(self) -> None:
"""Start daemon thread to monitor WebSocket stream health."""
self._ws_health_thread = threading.Thread(
target=self._ws_health_loop, daemon=True, name="WSHealthMonitor"
)
self._ws_health_thread.start()
logger.info("WebSocket health monitor started")step 8
$ sed -n '440,505p' src/engines/live/trading_engine.py; sed -n '1115,1285p' src/engines/live/trading_engine.py; sed -n '1285,1438p' src/engines/live/trading_engine.py; sed -n '1435,1580p' src/engines/live/trading_engine.py
output · 25 lines
strategy.strategy if isinstance(strategy, StrategyRuntime) else strategy
)
# Support component-based Strategy
if isinstance(managed_strategy, ComponentStrategy):
self.strategy_manager = StrategyManager()
self.strategy_manager.current_strategy = managed_strategy
self.strategy_manager.on_strategy_change = self._handle_strategy_change
self.strategy_manager.on_model_update = self._handle_model_update
logger.info("Hot swapping enabled for %s", managed_strategy.__class__.__name__)
else:
logger.info("Hot swapping disabled: provided strategy does not implement Strategy")
13 more lines
# Set up strategy logging if database is available
if self.db_manager:
if hasattr(self.strategy, "set_database_manager"):
…[8630 chars omitted]…
self.is_running = False
self.stop_event.set()
# Stop inbound WS streams FIRST (no new events arrive)
if self._ws_kline_provider and hasattr(self._ws_kline_provider, "stop_streams"):
self._ws_kline_provider.stop_streams()
if self.exchange_interface and hasattr(self.exchange_interface, "stop_streams"):
self.exchange_interface.stop_streams()step 9
$ sed -n '1,95p' src/data_providers/binance_provider.py; sed -n '235,460p' src/data_providers/binance_provider.py; sed -n '630,665p' src/data_providers/binance_provider.py; sed -n '1930,2110p' src/data_providers/binance_provider.py; sed -n '150,195p' src/config/constants.py; sed -n '265,295p' src/config/constants.py; sed -n '330,365p' src/config/constants.py
output · 42 lines
""" Unified Binance Provider This module combines both data provider and exchange functionality for Binance, providing a single interface for all Binance operations including: - Historical and live data fetching - Order execution and management - Account synchronization - Position management """ import logging
30 more lines
import math
import re
import threading
import time
from collections.abc import Callable
from datetime import UTC, datetime, timedelta
from enum import Enum
from functools import wraps
from typing import Any, TypeVar
import pandas as pd
from src.config import get_config
from src.config.constants import (
DEFAULT_DATA_FETCH_TIMEOUT,
DEFAULT_STARTUP_BAN_MAX_RETRIES,
DEFAULT_STARTUP_BAN_MAX_WAIT,
DEFAULT_WS_KLINE_STALENESS_THRESHOLD,
DEFAULT_WS_RECONNECT_MAX_RETRIES,
DEFAULT_WS_USER_STALENESS_THRESHOLD,
)
from src.infrastructure.timeout import TimeoutErro
…[8630 chars omitted]…
RIES = 3 # Max retry attempts for ban-related startup failures
# TFT (Temporal Fusion Transformer) Model Defaults
DEFAULT_TFT_N_HEADS = 4 # Number of attention heads in the temporal decoder
DEFAULT_TFT_HIDDEN_SIZE = 64 # Hidden dimension for GRN and attention layers
DEFAULT_TFT_DROPOUT = 0.1 # Dropout rate for regularization
DEFAULT_TFT_NUM_LSTM_LAYERS = 1 # Number of stacked LSTM encoder layersstep 10
$ sed -n '1140,1280p' src/engines/live/trading_engine.py && sed -n '1425,1575p' src/engines/live/trading_engine.py && sed -n '1640,1685p' src/engines/live/trading_engine.py && sed -n '1930,2045p' src/engines/live/trading_engine.py && sed -n '3080,3205p' src/engines/live/trading_engine.py && grep -n "def get_position\|def has_position\|class LivePositionTracker\|def positions" src/engines/live/execution/position_tracker.py
output · 31 lines
return None
context = self._build_runtime_context(balance, current_price, current_time)
try:
decision = self._runtime.process(index, context)
self._apply_policies_from_decision(decision)
return decision
except (ValueError, KeyError, IndexError, AttributeError) as exc:
logger.warning("Runtime decision failed in live engine at index %s: %s", index, exc)
return None
def _finalize_runtime(self) -> None:19 more lines
if self._is_runtime_strategy():
try:
self._runtime.finalize()
finally:
self._runtime_dataset = None
self._runtime_warmup = 0
def start(self, symbol: str, timeframe: str = "1h", max_steps: int | None = None) -> None:
"""Start the live trading engine"""
if self.is_running:
logger.
…[8630 chars omitted]…
f @ $%.2f). Remaining SL order is still active on exchange. "
70:class LivePositionTracker:
124: def positions(self) -> dict[str, LivePosition]:
144: def has_position(self, order_id: str) -> bool:
149: def has_position_for_symbol(self, symbol: str) -> bool:
159: def get_position(self, order_id: str) -> LivePosition | None:
814: def get_position_state(self, order_id: str) -> dict | None:step 11
$ find tests -type f | sort | grep -E 'live|trading_engine|binance_provider|db_resilience|order|websocket'
output · 28 lines
tests/integration/live/__init__.py tests/integration/live/test_correlation_enforcement.py tests/integration/live/test_dynamic_risk_integration_live_engine.py tests/integration/live/test_mfe_mae_live_integration.py tests/integration/live/test_mfe_mae_persistence_throttle.py tests/integration/live/test_partial_operations_live_engine.py tests/integration/live/test_reconciliation_integration.py tests/integration/live/test_regime_integration_live_engine.py tests/integration/live/test_strategy_manager_integration.py tests/integration/live/test_time_exits_live_engine_integration.py tests/integration/live_trading/test_data_validation.py tests/integration/live_trading/test_database_concurrency.py
16 more lines
tests/integration/live_trading/test_database_error_paths.py tests/integration/live_trading/test_database_logging.py tests/integration/live_trading/test_engine_core.py tests/integration/live_trading/te …[1650 chars omitted]… y tests/unit/live/test_user_data_processor.py tests/unit/test_backtest_live_parity.py tests/unit/test_binance_provider_interest.py tests/unit/test_binance_provider_websocket.py tests/unit/test_db_resilience.py tests/unit/test_health_liveness.py tests/unit/test_order_models.py tests/unit/test_order_status_methods.py tests/unit/test_trading_engine_websocket.py tests/unit/test_trading_engine_ws_health.py
step 12
$ sed -n '1240,1438p' src/engines/live/trading_engine.py; sed -n '1438,1575p' src/engines/live/trading_engine.py; sed -n '1640,1695p' src/engines/live/trading_engine.py; sed -n '1930,2045p' src/engines/live/trading_engine.py
output · 26 lines
)
# Set session ID on strategy for logging
if hasattr(self.strategy, "session_id"):
self.strategy.session_id = self.trading_session_id
# Wire session_id and strategy_name to execution engine for order journaling
self.live_execution_engine.session_id = self.trading_session_id
self.live_execution_engine.strategy_name = self._strategy_name()
# Perform account synchronization if available
self._pending_balance_correction = False14 more lines
self._pending_corrected_balance = None
if self.account_synchronizer and self.enable_live_trading:
try:
logger.info("🔄 Performing initial account synchronization...")
sync_result = self.account_synchronizer.sync_account_data(
force=True, symbol=self._active_symbol
)
…[8630 chars omitted]…
vailability, dropped connections, or DNS hiccups
should be ridden out with a backoff rather than counted toward
``max_consecutive_errors``. Otherwise a short infrastructure blip
shuts the live engine down — which is exactly what happened on
2026-05-19 when Railway's internal DNS failed to resolve
``postgres.railway.internal`` for several hours. ``pool_pre_ping``step 13
$ sed -n '1330,1438p' src/engines/live/trading_engine.py; sed -n '1438,1570p' src/engines/live/trading_engine.py; sed -n '1645,1685p' src/engines/live/trading_engine.py; sed -n '1938,2035p' src/engines/live/trading_engine.py; sed -n '200,245p' src/config/constants.py; sed -n '400,445p' src/data_providers/binance_provider.py; sed -n '1,240p' tests/unit/test_trading_engine_ws_health.py; sed -n '1,250p' tests/unit/test_db_resilience.py
output · 37 lines
)
self._periodic_reconciler.start()
logger.info("🔄 Periodic reconciler started")
except Exception as e:
logger.warning("Failed to start periodic reconciler: %s", e)
# Try to start WebSocket streams for reduced API weight
self._start_websocket_streams(symbol, timeframe)
# Start main trading loop in separate thread
self.main_thread = threading.Thread(
target=self._trading_loop, args=(symbol, timeframe, max_steps)25 more lines
)
self.main_thread.daemon = True
self.main_thread.start()
try:
# Keep main thread alive
while self.is_running and self.main_thread.is_alive():
time.sleep(1)
except KeyboardInterrupt:
logger.info("Received interrupt signal")
finally:
self.stop()
…[8630 chars omitted]…
consecutive_errors = 100
engine.is_running = True
engine._sleep_with_interrupt = Mock()
engine._get_latest_data = Mock(side_effect=_op_error("connection refused"))
# Pretend the outage began well beyond the close-only threshold.
engine.db_unreachable_since = time.monotonic() - 10_000
engine._trading_loop("BTCUSDT", "1h", max_steps=1)
assert engine._close_only_mode is Truestep 14
$ sed -n '220,455p' src/data_providers/binance_provider.py; grep -RIn "BINANCE_REST_TIMEOUT\|get_float(.*TIMEOUT" src tests | head -100; sed -n '260,360p' src/config/constants.py; grep -n "def get_float" -R src/config
output · 27 lines
class BinanceProvider(DataProvider, ExchangeInterface):
"""
Unified Binance provider that combines data fetching and exchange operations.
Inherits from both DataProvider and ExchangeInterface to provide complete
Binance functionality in a single class.
"""
@property
def is_margin_mode(self) -> bool:
"""Whether provider is operating in cross-margin mode."""
return self._use_margin15 more lines
TIMEFRAME_MAPPING = {
"1m": Client.KLINE_INTERVAL_1MINUTE if BINANCE_AVAILABLE else "1m",
"5m": Client.KLINE_INTERVAL_5MINUTE if BINANCE_AVAILABLE else "5m",
"15m": Client.KLINE_INTERVAL_15MINUTE if BINANCE_AVAILABLE else "15m",
"1h": Client.KLINE_INTERVAL_1HOUR if BINANCE_AVAILABLE else "1h",
"4h": Client.KLINE_INTERVAL_4HOUR if BINANCE_AVAILABLE else "4h",
"1d": Client.KLINE_INTERVAL_1DAY if BINANCE_AVAILABL
…[8629 chars omitted]…
) Model Defaults
DEFAULT_TFT_N_HEADS = 4 # Number of attention heads in the temporal decoder
DEFAULT_TFT_HIDDEN_SIZE = 64 # Hidden dimension for GRN and attention layers
DEFAULT_TFT_DROPOUT = 0.1 # Dropout rate for regularization
DEFAULT_TFT_NUM_LSTM_LAYERS = 1 # Number of stacked LSTM encoder layers
src/config/config_manager.py:104: def get_float(self, key: str, default: float = 0.0) -> float:step 15
$ sed -n '70,180p' src/engines/live/execution/position_tracker.py; sed -n '1470,1585p' src/engines/live/trading_engine.py; sed -n '2025,2098p' src/engines/live/trading_engine.py; sed -n '3110,3185p' src/engines/live/trading_engine.py; sed -n '380,455p' src/data_providers/binance_provider.py
output · 29 lines
class LivePositionTracker:
"""Tracks active position state, partial operations, and MFE/MAE metrics.
This class manages the lifecycle of multiple active positions including:
- Position state (entry, current size, trailing stop status)
- Partial exit and scale-in tracking
- Maximum Favorable/Adverse Excursion (MFE/MAE) metrics
- Unrealized P&L updates
Thread Safety:
All position state is protected by _positions_lock to prevent race conditions.
- Callbacks from OrderTracker may run concurrently with the main trading loop17 more lines
- All mutations to _positions and _position_db_ids must occur inside the lock
- When reading position data for calculations, keep operations inside the lock
- Lock acquisition order: _positions_lock only (no nested locks to avoid deadlocks)
- Properties return copies to prevent external concur
…[8629 chars omitted]…
ng client with server time request...")
test_response = client.get_server_time()
logger.debug("Server time test successful: %s", test_response)
# Only promote to self._client after all verification passes
self._client = client
if self._use_margin:
self._verify_margin_account()
@staticmethod
def _handle_startup_ban(
error: Exception,step 16
$ grep -n "stop_event\|main_thread" src/engines/live/trading_engine.py | head -50; sed -n '430,490p' src/engines/live/trading_engine.py; sed -n '1430,1488p' src/engines/live/trading_engine.py; sed -n '1530,1655p' src/engines/live/trading_engine.py; sed -n '1655,1715p' src/engines/live/trading_engine.py; sed -n '1940,2035p' src/engines/live/trading_engine.py
output · 28 lines
534: self.main_thread = None 535: self.stop_event = threading.Event() 546: if threading.current_thread() is threading.main_thread(): 1340: self.main_thread = threading.Thread( 1343: self.main_thread.daemon = True 1344: self.main_thread.start() 1348: while self.is_running and self.main_thread.is_alive(): 1442: self.stop_event.wait(DEFAULT_WS_HEALTH_CHECK_INTERVAL) 1444: while self.is_running and not self.stop_event.is_set(): 1450: self.stop_event.wait(DEFAULT_WS_HEALTH_CHECK_INTERVAL) 1574: self.stop_event.set() 1637: self.main_thread
16 more lines
1638: and self.main_thread.is_alive()
1639: and self.main_thread != threading.current_thread()
1641: self.main_thread.join(timeout=30)
1672: while self.is_running and not self.stop_event.is_set():
3831: if self
…[8630 chars omitted]…
Exponential backoff with adaptive intervals
sleep_time = min(self.error_cooldown, self.check_interval * self.consecutive_errors)
self._sleep_with_interrupt(sleep_time)
continue
# Sleep with current interval
self._sleep_with_interrupt(self.check_interval)
logger.info("Trading loop ended")
self._finalize_runtime()step 17
$ sed -n '1570,1665p' src/engines/live/trading_engine.py; sed -n '1648,1682p' src/engines/live/trading_engine.py; sed -n '1880,2045p' src/engines/live/trading_engine.py
output · 30 lines
return
logger.info("🛑 Stopping trading engine...")
self.is_running = False
self.stop_event.set()
# Stop inbound WS streams FIRST (no new events arrive)
if self._ws_kline_provider and hasattr(self._ws_kline_provider, "stop_streams"):
self._ws_kline_provider.stop_streams()
if self.exchange_interface and hasattr(self.exchange_interface, "stop_streams"):
self.exchange_interface.stop_streams()
# Stop/drain UserDataProcessor (process remaining queued events)18 more lines
if self._user_data_processor:
self._user_data_processor.stop()
# Stop periodic reconciler
if self._periodic_reconciler:
self._periodic_reconciler.stop()
logger.info("🔄 Periodic reconciler stopped")
# Stop order tracker
if self.order_tracker:
self.order_tra
…[8629 chars omitted]…
vailability, dropped connections, or DNS hiccups
should be ridden out with a backoff rather than counted toward
``max_consecutive_errors``. Otherwise a short infrastructure blip
shuts the live engine down — which is exactly what happened on
2026-05-19 when Railway's internal DNS failed to resolve
``postgres.railway.internal`` for several hours. ``pool_pre_ping``step 18
$ sed -n '1,150p' src/config/config_manager.py; sed -n '300,370p' src/config/constants.py; grep -RIn "_attempt_client_init\|reconnect_kline\|reconnect_user\|mark_kline_degraded\|mark_user_degraded" tests/unit/test_binance_provider_websocket.py tests/unit/data_providers/test_binance_provider.py tests/unit/test_binance_provider_interest.py
output · 41 lines
""" Configuration management system. This module provides centralized configuration management. """ from __future__ import annotations import threading from typing import TYPE_CHECKING, Any if TYPE_CHECKING:
29 more lines
from threading import Lock
from .providers.base import ConfigProvider
from .providers.dotenv_provider import DotEnvProvider
from .providers.env_provider import EnvVarProvider
from .providers.railway_provider import RailwayProvider
class ConfigManager:
"""
Manages configuration from multiple sources with fallback support.
Default priority order:
1. Railway environment variables (if available)
2. Environment variables
3. .env file
"""
def __init__(self, providers: list[ConfigProvider] | None = None):
"""
Initialize ConfigManager with providers.
Args:
providers: List of configuration providers in priori
…[8539 chars omitted]…
r_websocket.py:417: result = provider.reconnect_user()
tests/unit/test_binance_provider_websocket.py:423: def test_reconnect_user_returns_false_without_callback(self, provider):
tests/unit/test_binance_provider_websocket.py:424: """reconnect_user() returns False when no callback stored."""
tests/unit/test_binance_provider_websocket.py:426: result = provider.reconnect_user()step 19
$ sed -n '1910,2105p' src/data_providers/binance_provider.py; grep -RIn "_handle_user_stream_disconnect\|_handle_kline_disconnect\|drain_pending_fill\|handle_order_fill\|stop_loss.*filled\|loop_crashed\|exit_on_crash" tests
output · 24 lines
"""Route user data events, handling errors before user callback."""
if msg.get("e") == "error":
logger.error("User data WS error: %s", msg.get("m", "unknown"))
self._on_user_disconnect()
return
self._last_user_event_time = datetime.now(UTC)
self._user_event_received = True
on_user_event(msg)
if self._use_margin:
self._user_socket_key = self._twm.start_margin_socket(callback=_user_callback)
else:12 more lines
self._user_socket_key = self._twm.start_user_socket(callback=_user_callback)
self._last_user_event_time = datetime.now(UTC)
self._user_ws_state = WebSocketState.PRIMARY
return True
except Exception as e:
logger.error("Failed to start user data s
…[8629 chars omitted]…
ect()
tests/unit/test_trading_engine_ws_health.py:165: mock_engine._handle_kline_disconnect()
tests/unit/test_trading_engine_ws_health.py:172: """Tests for _handle_user_stream_disconnect()."""
tests/unit/test_trading_engine_ws_health.py:189: mock_engine._handle_user_stream_disconnect()
tests/unit/test_trading_engine_ws_health.py:212: mock_engine._handle_user_stream_disconnect()step 20
$ grep -RIn "stop_loss_order_id.*get\|def .*stop.*loss\|stop_loss_order_id" src/engines/live | head -120; sed -n '400,510p' tests/unit/live/test_order_execution.py; sed -n '1,245p' tests/unit/test_trading_engine_ws_health.py
output · 22 lines
src/engines/live/execution/exit_handler.py:157: def _is_stop_loss_reason(self, exit_reason: str) -> bool: src/engines/live/execution/exit_handler.py:202: def _apply_stop_loss_gap_pricing( src/engines/live/execution/exit_handler.py:591: def _check_stop_loss( src/engines/live/execution/position_tracker.py:53: stop_loss_order_id: str | None = None src/engines/live/execution/position_tracker.py:273: def set_stop_loss_order_id(self, order_id: str, stop_loss_order_id: str) -> None: src/engines/live/execution/position_tracker.py:279: position.stop_loss_order_id = stop_loss_order_id src/engines/live/execution/position_tracker.py:286: stop_loss_order_id=stop_loss_order_id, src/engines/live/reconciliation.py:604: position.stop_loss_order_id = sl_order_id src/engines/live/reconciliation.py:617: …[8629 chars omitted]… mock_engine.exchange_interface = mock_exchange
10 more lines
mock_tracker = MagicMock()
mock_engine.order_tracker = mock_tracker
mock_reconciler = MagicMock()
mock_engine._periodic_reconciler = mock_reconciler
mock_engine._handle_user_stream_disconnect()
mock_exchange.mark_user_degraded.assert_called_once()
mock_tracker.enable_polling.assert_called_once()step 21
$ sed -n '1,220p' tests/unit/data_providers/test_binance_provider.py; sed -n '1,120p' tests/unit/test_binance_provider_websocket.py; grep -n "Client\|_attempt_client_init\|get_config" tests/unit/data_providers/test_binance_provider.py | head -100; grep -n "DEFAULT_.*TIMEOUT" src/config/constants.py
output · 39 lines
from datetime import UTC, datetime
from unittest.mock import Mock, patch
import pandas as pd
import pytest
pytestmark = pytest.mark.unit
try:
from src.data_providers.binance_provider import (
STOP_LOSS_LIMIT_SLIPPAGE_FACTOR,
BinanceProvider,27 more lines
with_rate_limit_retry,
)
from src.data_providers.exchange_interface import OrderSide
BINANCE_AVAILABLE = True
except ImportError:
BINANCE_AVAILABLE = False
BinanceProvider = Mock
with_rate_limit_retry = None
STOP_LOSS_LIMIT_SLIPPAGE_FACTOR = 0.005
OrderSide = Mock
@pytest.mark.skipif(not BINANCE_AVAILABLE, reason="Binance provider not available")
class TestBinanceDataProvider:
@pytest.mark.data_provider
def test_binance_provider_initialization(self):
with patch("src.data_providers.binance_provider.get_config") as mock_config:
mock_config_obj = Mock()
…[8629 chars omitted]…
tracker thread join
175:DEFAULT_REQUEST_TIMEOUT = 10 # Timeout for HTTP requests (webhooks, etc.)
282:DEFAULT_MODEL_LOAD_TIMEOUT = 60.0 # Timeout for loading ML models (ONNX, Keras)
283:DEFAULT_INFERENCE_TIMEOUT = 30.0 # Timeout for model inference
284:DEFAULT_API_REQUEST_TIMEOUT = 30.0 # Timeout for external API requests
285:DEFAULT_DATA_FETCH_TIMEOUT = 60.0 # Timeout for historical data fetchesstep 22
$ sed -n '400,505p' tests/unit/live/test_order_execution.py; sed -n '1,210p' tests/unit/test_trading_engine_ws_health.py; sed -n '1,115p' tests/unit/test_db_resilience.py
output · 40 lines
)
live_position_tracker.track_recovered_position(position, db_id=None)
result = live_exit_handler.execute_filled_exit(
position=position,
exit_reason="stop_loss",
filled_price=220.0,
current_balance=10000.0,
)
assert result.success is True
assert result.exit_price == pytest.approx(220.0)28 more lines
assert not live_position_tracker.has_position("entry_order_456")
# ============================================================================
# Tests for _handle_order_fill
# ============================================================================
class TestHandleOrderFill:
"""Tests for the _handle_order_fill method."""
def test_handle_fill_logs_event(self, engine_with_exchange):
"""Fill callback logs the event."""
with patch("src.engines.live.trading_engine.log_
…[8629 chars omitted]…
ion size"),
KeyError("close"),
RuntimeError("strategy produced an invalid signal"),
Exception("some unrelated application failure"),
],
)
def test_non_transient_errors_not_flagged(exc: BaseException) -> None:
assert is_transient(exc) is False
@pytest.mark.fast
def test_wrapped_transient_error_is_unwrapped() -> None:
inner = _op_error("could not connect to server")step 23
$ grep -n "def _initialize_client\|def _attempt_client_init" src/data_providers/binance_provider.py; sed -n '330,440p' src/data_providers/binance_provider.py; sed -n '510,620p' src/data_providers/binance_provider.py; grep -n "timeout" src/data_providers/binance_provider.py
output · 29 lines
364: def _initialize_client(self):
414: def _attempt_client_init(self, api_endpoint: str):
ValueError: If credentials are provided but malformed
"""
# If both are missing/None, return empty strings for read-only mode
if not api_key and not api_secret:
return "", ""
# If only one is provided, that's an error
if bool(api_key) != bool(api_secret):
raise ValueError(
"Binance credentials must be provided together. "17 more lines
"Either provide both BINANCE_API_KEY and BINANCE_API_SECRET, or neither."
)
# Validate credential format (reasonable minimum length)
if api_key and len(str(api_key).strip()) < 20:
if allow_test_credentials:
logger.debug("Binance provider allowing short API key for test environment")
else:
…[8035 chars omitted]…
= get_config().get_float(
651: klines = run_with_timeout(
654: timeout_seconds=data_timeout,
657: except InfraTimeoutError as timeout_err:
659: f"Binance API timeout after {data_timeout}s fetching {symbol} {timeframe}"
660: ) from timeout_err
1465: # this from ambiguous network/timeout errors (which return None).step 24
$ grep -RIn "\.start(.*max_steps\|engine.start\|target.*_trading_loop\|main_thread" tests/unit tests/integration/live_trading | head -200; grep -n "_trading_loop" tests/unit/live/test_live_engine_comprehensive_safety.py tests/integration/live_trading/test_engine_core.py
output · 7 lines
tests/unit/live/test_live_engine_comprehensive_safety.py:863: # Session ID should be None until engine starts tests/unit/test_trading_engine_websocket.py:177: mock_engine.main_thread = MagicMock() tests/unit/test_trading_engine_websocket.py:178: mock_engine.main_thread.is_alive.return_value = False tests/unit/test_trading_engine_websocket.py:190: mock_engine.main_thread = MagicMock() tests/unit/test_trading_engine_websocket.py:191: mock_engine.main_thread.is_alive.return_value = False tests/integration/live_trading/test_database_concurrency.py:32: engine.start(symbol="BTCUSDT", timeframe="1m", max_steps=1) tests/integration/live_trading/test_database_logging.py:81: "message": "Trading engine started",
step 25
$ grep -n "get_float" tests/unit/data_providers/test_binance_provider.py | head -50; sed -n '300,460p' tests/unit/test_binance_provider_websocket.py; grep -RIn "_attempt_client_init" tests | head -30; grep -n "def _make_engine\|mock_engine" tests/conftest.py tests/unit/test_trading_engine_ws_health.py
output · 20 lines
75: mock_config_obj.get_float.return_value = 60.0 # Timeout config
94: mock_config_obj.get_float.return_value = 60.0 # Timeout config
123: mock_config_obj.get_float.return_value = 60.0 # Timeout config
141: mock_config_obj.get_float.return_value = 60.0 # Timeout config
652: mock_config_obj.get_float.return_value = 60.0
def test_ws_healthy_false_when_user_stream_configured_but_resyncing(self, provider):
"""When user stream IS configured, RESYNCING flips ws_healthy to False.
Live margin trading subscribes a user stream — its state must be
reflected in the global health flag so the engine can fall back to
REST account-sync rather than trusting potentially stale WS state.
"""8 more lines
provider._kline_ws_state = WebSocketState.PRIMARY
provider._kline_event_received = True
provider
…[8629 chars omitted]…
ts/unit/test_trading_engine_ws_health.py:204: mock_engine.exchange_interface = mock_exchange
tests/unit/test_trading_engine_ws_health.py:207: mock_engine.order_tracker = mock_tracker
tests/unit/test_trading_engine_ws_health.py:210: mock_engine._periodic_reconciler = mock_reconciler
tests/unit/test_trading_engine_ws_health.py:212: mock_engine._handle_user_stream_disconnect()step 26
$ grep -RIn "def mock_engine\|@pytest.fixture.*mock_engine\|def engine_with_exchange" tests | head -60; grep -n "@pytest.fixture" tests/conftest.py | head -80; grep -n "mock_engine" tests/conftest.py
output · 28 lines
tests/unit/live/test_order_execution.py:99:def engine_with_exchange(mock_data_provider, mock_exchange, mock_order_tracker): tests/unit/test_trading_engine_websocket.py:13:def mock_engine(): tests/unit/test_trading_engine_ws_health.py:13:def mock_engine(): 132:@pytest.fixture(scope="session", autouse=True) 256:@pytest.fixture 297:@pytest.fixture 322:@pytest.fixture(scope="session") 343:@pytest.fixture 355:@pytest.fixture 372:@pytest.fixture 379:@pytest.fixture 391:@pytest.fixture
16 more lines
406:@pytest.fixture 431:@pytest.fixture 445:@pytest.fixture(scope="session") 451:@pytest.fixture(scope="session") 459:@pytest.fixture(scope="session") 477:@pytest.fixture(scope="session") 497:@pytest.fixture(scope="session") 513:@pytest.fixture 534:@pytest.fixture 556:@pytest.fixture 567:@pytest.fixture 575:@pytest.fixture 583:@pytest.fixture 618:@pytest.fixture(scope="session") 624:@pytest.fixture 676:@pytest.fixture(autouse=True)
step 27
$ sed -n '1,80p' tests/unit/test_trading_engine_ws_health.py; sed -n '1,145p' tests/unit/live/test_order_execution.py; sed -n '1,100p' tests/unit/test_trading_engine_websocket.py; git status --short
output · 39 lines
"""Tests for WebSocket health monitoring in LiveTradingEngine.""" from datetime import UTC, datetime, timedelta from unittest.mock import MagicMock, patch import pandas as pd import pytest from src.data_providers.binance_provider import WebSocketState @pytest.fixture
27 more lines
def mock_engine():
"""Create a LiveTradingEngine with mocked dependencies for WS health testing."""
with patch("src.engines.live.trading_engine.DatabaseManager"):
from src.engines.live.trading_engine import LiveTradingEngine
mock_strategy = MagicMock()
mock_strategy.get_risk_overrides.return_value = {}
mock_strategy.__class__.__name__ = "MockStrategy"
mock_strategy.config = {}
mock_dp = MagicMock()
mock_dp.get_live_data.return_value = _make_df()
mock_dp.get_current_price.return_value = 50000.0
engine = LiveTradingEngine(
st
…[8629 chars omitted]…
, "1h")
mock_engine.data_provider.get_live_data.assert_called_once()
@pytest.mark.fast
def test_returns_none_during_resyncing_live_mode(self, mock_engine):
"""Returns None during RESYNCING in live mode to freeze trading."""
mock_engine.enable_live_trading = True
mock_ws_provider = MagicMock()
mock_ws_provider._kline_ws_state = WebSocketState.RESYNCINGstep 28
$ cat pyproject.toml | sed -n '1,120p'; grep -n "^from typing\|import queue" src/engines/live/trading_engine.py; grep -n "def _execute_exit" -A25 src/engines/live/trading_engine.py; grep -n "_execute_exit(" src/engines/live/trading_engine.py | head -25output · 48 lines
[tool.black] line-length = 100 [tool.ruff] line-length = 100 [tool.ruff.lint] select = ["E","F","I","B","UP"] ignore = ["E203","E501","E402"] [tool.ruff.lint.isort] known-first-party = ["src","config","data_providers","database","live","monitoring","prediction","risk","strategies","utils","backtesting"]
36 more lines
[build-system]
requires = ["setuptools>=67", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "ai-trading-bot"
version = "0.1.0"
description = "AI Trading Bot CLI and tools"
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
"python-binance>=1.0.36",
"nest_asyncio>=1.6.0",
"pandas>=2.1.4",
"python-dotenv>=1.0.0",
"numpy>=1.26.2",
"scikit-learn>=1.3.2",
"requests>=2.31.0",
"textblob>=0.17.1",
"nltk>=3.9.0",
"ccxt>=4.4.85",
"tqdm>=4.66.1",
"sqlalchemy>=2.0.23",
"psycopg2-binary>=2.9.9",
"psutil>=5
…[1582 chars omitted]…
position.symbol,
1612: self._execute_exit(
2420: self._execute_exit(
2953: self._execute_exit(
3094: self._execute_exit(
3163: self._execute_exit(
3223: # portion. Calling _execute_exit(skip_live_close=False) would
3313: def _execute_exit(
4517: self._execute_exit(step 29
$ python - <<'PY'
output · 1 lines
Client kwargs before fix: {'testnet': False}step 30
$ python - <<'PY'
step 31
$ nl -ba src/engines/live/trading_engine.py | sed -n '1650,1715p'; nl -ba src/engines/live/trading_engine.py | sed -n '1910,2040p'; nl -ba src/engines/live/trading_engine.py | sed -n '2035,2085p'; grep -n "_pending_fill_exits" src/engines/live/trading_engine.py
output · 31 lines
1650 )
1651
1652 logger.info("Trading engine stopped")
1653
1654 def _signal_handler(self, signum: int, frame: Any) -> None:
1655 """Handle shutdown signals"""
1656 logger.info("Received signal %s", signum)
1657 self.stop()
1658 sys.exit(0)
1659
1660 def _trading_loop(self, symbol: str, timeframe: str, max_steps: int | None = None) -> None:
1661 """Main trading loop"""19 more lines
1662 from src.infrastructure import liveness # shared loop-liveness for /health (#627)
1663
1664 logger.info("Trading loop started")
1665 steps = 0
1666 cfg = get_config()
1667 self._active_symbol = symbol
1668 try:
1669 heartbeat_every = int(cfg.get("ENGINE_HEARTBEAT_STEPS", "60"))
1670 except Exception:
1671 heartbeat_every
…[8629 chars omitted]…
seen: set[int] = set()
2079 cur: BaseException | None = exc
2080 while cur is not None and id(cur) not in seen:
2081 seen.add(id(cur))
2082 text = str(cur).lower()
2083 if any(marker in text for marker in permanent_markers):
2084 return False
2085 if any(marker in text for marker in transient_markers):step 32
$ python - <<'PY'
step 33
$ sed -n '1,35p' src/engines/live/trading_engine.py; sed -n '455,510p' src/engines/live/trading_engine.py; sed -n '1150,1195p' src/engines/live/trading_engine.py; sed -n '1340,1395p' src/engines/live/trading_engine.py; sed -n '1420,1605p' src/engines/live/trading_engine.py; sed -n '1635,1705p' src/engines/live/trading_engine.py; sed -n '3150,3235p' src/engines/live/trading_engine.py
output · 46 lines
from __future__ import annotations import json import logging import math import os import queue import signal import sys import threading import time from dataclasses import asdict
34 more lines
from datetime import UTC, datetime, timedelta
from typing import Any
import pandas as pd
from sqlalchemy.exc import DBAPIError, InterfaceError, OperationalError
from src.config import get_config
from src.config.constants import (
DEFAULT_ACCOUNT_SNAPSHOT_INTERVAL,
DEFAULT_CHECK_INTERVAL,
DEFAULT_DATA_FRESHNESS_THRESHOLD,
DEFAULT_DB_OUTAGE_CLOSE_ONLY_SECONDS,
DEFAULT_DYNAMIC_RISK_ENABLED,
DEFAULT_END_OF_DAY_FLAT,
DEFAULT_ERROR_COOLDOWN,
DEFAULT_EXECUTION_FILL_POLICY,
DEFAULT_FEE_RATE,
DEFAULT_INITIAL_BALANCE,
DEFAULT_MARKET_TIMEZONE,
DEFAULT_MAX_CHECK_INTERVAL,
DEFAULT_MAX_FILLED_PRICE_DEVIATION,
DEFAULT_MAX_HOLDING_HOURS,
DEFAULT_MAX_POSITION_S
…[8630 chars omitted]…
self._pending_fill_exits.put((position_order_id, avg_price))
break
def _drain_pending_fill_exits(self) -> None:
"""Close positions whose server-side stop-loss fills were deferred."""
while True:
try:
position_order_id, fill_price = self._pending_fill_exits.get_nowait()
except queue.Empty:
returnstep 34
$ sed -n '1340,1425p' src/engines/live/trading_engine.py; sed -n '1425,1548p' src/engines/live/trading_engine.py; sed -n '1548,1635p' src/engines/live/trading_engine.py; sed -n '1635,1668p' src/engines/live/trading_engine.py; sed -n '1668,1705p' src/engines/live/trading_engine.py; sed -n '3190,3250p' src/engines/live/trading_engine.py
output · 30 lines
use_margin = getattr(self.exchange_interface, 'is_margin_mode', False)
self._periodic_reconciler = PeriodicReconciler(
exchange_interface=self.exchange_interface,
position_tracker=self.live_position_tracker,
db_manager=self.db_manager,
session_id=self.trading_session_id,
on_critical=self._enter_close_only_mode,
use_margin=use_margin,
)
self._periodic_reconciler.start()
logger.info("🔄 Periodic reconciler started")
except Exception as e:18 more lines
logger.warning("Failed to start periodic reconciler: %s", e)
# Try to start WebSocket streams for reduced API weight
self._start_websocket_streams(symbol, timeframe)
# Start main trading loop in separate
…[8629 chars omitted]…
dy have removed the position.
continue
self._execute_exit(
position,
reason="stop_loss",
limit_price=fill_price,
current_price=fill_price,
candle_high=None,
candle_low=None,
candle=None,
skip_live_close=True,step 35
$ nl -ba src/engines/live/trading_engine.py | sed -n '1385,1435p'; nl -ba src/engines/live/trading_engine.py | sed -n '1530,1660p'; nl -ba src/engines/live/trading_engine.py | sed -n '1800,1830p'; python -m py_compile src/engines/live/trading_engine.py
output · 29 lines
1385 def _exit_if_loop_crashed(self, exit_on_crash: bool) -> None: 1386 """Raise a non-zero exit only when a crashed loop is explicitly fatal.""" 1387 if self._loop_crashed and exit_on_crash: 1388 raise SystemExit(1) 1389 return None 1390 1391 def _enter_close_only_mode(self) -> None: 1392 """Enter close-only mode: no new entries, exits/stops/trailing still active.""" 1393 if not self._close_only_mode: 1394 self._close_only_mode = True 1395 logger.critical( 1396 "🚨 CLOSE-ONLY MODE ACTIVATED — no new entries until manual review"
17 more lines
1397 ) 1398 1399 def resume_trading(self) -> None: 1400 """Resume normal trading after close-only mode review.""" 1401 if self._close_only_mode: 1402 self._close_only_mode = False …[8629 chars omitted]… elf.check_interval) 1823 continue 1824 1825 # Validate DataFrame is not empty before iloc access 1826 current_index = len(df) - 1 1827 if current_index < 0: 1828 logger.error( 1829 "DataFrame became empty after readiness check - skipping iteration" 1830 )
step 36
$ git diff --stat && git diff -- src/config/constants.py src/data_providers/binance_provider.py src/engines/live/trading_engine.py | sed -n '1,360p';
output · 26 lines
src/config/constants.py | 1 + src/data_providers/binance_provider.py | 25 +++- src/engines/live/trading_engine.py | 243 ++++++++++++++++++++++----------- 3 files changed, 186 insertions(+), 83 deletions(-) diff --git a/src/config/constants.py b/src/config/constants.py index 342b5d0..6ea5216 100644 --- a/src/config/constants.py +++ b/src/config/constants.py @@ -283,6 +283,7 @@ DEFAULT_MODEL_LOAD_TIMEOUT = 60.0 # Timeout for loading ML models (ONNX, Keras) DEFAULT_INFERENCE_TIMEOUT = 30.0 # Timeout for model inference DEFAULT_API_REQUEST_TIMEOUT = 30.0 # Timeout for external API requests DEFAULT_DATA_FETCH_TIMEOUT = 60.0 # Timeout for historical data fetches
14 more lines
+DEFAULT_BINANCE_REST_TIMEOUT = 15.0 # Timeout for Binance REST API requests
# Numeric Precision Constants
DEFAULT_EPSILON = 1e-9 # Small value for floating point comparisons
diff --git a/src/d
…[8629 chars omitted]…
ions.items():
if position.stop_loss_order_id == order_id:
- position_to_close = position
logger.warning(
- "Stop-loss order %s filled for position %s at $%.2f - closing position",
+ "Stop-loss order %s filled for position %s at $%.2f - deferring closure",
order_id,
- pos_order_id,step 37
$ sed -n '1558,1635p' src/engines/live/trading_engine.py; sed -n '1915,1995p' src/engines/live/trading_engine.py
output · 28 lines
if reconnected:
self._ws_kline_active = True
logger.info("Kline WebSocket reconnected")
else:
self._ws_kline_provider.mark_kline_degraded()
self._ws_kline_active = False
logger.warning("Kline reconnect failed — REST polling resumed")
def _handle_user_stream_disconnect(self) -> None:
"""Handle user data stream failure. Resync orders and attempt reconnect."""
from src.engines.live.user_data_processor import UserDataProcessor
16 more lines
if not self.enable_live_trading or not self.exchange_interface:
return
try:
# 1. Stop the old user socket FIRST to prevent new events arriving
if hasattr(self.exchange_interface, "stop_user_stream"):
self.exchange_interface.stop_user_stream()
# 2. Now drain the UserDataProcessor (no new even
…[7192 chars omitted]…
short_take_profit = current_price * (
1
- getattr(
self.strategy,
"take_profit_pct",
DEFAULT_TAKE_PROFIT_PCT,
)
)step 38
$ grep -n -A15 -B5 'get_config' tests/unit/data_providers/test_binance_provider.py | sed -n '1,240p'; grep -RIn 'patch("src.data_providers.binance_provider.get_config"\|patch.*binance_provider.*get_config' tests | head -150output · 25 lines
25-
26-@pytest.mark.skipif(not BINANCE_AVAILABLE, reason="Binance provider not available")
27-class TestBinanceDataProvider:
28- @pytest.mark.data_provider
29- def test_binance_provider_initialization(self):
30: with patch("src.data_providers.binance_provider.get_config") as mock_config:
31- mock_config_obj = Mock()
32- mock_config_obj.get_required.return_value = "fake_key"
33- mock_config.return_value = mock_config_obj
34- provider = BinanceProvider()
35- assert provider is not None
36-13 more lines
37- @pytest.mark.data_provider
38- @patch("src.data_providers.binance_provider.Client")
39- def test_binance_historical_data_success(self, mock_client_class):
40- mock_client = Mock()
41- mock_client_class.return_value = mock_client
42- mock_client.get_historical_klines.return_value = [
43- [
4
…[8629 chars omitted]…
rs.binance_provider.get_config")
tests/unit/data_providers/test_startup_ban_retry.py:232: @patch("src.data_providers.binance_provider.get_config")
tests/unit/data_providers/test_startup_ban_retry.py:275: @patch("src.data_providers.binance_provider.get_config")
tests/unit/engines/live/test_margin_side_effect.py:417: with patch("src.data_providers.binance_provider.get_config") as mock_config, \step 39
$ cat src/infrastructure/timeout.py; grep -n "def reconnect_kline" -A42 src/data_providers/binance_provider.py; git diff --check; command -v ruff || true
output · 49 lines
"""Timeout utilities for protecting against blocking operations. Provides timeout decorators and context managers for operations that may block indefinitely (file I/O, model loading, network calls). """ import builtins import logging import math import signal import threading import time
37 more lines
from collections.abc import Callable
from contextlib import contextmanager
from functools import wraps
from typing import Any, TypeVar
logger = logging.getLogger(__name__)
T = TypeVar("T")
class OperationTimeoutError(builtins.TimeoutError):
"""Raised when an operation exceeds its timeout.
Inherits from the built-in TimeoutError to maintain compatibility
with code that catches the built-in exception.
"""
pass
# Backwards compatibility alias
TimeoutError = OperationTimeoutError # noqa: A001
def _timeout_handler(signum: int, _frame: Any) -> None:
"""Signal handler for
…[7195 chars omitted]…
ONNECT_MAX_RETRIES times before giving up.
2086-
2087- Args:
2088- on_user_event: Fresh callback for the new stream. If None, reuses
2089- the previously stored callback (for backward compatibility).
2090-
2091- Returns:
2092- True if reconnect succeeded, False otherwise.
2093- """
2094- callback = on_user_event or self._on_user_event_cbstep 40
$ grep -RIn 'Client(' src/data_providers/binance_provider.pyoutput · 5 lines
432: client = Client( 440: client = Client( 449: client = Client(tld="us", requests_params=requests_params) 451: client = Client(requests_params=requests_params) 635: return _OfflineClient()
step 41
$ pytest -q tests/unit/test_db_resilience.py tests/unit/test_trading_engine_ws_health.py tests/unit/test_binance_provider_websocket.py tests/unit/data_providers/test_binance_provider.py tests/unit/live/test_order_execution.py --disable-warnings --maxfail=5
output · 7 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [ 48%] [0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m …[2494 chars omitted]… adds_isIsolated_false [36m[1m=========================== short test summary info ============================[0m [31mFAILED[0m tests/unit/live/test_order_execution.py::[1mTestHandleOrderFill::test_handle_fill_detects_stop_loss_fill[0m - AssertionError: Expected '_execute_exit' to have been called once. Called 0... [31m[31m[1m1 failed[0m, [32m149 passed[0m[31m in 143.56s (0:02:23)[0m[0m
step 42
$ find .. -name AGENTS.md -o -name CLAUDE.md | sort
output · 1 lines
Refused: `find ..` walks the whole container FS. Scope under /repo.
step 43
$ python - <<'PY'
output · 21 lines
binance authenticated call('aaaaaaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbbbb', testnet=False, requests_params={'timeout': 15.0})
binanceus authenticated call('aaaaaaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbbbb', testnet=False, tld='us', requests_params={'timeout': 15.0})
binance public call(requests_params={'timeout': 15.0})
binanceus public call(tld='us', requests_params={'timeout': 15.0})
INFO:Strategy.MlBasic:Strategy 'MlBasic' initialized with components: SignalGen=MlBasic_signals, RiskMgr=core_risk_adapter, PosSizer=confidence_weighted_sizer
ERROR:src.engines.live.trading_engine:Trading loop crashed unexpectedly
Traceback (most recent call last):
File "/repo/src/engines/live/trading_engine.py", line 1380, in _run_trading_loop
self._trading_loop(symbol, timeframe, max_steps)
File "/usr/local/lib/python3.12/unittest/mock.py", line 1139, in __call__
return self._mock_call(*args, *
…[815 chars omitted]…9 more lines
ck.py", line 1143, in _mock_call
return self._execute_mock_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/unittest/mock.py", line 1202, in _execute_mock_call
raise result
RuntimeError: first
Configuration providers available: ['Environment Variables']
loop crash wrapper and opt-in exit OK
deferred exit drain continues after errors OKstep 44
$ grep -RIn "Client(" src | grep -E 'binance|Client' | head -100output · 5 lines
src/data_providers/binance_provider.py:432: client = Client( src/data_providers/binance_provider.py:440: client = Client( src/data_providers/binance_provider.py:449: client = Client(tld="us", requests_params=requests_params) src/data_providers/binance_provider.py:451: client = Client(requests_params=requests_params) src/data_providers/binance_provider.py:635: return _OfflineClient()
step 45
$ git diff -- src/engines/live/trading_engine.py | sed -n '1,460p'; git diff --check
output · 36 lines
diff --git a/src/engines/live/trading_engine.py b/src/engines/live/trading_engine.py index 8aecd9c..04a7c14 100644 --- a/src/engines/live/trading_engine.py +++ b/src/engines/live/trading_engine.py @@ -4,6 +4,7 @@ import json import logging import math import os +import queue import signal import sys import threading
24 more lines
@@ -470,6 +471,11 @@ class LiveTradingEngine:
self._ws_kline_active = False
self._ws_kline_provider = None
self._ws_health_thread = None
+ self._ws_health_monitor_lock = threading.Lock()
+
+ # Filled stop-loss callbacks may arrive on the order-polling thread.
+ # Defer position updates to the trading loop to avoid racing it.
+ self._pending_fill_exits: queue.SimpleQueue[tuple[str, float]] = queue.SimpleQueue()
# Performance tracker (unified with backtest engine)
from src.performance.tracker im
…[8629 chars omitted]…
skip_live_close=True,
+ )
+ except Exception:
+ logger.critical(
+ "Error draining deferred stop-loss exit for position %s",
+ position_order_id,
+ exc_info=True,
+ )
def _handle_partial_fill(
self, order_id: str, symbol: str, new_filled_qty: float, avg_price: floatstep 46
$ python - <<'PY'
step 47
$ grep -n 'stop_event.clear' src/engines/live/trading_engine.py; python - <<'PY'
step 48
$ grep -n "def __init__" -A45 src/data_providers/exchange_interface.py | head -100; sed -n '305,340p' src/data_providers/binance_provider.py
output · 39 lines
126: def __init__(self, api_key: str, api_secret: str, testnet: bool = False): 127- """ 128- Initialize the exchange interface. 129- 130- Args: 131- api_key: Exchange API key 132- api_secret: Exchange API secret 133- testnet: Whether to use testnet/sandbox mode 134- """ 135- self.api_key = api_key 136- self.api_secret = api_secret 137- self.testnet = testnet
27 more lines
138- self._client = None
139- self._initialize_client()
140-
141- @abstractmethod
142- def _initialize_client(self):
143- """Initialize the exchange-specific client"""
144- pass
145-
146- @abstractmethod
147- def test_connection(self) -> bool:
148- """Test the connection to the exchange"""
149- pass
150-
151- @abstractmethod
152- def get_account_info(self) -> dict[str, Any]:
153- "
…[1464 chars omitted]…
ValueError: If credentials are provided but malformed
"""
# If both are missing/None, return empty strings for read-only mode
if not api_key and not api_secret:
return "", ""
# If only one is provided, that's an error
if bool(api_key) != bool(api_secret):
raise ValueError(
"Binance credentials must be provided together. "step 49
$ python - <<'PY'
step 50
$ pytest -q tests/unit/test_db_resilience.py tests/unit/test_trading_engine_ws_health.py tests/unit/test_binance_provider_websocket.py tests/unit/live/test_order_execution.py -k 'not test_handle_fill_detects_stop_loss_fill' --disable-warnings --maxfail=5
output · 9 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [ 72%] [0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m …[872 chars omitted]… uts_down 0.01s setup tests/unit/test_binance_provider_websocket.py::TestStartKlineStream::test_starts_kline_stream_successfully 0.01s call tests/unit/test_trading_engine_ws_health.py::TestHandleUserStreamDisconnect::test_enables_polling_on_reconnect_failure (2 durations < 0.005s hidden. Use -vv to show these durations.) [32m[32m[1m100 passed[0m, [33m1 deselected[0m[32m in 9.78s[0m[0m
step 51
$ python - <<'PY'
output · 22 lines
INFO:Strategy.MlBasic:Strategy 'MlBasic' initialized with components: SignalGen=MlBasic_signals, RiskMgr=core_risk_adapter, PosSizer=confidence_weighted_sizer
ERROR:src.engines.live.trading_engine:Kline reconnect raised an exception: timeout
Traceback (most recent call last):
File "/repo/src/engines/live/trading_engine.py", line 1556, in _handle_kline_disconnect
reconnected = bool(self._ws_kline_provider.reconnect_kline())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/unittest/mock.py", line 1139, in __call__
return self._mock_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/unittest/mock.py", line 1143, in _mock_call
return self._execute_mock_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^10 more lines
File "/usr/local/lib/python3.12/unittest/mock.py",
…[606 chars omitted]…
usr/local/lib/python3.12/unittest/mock.py", line 1143, in _mock_call
return self._execute_mock_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/unittest/mock.py", line 1198, in _execute_mock_call
raise effect
TimeoutError: timeout
Configuration providers available: ['Environment Variables']
watchdog and reconnect recovery checks passedstep 52
$ sed -n '272,292p' src/config/constants.py; sed -n '415,465p' src/data_providers/binance_provider.py; sed -n '1160,1185p' src/engines/live/trading_engine.py; sed -n '1350,1395p' src/engines/live/trading_engine.py; sed -n '1720,1765p' src/engines/live/trading_engine.py; sed -n '2075,2100p' src/engines/live/trading_engine.py; sed -n '3195,3260p' src/engines/live/trading_engine.py; git diff --check; git status --short
output · 35 lines
DEFAULT_MFE_MAE_UPDATE_FREQUENCY_SECONDS = 60 DEFAULT_MFE_MAE_PRECISION_DECIMALS = 8 DEFAULT_MFE_MAE_LOG_LEVEL = "INFO" # Timeout Configuration Defaults # These values can be overridden via environment variables: # - MODEL_LOAD_TIMEOUT_SECONDS # - INFERENCE_TIMEOUT_SECONDS # - API_REQUEST_TIMEOUT_SECONDS # - DATA_FETCH_TIMEOUT_SECONDS # - BINANCE_REST_TIMEOUT_SECONDS DEFAULT_MODEL_LOAD_TIMEOUT = 60.0 # Timeout for loading ML models (ONNX, Keras)
23 more lines
DEFAULT_INFERENCE_TIMEOUT = 30.0 # Timeout for model inference
DEFAULT_API_REQUEST_TIMEOUT = 30.0 # Timeout for external API requests
DEFAULT_DATA_FETCH_TIMEOUT = 60.0 # Timeout for historical data fetches
DEFAULT_BINANCE_REST_TIMEOUT = 15.0 # Timeout for Binance REST API requests
# Numeric Precision Constants
DEFAULT_EPSILON = 1e-9 # Small value for floating point comparisons
DEFAULT_NORMALIZATION_EPSILON = (
1e-8 # Epsilon for f
…[8629 chars omitted]…
ne,
skip_live_close=True,
)
except Exception:
logger.critical(
"Error draining deferred stop-loss exit for position %s",
position_order_id,
exc_info=True,
)
M src/config/constants.py
M src/data_providers/binance_provider.py
M src/engines/live/trading_engine.pystep 53
$ pytest -q tests/integration/live_trading/test_database_concurrency.py tests/unit/test_health_liveness.py --disable-warnings --maxfail=3
output · 8 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [100%][0m ============================= slowest 10 durations ============================= 1.85s call tests/integration/live_trading/test_database_concurrency.py::test_concurrent_open_close_with_database 0.75s setup tests/integration/live_trading/test_database_concurrency.py::test_concurrent_open_close_with_database 0.33s setup tests/unit/test_health_liveness.py::test_health_starting_when_loop_not_beaten (7 durations < 0.005s hidden. Use -vv to show these durations.) [32m[32m[1m5 passed[0m[32m in 3.47s[0m[0m
step 54
$ python -m ruff check src/config/constants.py src/data_providers/binance_provider.py src/engines/live/trading_engine.py || true
output · 1 lines
/usr/local/bin/python: No module named ruff
step 55
$ python - <<'PY'
output · 28 lines
INFO:Strategy.MlBasic:Strategy 'MlBasic' initialized with components: SignalGen=MlBasic_signals, RiskMgr=core_risk_adapter, PosSizer=confidence_weighted_sizer
WARNING:src.engines.live.trading_engine:⚠️ PAPER TRADING MODE - No real orders will be executed
ERROR:src.engines.live.trading_engine:Trading loop crashed unexpectedly
Traceback (most recent call last):
File "/repo/src/engines/live/trading_engine.py", line 1382, in _run_trading_loop
self._trading_loop(symbol, timeframe, max_steps)
File "/usr/local/lib/python3.12/unittest/mock.py", line 1139, in __call__
return self._mock_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/unittest/mock.py", line 1143, in _mock_call
return self._execute_mock_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^16 more lines
File "/usr/local/lib/python3.12/unittest/mock.py", …[2267 chars omitted]… ======================================= 🏁 FINAL TRADING STATISTICS ============================================================ Initial Balance: $1,000.00 Final Balance: $1,000.00 Total Return: +0.00% Total PnL: $+0.00 Max Drawdown: 0.00% Total Trades: 0 Winning Trades: 0 Win Rate: 0.0% Active Positions: 0 ============================================================ clean loop stop does not mark crash
step 56
$ python - <<'PY'
output · 4 lines
INFO:Strategy.MlBasic:Strategy 'MlBasic' initialized with components: SignalGen=MlBasic_signals, RiskMgr=core_risk_adapter, PosSizer=confidence_weighted_sizer WARNING:src.engines.live.trading_engine:Stop-loss order sl filled for position entry at $91.25 - deferring closure Configuration providers available: ['Environment Variables'] stop-loss fill deferral and get_position-only drain checks passed
step 57
$ command -v black || true; command -v ruff || true; python - <<'PY'
step 58
$ git diff -- src/engines/live/trading_engine.py | sed -n '100,280p'; git diff --stat; git status --short; git diff --check
output · 29 lines
"""Start daemon thread to monitor WebSocket stream health.""" - self._ws_health_thread = threading.Thread( - target=self._ws_health_loop, daemon=True, name="WSHealthMonitor" - ) - self._ws_health_thread.start() + with self._ws_health_monitor_lock: + if self._ws_health_thread is not None and self._ws_health_thread.is_alive(): + return + self._ws_health_thread = threading.Thread( + target=self._ws_health_loop, daemon=True, name="WSHealthMonitor" + ) + self._ws_health_thread.start()
17 more lines
logger.info("WebSocket health monitor started")
+ def _ensure_ws_health_monitor_alive(self) -> None:
+ """Restart the health monitor when a configured stream outlives it."""
+ if not self.is_running or self.stop_event.is_set():
+ return
+ if no
…[8055 chars omitted]…
elf.consecutive_errors}): {e}", exc_info=True
)
src/config/constants.py | 2 +
src/data_providers/binance_provider.py | 37 ++++-
src/engines/live/trading_engine.py | 249 ++++++++++++++++++++++-----------
3 files changed, 203 insertions(+), 85 deletions(-)
M src/config/constants.py
M src/data_providers/binance_provider.py
M src/engines/live/trading_engine.pystep 59
$ git diff -- src/engines/live/trading_engine.py src/data_providers/binance_provider.py src/config/constants.py
output · 30 lines
diff --git a/src/config/constants.py b/src/config/constants.py index 342b5d0..4483275 100644 --- a/src/config/constants.py +++ b/src/config/constants.py @@ -279,10 +279,12 @@ DEFAULT_MFE_MAE_LOG_LEVEL = "INFO" # - INFERENCE_TIMEOUT_SECONDS # - API_REQUEST_TIMEOUT_SECONDS # - DATA_FETCH_TIMEOUT_SECONDS +# - BINANCE_REST_TIMEOUT_SECONDS DEFAULT_MODEL_LOAD_TIMEOUT = 60.0 # Timeout for loading ML models (ONNX, Keras) DEFAULT_INFERENCE_TIMEOUT = 30.0 # Timeout for model inference DEFAULT_API_REQUEST_TIMEOUT = 30.0 # Timeout for external API requests
18 more lines
DEFAULT_DATA_FETCH_TIMEOUT = 60.0 # Timeout for historical data fetches
+DEFAULT_BINANCE_REST_TIMEOUT = 15.0 # Timeout for Binance REST API requests
# Numeric Precision Constants
DEFAULT_EPSILON = 1e-9 # Small value for floating point comparisons
diff --git a/src/data_providers/binance_provider.py b/src/data_providers/binance_p
…[8630 chars omitted]…
skip_live_close=True,
+ )
+ except Exception:
+ logger.critical(
+ "Error draining deferred stop-loss exit for position %s",
+ position_order_id,
+ exc_info=True,
+ )
def _handle_partial_fill(
self, order_id: str, symbol: str, new_filled_qty: float, avg_price: floatstep 60
$ python - <<'PY'
output · 4 lines
src/engines/live/trading_engine.py:1236:105: "trading_hours_only": self.time_exit_policy.time_restrictions.trading_hours_only,
src/engines/live/trading_engine.py:1322:114: "⚠️ Balance correction pending but no trading session ID available - skipping database update"
src/engines/live/trading_engine.py:1629:103: logger.exception("Failed to enable order polling after user-stream resync failure")
src/engines/live/trading_engine.py:2093:101: f"Too many consecutive errors ({self.consecutive_errors}). Stopping engine.",step 61
$ python - <<'PY'
step 62
$ python -m py_compile src/config/constants.py src/data_providers/binance_provider.py src/engines/live/trading_engine.py && git diff --check && python - <<'PY'
output · 3 lines
src/config/constants.py long lines: [(176, 127), (334, 143), (350, 115)] count: 3 src/data_providers/binance_provider.py long lines: [(385, 111), (560, 106), (701, 121), (705, 115), (1194, 106)] count: 5 src/engines/live/trading_engine.py long lines: [(176, 128), (347, 111), (358, 145), (432, 114), (575, 111), (764, 108), (1236, 105), (1322, 114), (1778, 101), (1987, 127), (2095, 101), (2295, 117), (2417, 103), (2437, 122), (2449, 112), (2452, 132), (2605, 165), (2607, 103), (2617, 108), (2620, 128)] count: 31
step 63
$ pytest -q tests/unit/data_providers/test_binance_provider.py tests/unit/data_providers/test_startup_ban_retry.py --disable-warnings --maxfail=3
output · 9 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [100%][0m ============================= slowest 10 durations ============================= 3.04s call tests/unit/data_providers/test_binance_provider.py::TestGetOrderIdRouting::test_underscore_order_id_delegates_to_get_order_by_client_id 3.01s call tests/unit/d …[809 chars omitted]… ::TestMarginBaseAssetGuard::test_warns_non_usdt_holdings_in_live_mode 3.01s call tests/unit/data_providers/test_binance_provider.py::TestPlaceStopLossOrder::test_custom_limit_price_used 3.01s call tests/unit/data_providers/test_binance_provider.py::TestGetOrderIdRouting::test_alphanumeric_order_id_delegates_to_get_order_by_client_id [32m[32m[1m61 passed[0m[32m in 130.16s (0:02:10)[0m[0m
step 64
$ sed -n '1,220p' AGENTS.md
output · 32 lines
# AGENTS.md This file provides guidance to AGENTS other than Claude Code. If you are Claude Code, stop and read CLAUDE.md instead. ## Project Overview This is a modular cryptocurrency trading system capable of multiple strategies with different risk and return profiles. It supports backtesting, live trading (paper and live), ML-driven predictions (price and sentiment), PostgreSQL logging, and Railway deployment. **Tech Stack**: Python 3.11+, PostgreSQL, TensorFlow/ONNX, Flask, SQLAlchemy, pandas, scikit-learn ## Essential Commands
20 more lines
### Environment Setup **Remote Environments (Claude Code Web)**: You are operating in a claude code web environment if CLAUDE_CODE_REMOTE == true: - The virtual environment is automatically created and configured by the `sessionStart` hook - Dependencies from `requirements-server.txt` are installed automatically at session start - Use the existing ` …[6894 chars omitted]… n: - `Strategy` class composes `SignalGenerator`, `RiskManager`, `PositionSizer` - Main interface: `process_candle(df, index, balance, positions) -> TradingDecision` - `TradingDecision` contains signal, position size, regime context, risk metrics - Components are independently testable and reusable ### Database Schema **Core Tables**: - `trading_sessions` - Track sessions with strategy configuration
step 65
$ pytest -q tests/unit/test_db_resilience.py tests/unit/test_trading_engine_ws_health.py tests/unit/test_binance_provider_websocket.py tests/unit/live/test_order_execution.py -k 'not test_handle_fill_detects_stop_loss_fill' --disable-warnings --maxfail=3
output · 9 lines
[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m [ 72%] [0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m[32m.[0m …[1011 chars omitted]… ure 0.01s call tests/unit/live/test_order_execution.py::TestReconcilePositionsWithExchange::test_reconcile_uses_entry_balance_not_current 0.01s setup tests/unit/test_binance_provider_websocket.py::TestStartKlineStream::test_starts_kline_stream_successfully (1 durations < 0.005s hidden. Use -vv to show these durations.) [32m[32m[1m100 passed[0m, [33m1 deselected[0m[32m in 11.09s[0m[0m
step 66
$ done: {"notes": "Implemented live-loop crash tracking/optional non-zero exit, finite Binance REST client timeouts, resilient websocket recovery/watchdog, and deferred stop-loss fill exits. Targeted tests and manual reproductions pass; the existing immediate-stop-loss-close test is incompatible with the required deferred behavior."}output · 1 lines
submitted.