Don't blow up if fields are missing

[brndnmtthws]

Sometimes we don't have data (i.e., no bids/asks, no market price) for contracts, and rather than blowing up we should just continue gracefully

FYI @junyuanz1

[brndnmtthws]

I'm not 100% sure this is the desired behaviour, I need to understand some of the changes from #521 a little better. I think in some cases it might be better to go through the path above this exception (i.e., in case it's better to close the contract).

[junyuanz1]

so in this code block, there are several potential calls that could cause this problem:

buy_ticker = await self.ibkr.get_ticker_for_contract(position.contract) (requires market price)

await self.put_is_itm(position.contract) (requires market price)

sell_ticker = await self.find_eligible_contracts(...) (required market price for the symbol, no required field for option contracts)

maximum_new_contracts = await self.get_maximum_new_contracts_for (requires market price)

My assumption was marketPrice is the the least strict data points according to the ib_async code:

if self.hasBidAsk():
    if self.bid <= self.last <= self.ask:
        price = self.last
    else:
        price = self.midpoint()
else:
    price = self.last

so if this data is not available, then it means the symbol does not have an open interest or there is no last price. so that's why I make it a required field for most of the underlying related API calls.

[brndnmtthws]

Here's the stack trace for the case where this was triggered:

 [121B blob data]
 [123B blob data]
 [120B blob data]
 [356B blob data]
 │ /usr/local/lib/python3.10/dist-packages/thetagang/portfolio_manager.py:578 in manage             │
 │                                                                                                  │
 │    575 │   │   │   console.print(Panel(Group(group1, group2)))                                   │
 │    576 │   │   │                                                                                 │
 │    577 │   │   │   await self.close_puts(                                                        │
 │ ❱  578 │   │   │   │   closeable_puts + await self.roll_puts(rollable_puts, account_summary)     │
 │    579 │   │   │   )                                                                             │
 │    580 │   │   │   await self.close_calls(                                                       │
 │    581 │   │   │   │   closeable_calls                                                           │
 │                                                                                                  │
 │ /usr/local/lib/python3.10/dist-packages/thetagang/portfolio_manager.py:1233 in roll_puts         │
 │                                                                                                  │
 │   1230 │   │   puts: List[PortfolioItem],                                                        │
 │   1231 │   │   account_summary: Dict[str, AccountValue],                                         │
 │   1232 │   ) -> List[PortfolioItem]:                                                             │
 │ ❱ 1233 │   │   return await self.roll_positions(puts, "P", account_summary)                      │
 │   1234 │                                                                                         │
 │   1235 │   async def close_calls(self, calls: List[PortfolioItem]) -> None:                      │
 │   1236 │   │   return await self.close_positions("C", calls)                                     │
 │                                                                                                  │
 │ /usr/local/lib/python3.10/dist-packages/thetagang/portfolio_manager.py:1478 in roll_positions    │
 │                                                                                                  │
 │   1475 │   │   │   │   )                                                                         │
 │   1476 │   │                                                                                     │
 │   1477 │   │   tasks = [row_position_task(position) for position in positions]                   │
 │ ❱ 1478 │   │   await tqdm_asyncio.gather(*tasks, desc=f"Rolling {right} positions...")           │
 │   1479 │   │                                                                                     │
 │   1480 │   │   return closeable_positions                                                        │
 │   1481                                                                                           │
 │                                                                                                  │
 │ /usr/local/lib/python3.10/dist-packages/tqdm/asyncio.py:79 in gather                             │
 │                                                                                                  │
 │   76 │   │   │   return i, await f                                                               │
 │   77 │   │                                                                                       │
 │   78 │   │   ifs = [wrap_awaitable(i, f) for i, f in enumerate(fs)]                              │
 │ ❱ 79 │   │   res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout,           │
 │   80 │   │   │   │   │   │   │   │   │   │   │   │    total=total, **tqdm_kwargs)]               │
 │   81 │   │   return [i for _, i in sorted(res)]                                                  │
 │   82                                                                                             │
 │                                                                                                  │
 │ /usr/local/lib/python3.10/dist-packages/tqdm/asyncio.py:79 in <listcomp>                         │
 │                                                                                                  │
 │   76 │   │   │   return i, await f                                                               │
 │   77 │   │                                                                                       │
 │   78 │   │   ifs = [wrap_awaitable(i, f) for i, f in enumerate(fs)]                              │
 │ ❱ 79 │   │   res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout,           │
 │   80 │   │   │   │   │   │   │   │   │   │   │   │    total=total, **tqdm_kwargs)]               │
 │   81 │   │   return [i for _, i in sorted(res)]                                                  │
 │   82                                                                                             │
 │                                                                                                  │
 │ /usr/lib/python3.10/asyncio/tasks.py:571 in _wait_for_one                                        │
 │                                                                                                  │
 │   568 │   │   if f is None:                                                                      │
 │   569 │   │   │   # Dummy value from _on_timeout().                                              │
 │   570 │   │   │   raise exceptions.TimeoutError                                                  │
 │ ❱ 571 │   │   return f.result()  # May raise f.exception().                                      │
 │   572 │                                                                                          │
 │   573 │   for f in todo:                                                                         │
 │   574 │   │   f.add_done_callback(_on_completion)                                                │
 │                                                                                                  │
 │ /usr/lib/python3.10/asyncio/futures.py:201 in result                                             │
 │                                                                                                  │
 │   198 │   │   │   raise exceptions.InvalidStateError('Result is not ready.')                     │
 │   199 │   │   self.__log_traceback = False                                                       │
 │   200 │   │   if self._exception is not None:                                                    │
 │ ❱ 201 │   │   │   raise self._exception.with_traceback(self._exception_tb)                       │
 │   202 │   │   return self._result                                                                │
 │   203 │                                                                                          │
 │   204 │   def exception(self):                                                                   │
 │                                                                                                  │
 │ /usr/lib/python3.10/asyncio/tasks.py:232 in __step                                               │
 │                                                                                                  │
 │   229 │   │   │   if exc is None:                                                                │
 │   230 │   │   │   │   # We use the `send` method directly, because coroutines                    │
 │   231 │   │   │   │   # don't have `__iter__` and `__next__` methods.                            │
 │ ❱ 232 │   │   │   │   result = coro.send(None)                                                   │
 │   233 │   │   │   else:                                                                          │
 │   234 │   │   │   │   result = coro.throw(exc)                                                   │
 │   235 │   │   except StopIteration as exc:                                                       │
 │                                                                                                  │
 │ /usr/local/lib/python3.10/dist-packages/tqdm/asyncio.py:76 in wrap_awaitable                     │
 │                                                                                                  │
 │   73 │   │   Wrapper for `asyncio.gather`.                                                       │
 │   74 │   │   """                                                                                 │
 │   75 │   │   async def wrap_awaitable(i, f):                                                     │
 │ ❱ 76 │   │   │   return i, await f                                                               │
 │   77 │   │                                                                                       │
 │   78 │   │   ifs = [wrap_awaitable(i, f) for i, f in enumerate(fs)]                              │
 │   79 │   │   res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout,           │
 │                                                                                                  │
 │ /usr/local/lib/python3.10/dist-packages/thetagang/portfolio_manager.py:1302 in row_position_task │
 │                                                                                                  │
 │   1299 │   │   │   │   symbol = position.contract.symbol                                         │
 │   1300 │   │   │   │                                                                             │
 │   1301 │   │   │   │   position.contract.exchange = self.get_order_exchange()                    │
 │ ❱ 1302 │   │   │   │   buy_ticker = await self.ibkr.get_ticker_for_contract(position.contract)   │
 │   1303 │   │   │   │                                                                             │
 │   1304 │   │   │   │   strike_limit = get_strike_limit(self.config, symbol, right)               │
 │   1305 │   │   │   │   if right.startswith("C"):                                                 │
 │                                                                                                  │
 │ /usr/local/lib/python3.10/dist-packages/thetagang/ibkr.py:151 in get_ticker_for_contract         │
 │                                                                                                  │
 │   148 │   │   │   │   │   "Not all required fields were processed successfully"                  │
 │   149 │   │   │   │   )                                                                          │
 │   150 │   │                                                                                      │
 │ ❱ 151 │   │   return await self.__market_data_streaming_handler__(                               │
 │   152 │   │   │   contract,                                                                      │
 │   153 │   │   │   generic_tick_list,                                                             │
 │   154 │   │   │   lambda ticker: ticker_handler(ticker),                                         │
 │                                                                                                  │
 │ /usr/local/lib/python3.10/dist-packages/thetagang/ibkr.py:234 in                                 │
 │ __market_data_streaming_handler__                                                                │
 │                                                                                                  │
 │   231 │   │   await self.ib.qualifyContractsAsync(contract)                                      │
 │   232 │   │   ticker = self.ib.reqMktData(contract, genericTickList=generic_tick_list)           │
 │   233 │   │   try:                                                                               │
 │ ❱ 234 │   │   │   await handler(ticker)                                                          │
 │   235 │   │   │   return ticker                                                                  │
 │   236 │   │   finally:                                                                           │
 │   237 │   │   │   self.ib.cancelMktData(contract)                                                │
 │                                                                                                  │
 │ /usr/local/lib/python3.10/dist-packages/thetagang/ibkr.py:147 in ticker_handler                  │
 │                                                                                                  │
 │   144 │   │   │   │   asyncio.gather(*required_tasks), asyncio.gather(*optional_tasks)           │
 │   145 │   │   │   )                                                                              │
 │   146 │   │   │   if not all(required_results):                                                  │
 │ ❱ 147 │   │   │   │   raise RequiredFieldValidationError(                                        │
 │   148 │   │   │   │   │   "Not all required fields were processed successfully"                  │
 │   149 │   │   │   │   )                                                                          │
 │   150                                                                                            │
 ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
 RequiredFieldValidationError: Not all required fields were processed successfully
 [62B blob data]

In this particular case, the contract has no bids, and it would be preferable to simply close it out because it's profitable, so I think it should be triggering the close_if_unable_to_roll() ... path.

This might be a special case where we should just treat contracts with no market price as $0.01 or something. With the old behaviour, it wouldn't throw an exception if there's no market price, it would just make a best effort.

[brndnmtthws]

See #524, I think that is the preferred behaviour in this case.

[junyuanz1]

okay, so if that's the case, then we should:

- buy_ticker = await self.ibkr.get_ticker_for_contract(position.contract)
+ buy_ticker = await self.ibkr.get_ticker_for_contract(position.contract, required_fields = [], optional_fields = [TickerField.MARKET_PRICE, TickerField.MIDPOINT])