Engineering an Automated Trading System from Scratch
How modular design, custom technical indicators, and real-time data collection power quantitative trading models.
This open-source framework is a comprehensive, pure-Python solution designed to streamline the entire workflow of quantitative trading. It acts as an end-to-end toolkit that bridges the gap between raw market data and actionable trading strategies. By intentionally avoiding heavy, C-based dependencies, the architecture remains highly accessible for developers looking to build, test, and deploy financial algorithms natively. Its primary goal is to provide a seamless pipeline for real-time data collection, intricate market analysis, and the rigorous backtesting of custom investment models, making it easier to automate complex trading rules.
Download the entire source code using the link below. And also the entire explanation alright of each function, each code cell.
Under the hood, the system is powered by four highly modular engines that handle distinct phases of the trading lifecycle. The data ingestion layer pulls historical and real-time stock quotes from web APIs or local databases like Excel and HBase. Once the data is acquired, a custom-built Technical Analysis Library processes market indicators directly in Python. This feeds into a dynamic stock-picking component that filters the market based on user-defined criteria. Finally, a robust backtesting engine allows developers to validate algorithmic models—such as momentum trading, z-score strategies, or dollar-cost averaging—against historical market conditions before risking any real capital.
Today, the framework is best utilized as a powerful educational resource rather than a production-ready trading bot. Because the original codebase was developed over a decade ago, it relies on early Python standards and legacy data endpoints that have since evolved or been deprecated. However, it excels as a transparent, foundational case study in financial engineering. For developers, data scientists, and researchers building their own quantitative tools today, the repository offers a masterclass in structuring a modular trading environment, illustrating exactly how to orchestrate data scraping, indicator calculation, and automated investment planning from scratch.
# file path: deprecated/example/chinaReturn.py
from operator import itemgetter
import time
import os
from ultrafinance.lib.stockMeasurement import StockMeasurement
from ultrafinance.lib.historicalDataStorage import HistoricalDataStorage
from ultrafinance.lib.yahooFinance import YahooFinance
from ultrafinance.lib.dataType import StockDailyType
from ultrafinance.lib.excelLib import ExcelLib
import logging
The imports bring together utility, data-access, measurement and persistence pieces that ChinaReturn orchestrates for its example workflow. operator.itemgetter is used for concise extraction or sorting of fields when the module manipulates lists or tuples of market records. The standard time and os modules supply simple timing and filesystem operations needed when fetching data and writing files. YahooFinance represents the data-access module that retrieves benchmark historical series from an external source, while StockDailyType is the project-level representation for daily tick/quote records so the code can treat rows consistently with the rest of the pipeline. HistoricalDataStorage is the persistence helper that assembles and writes the Excel workbooks ChinaReturn generates, and ExcelLib is the complementary reader used later in the analysis step to open those sheets. StockMeasurement provides the domain metrics and return calculations that ChinaReturn uses to build its benchmark values and summaries. Finally, logging is pulled in to record progress and any problems during the fetch, build and Excel read/write stages of this deprecated example.
# file path: deprecated/example/chinaReturn.py
LOG = logging.getLogger()
LOG is initialized by calling logging.getLogger to create a module-level logger reference that the ChinaReturn class and its methods can reuse to record runtime diagnostics. By holding that logger in LOG, ChinaReturn.init, ChinaReturn.saveHistroyDataIntoExcel, and ChinaReturn.analyze can emit informational, debug and error messages about assembling historical market data, building benchmark series with buildBenchmarkValues and YahooFinance, persisting results via HistoricalDataStorage.buildExlsFromFile, and later reading sheets with ExcelRead and ExcelLib. Because logging.getLogger yields the root/default logger when no specific name is supplied, LOG inherits the application’s global logging configuration so messages from this deprecated example flow into the same centralized logging pipeline used across the project’s data acquisition and backtest layers.
# file path: deprecated/example/chinaReturn.py
benchmarks = {'DEFAULT': '^GSPC',
'L': '^STI',
'HK': '^HSI',
'SI': '^FTSE'}
The variable benchmarks declares a small lookup that associates short label keys to the Yahoo Finance index symbols that ChinaReturn will treat as reference series. When ChinaReturn runs its workflow it uses these labels to decide which external index tickers to fetch via YahooFinance and to hand into buildBenchmarkValues so benchmark time series get constructed for persistence; the entry labeled DEFAULT provides the fallback S&P 500 benchmark while the other labels point to regional indices (the entries for L, HK and SI correspond to additional market benchmarks). Those constructed benchmark series are then written out by HistoricalDataStorage.buildExlsFromFile and later read back by ExcelRead and ExcelLib during the analyze step.
# file path: deprecated/example/chinaReturn.py
benchmarkValues = {}
When ChinaReturn begins its work it allocates an empty mapping called benchmarkValues to serve as the accumulator for benchmark time series that will be produced and persisted during the run. This mapping is the place where buildBenchmarkValues and the YahooFinance data pulls will store each benchmark’s normalized series (keyed by the benchmark identifier), and those populated entries are then handed off to HistoricalDataStorage.buildExlsFromFile by saveHistroyDataIntoExcel and later read back by ExcelRead and ExcelLib inside analyze. Initializing benchmarkValues up front gives the orchestration code a predictable container to collect multiple benchmark series as the data acquisition and Excel persistence steps execute.
# file path: deprecated/example/chinaReturn.py
def buildBenchmarkValues():
print "Building BenchmarkValues"
for key in benchmarks.values():
benchmarkValues[key] = YahooFinance().getHistoricalPrices(key, '19010101', '20130101')
time.sleep(1)
print "BenchmarkValues %s built" % benchmarkValues.keys()
ChinaReturn orchestrates creation of benchmark time series for the analysis pipeline, and buildBenchmarkValues is the routine that fetches and caches those benchmark series so downstream measurement code has market references to compare against. It starts by announcing the build, then iterates the set of benchmark symbols held in the benchmarks collection; for each symbol it calls YahooFinance.getHistoricalPrices to retrieve daily historical price records for the fixed date window from early 1901 through the start of 2013. YahooFinance.getHistoricalPrices performs network I/O via YahooFinance.__request and returns a time-ordered list of Quote/StockDailyType records (date, open, high, low, close, volume, adjClose), and buildBenchmarkValues stores each of those lists into the benchmarkValues mapping under the corresponding symbol. The routine pauses one second between requests to rate-limit calls to Yahoo, and when finished it prints a summary of the benchmark keys that were built. The populated benchmarkValues mapping is then consumed by ChinaReturn.analyze and by StockMeasurement methods such as linearRegression, returnRate, alpha and relativeReturnRate to compute market returns, regression parameters, and relative performance; any network failures during the Yahoo fetches surface as the
# file path: ultrafinance/dam/yahooFinance.py
import urllib
import traceback
from ultrafinance.model import Quote
from ultrafinance.lib.errors import UfException, Errors
import logging
urllib is brought in to perform the raw HTTP interactions and URL encoding needed when YahooFinance reaches out to Yahoo endpoints for historical data; traceback is available so exception handlers can capture and include the Python stack trace when something goes wrong during those network or parsing operations. Quote is the domain model class used to convert the fetched rows into the normalized Quote objects that TickFeeder and downstream strategies expect. UfException and Errors provide the project’s custom error types and error codes so YahooFinance can raise consistent, high-level failures when requests, HBase reads, or parsing fail. logging is used throughout to emit informational and error diagnostics during fetches, HBase lookups, and the subsequent normalization into Quote instances.
# file path: ultrafinance/dam/yahooFinance.py
class YahooFinance(object):
def __request(self, symbol, stat):
try:
url = 'http://finance.yahoo.com/d/quotes.csv?s=%s&f=%s' % (symbol, stat)
return urllib.urlopen(url).read().strip().strip('"')
except IOError:
raise UfException(Errors.NETWORK_ERROR, "Can't connect to Yahoo server")
except BaseException:
raise UfException(Errors.UNKNOWN_ERROR, "Unknown Error in YahooFinance.__request %s" % traceback.format_exc())
def getAll(self, symbol):
values = self.__request(symbol, 'l1c1va2xj1b4j4dyekjm3m4rr5p5p6s7').split(',')
data = {}
data['price'] = values[0]
data['change'] = values[1]
data['volume'] = values[2]
data['avg_daily_volume'] = values[3]
data['stock_exchange'] = values[4]
data['market_cap'] = values[5]
data['book_value'] = values[6]
data['ebitda'] = values[7]
data['dividend_per_share'] = values[8]
data['dividend_yield'] = values[9]
data['earnings_per_share'] = values[10]
data['52_week_high'] = values[11]
data['52_week_low'] = values[12]
data['50day_moving_avg'] = values[13]
data['200day_moving_avg'] = values[14]
data['price_earnings_ratio'] = values[15]
data['price_earnings_growth_ratio'] = values[16]
data['price_sales_ratio'] = values[17]
data['price_book_ratio'] = values[18]
data['short_ratio'] = values[19]
return data
def getQuotes(self, symbol, start, end):
try:
start = str(start).replace('-', '')
end = str(end).replace('-', '')
url = 'http://ichart.yahoo.com/table.csv?s=%s&' % symbol + \
'd=%s&' % str(int(end[4:6]) - 1) + \
'e=%s&' % str(int(end[6:8])) + \
'f=%s&' % str(int(end[0:4])) + \
'g=d&' + \
'a=%s&' % str(int(start[4:6]) - 1) + \
'b=%s&' % str(int(start[6:8])) + \
'c=%s&' % str(int(start[0:4])) + \
'ignore=.csv'
days = urllib.urlopen(url).readlines()
values = [day[:-2].split(',') for day in days]
data = []
for value in values[1:]:
data.append(Quote(value[0], value[1], value[2], value[3], value[4], value[5], value[6]))
dateValues = sorted(data, key = lambda q: q.time)
return dateValues
except IOError:
raise UfException(Errors.NETWORK_ERROR, "Can't connect to Yahoo server")
except BaseException:
raise UfException(Errors.UNKNOWN_ERROR, "Unknown Error in YahooFinance.getHistoricalPrices %s" % traceback.format_exc())
YahooFinance implements the Yahoo-specific data-access layer that the rest of the pipeline uses to fetch snapshots and historical price series and to normalize that raw CSV data into Quote objects for the TickFeeder, YahooDAM and higher-level modules. Its internal helper __request builds a Yahoo CSV snapshot endpoint from a symbol and a field mask, performs an HTTP read and returns the trimmed response, and maps low-level IO or unexpected exceptions into UfException with appropriate error codes so callers see a consistent error model. getAll uses __request to pull the snapshot CSV, splits the comma-separated fields and assembles them into a dictionary of named fundamentals (price, change, volume, market cap, book value, various ratios, etc.) so strategy and picker code can consume named values instead of raw CSV positions. getQuotes accepts symbol, start and end, normalizes incoming date strings, constructs the Yahoo historical table CSV query (including the zero-based month parameter Yahoo expects), fetches the CSV, parses each data row into Quote instances (the Quote constructor, as described earlier, handles numeric conversion and converts missing markers into zeros), sorts the resulting Quote list by date/time and returns that chronological list. On network failure or unexpected exceptions getQuotes also raises UfException with network or unknown error codes. Overall YahooFinance converts Yahoo’s HTTP/CSV interface into the domain objects and error semantics the backtest pipeline expects, and it is called by YahooDAM, HistoricalDataFeeder, StockMeasurement and the unit tests to supply both snapshot fundamentals and time-series quotes.
# file path: ultrafinance/dam/yahooFinance.py
def __request(self, symbol, stat):
try:
url = 'http://finance.yahoo.com/d/quotes.csv?s=%s&f=%s' % (symbol, stat)
return urllib.urlopen(url).read().strip().strip('"')
except IOError:
raise UfException(Errors.NETWORK_ERROR, "Can't connect to Yahoo server")
except BaseException:
raise UfException(Errors.UNKNOWN_ERROR, "Unknown Error in YahooFinance.__request %s" % traceback.format_exc())
YahooFinance.__request is the low-level helper that talks to Yahoo to fetch a single CSV snapshot line for a given symbol and a Yahoo format code (stat). It builds a Yahoo CSV-quote API URL using the provided symbol and stat flag, performs a synchronous HTTP fetch via urllib, reads the full response body and normalizes it by trimming whitespace and surrounding quotation marks so higher-level callers can reliably split the CSV fields. YahooFinance.getAll and other YahooFinance methods rely on the cleaned string returned by __request to parse individual fields into the Quote/data structures used by the TickFeeder and strategies. If the network call fails with an I/O problem, __request converts that into a UfException using Errors.NETWORK_ERROR; any other unexpected failure is wrapped into a UfException using Errors.UNKNOWN_ERROR and includes the formatted traceback, so error semantics are consistent with the rest of the data-access layer.
# file path: ultrafinance/backTest/stateSaver/hbaseSaver.py
from ultrafinance.lib.errors import Errors, UfException
from ultrafinance.dam.hbaseLib import HBaseLib
from ultrafinance.backTest.stateSaver import StateSaver
from hbase.Hbase import Mutation, ColumnDescriptor
import logging
The imports wire HbaseSaver into three parts of the system: error handling, the HBase access layer, and the StateSaver contract, plus a couple of low-level HBase primitives and logging for runtime diagnostics. Errors and UfException provide the project’s exception types and error codes that HbaseSaver will raise or map to when persistence operations fail, so it can integrate with the backtest pipeline’s unified error handling. HBaseLib is the higher-level HBase helper that HbaseSaver delegates table lifecycle and row-level read/write operations to, reflecting the file’s role as the persistence layer for backtest state. StateSaver is the abstract/stateful interface HbaseSaver implements so the backtest runtime can call reset and commit without knowing storage details. Mutation and ColumnDescriptor are the low-level HBase client primitives HbaseSaver will use to construct row mutations and define column families when interacting directly with HBase. Finally, logging is imported so HbaseSaver can emit operational logs and diagnostics during resets, commits, and error handling.
# file path: ultrafinance/backTest/stateSaver/hbaseSaver.py
def read(self, row, col):
oneRow = self.__hbase.getRow(self.tableName, row)
keyValues = oneRow.columns
key = "%s:" % col
if key in keyValues:
return keyValues[key].value
else:
return None
HbaseSaver.read is the small runtime entry that fetches a single persisted cell from HBase so the backtest can restore runtime state. It uses the HBase client instance created in HbaseSaver.init (self.__hbase) and asks HBaseLib.getRow for the row from the table identified by self.tableName; HBaseLib.getRow returns either a falsy result when no row is found or the first row result when there is data. Once it has the row result it pulls the row’s columns mapping and constructs the HBase-style column key by taking the requested column name and appending a colon to form the lookup key used in that mapping. The control flow then checks whether that constructed key exists in the row’s columns; on the happy path it returns the stored value for that column, and if the column key is not present it returns None so callers know no persisted value exists for that row/column pair.
# file path: ultrafinance/dam/hbaseLib.py
from thrift.transport import TSocket, TTransport
from thrift.protocol import TBinaryProtocol
from hbase import ttypes
from hbase.Hbase import Client, ColumnDescriptor, Mutation
from ultrafinance.lib.errors import UfException, Errors
import logging
The file pulls in the Thrift transport and protocol primitives (TSocket and TTransport for opening and managing the underlying socket/transport and TBinaryProtocol for serializing RPC calls) so HBaseLib can speak the HBase Thrift API, and it imports the Thrift-generated HBase types module so it has access to HBase-specific type definitions and exceptions; the Hbase Client, ColumnDescriptor and Mutation classes are the Thrift client and schema/operation primitives HBaseLib uses to open a connection, create or modify tables and apply row updates. The ultrafinance.lib.errors UfException and Errors are brought in so HBase-related Thrift errors can be mapped into the project’s unified error model for callers such as HbaseSaver and HBaseDAM to handle schema resets, commits and reads consistently, and logging is available for runtime diagnostics while HBaseLib manages connection lifecycle and table operations.
# file path: ultrafinance/dam/hbaseLib.py
def getRow(self, tName, rowName):
result = self.__client.getRow(tName, rowName)
if not result:
return result
else:
return result[0]
HBaseLib.getRow is the simple accessor that uses the Thrift client initialized by HBaseLib.init to fetch a single row from HBase for a given table name and row key. It asks the underlying client for the row(s) for the provided table and row identifier, then makes a single binary decision: if the client returned no result it propagates that empty response, otherwise it extracts and returns the first row object from the client response. The practical effect is to hide the Thrift client’s list-shaped return value and present callers such as HbaseSaver.read with a single row object (the one HbaseSaver.read then inspects via its columns property), while still allowing callers to detect the “no row found” case by receiving the falsy result. This method therefore centralizes the small normalization between the raw client API and the rest of the persistence layer and contains a single branch for the empty-vs-first-row outcome.
# file path: ultrafinance/lib/errors.py
import traceback
errors.py brings in the standard traceback module so UfException and the centralized Errors mapping can capture and format Python stack traces when wrapping or re-raising errors, allowing PyConfig and other components to raise inspectable exceptions that include a human-readable traceback. You saw the same intent earlier when YahooFinance used traceback to attach stack information to network and parsing failures; importing traceback here centralizes that capability so the pipeline receives consistent, formatted tracebacks whenever configuration or runtime errors are encapsulated in UfException.
# file path: ultrafinance/lib/errors.py
class UfException(Exception):
def __init__(self, error, errorMsg):
super(UfException, self).__init__()
self.__error = error
self.__errorMsg = errorMsg
def __str__(self):
return repr(self.__errorMsg)
def getCode(self):
return self.__error
def getMsg(self):
return "%s: %s" % (self.__errorMsg, traceback.format_exc(5))
UfException is the project’s single, uniform exception type that carries an error code from the Errors mapping together with a human-readable message so every layer of the pipeline can raise and inspect failures in a consistent way. When an error condition is detected (for example PyConfig detecting a missing config file), callers construct a UfException with an Errors entry and a descriptive message; UfException’s constructor calls the base Exception initializer and stores the provided code and message in private attributes for later inspection. Its getCode method returns the stored error code so callers or handlers can programmatically branch on the specific error, while its getMsg method returns the stored message augmented with the most recent traceback snapshot by using traceback.format_exc to give additional diagnostic context. The string form (str) returns a readable representation of the stored message so logging or printing the exception yields a concise human-facing description. In short, UfException centralizes error identity (code) and context (message plus traceback) so the DAMs, TickFeeder, TradingCenter, Account, StateSaver and higher-level strategy/utility code all communicate failures in a consistent, machine-inspectable and debuggable way.
# file path: ultrafinance/lib/errors.py
def __init__(self, error, errorMsg):
super(UfException, self).__init__()
self.__error = error
self.__errorMsg = errorMsg
Within the error-centralization layer of the pipeline, UfException.init is the constructor that captures the canonical error code and the human-readable error message provided by callers (for example PyConfig, TradingCenter, Account, the DAMs and the Excel/Google/Yahoo accessors) and stores them on the exception instance so the rest of the system can inspect them consistently. It invokes the base Exception initializer so the object behaves like a normal Python exception (preserving standard exception semantics and traceability) and then assigns the provided error code to a private field and the provided message to another private field; those stored values are later returned or formatted by UfException.getCode, UfException.getMsg and UfException.str when exception handlers or callers need to log, compare or display the error. The use of name-mangled private attributes keeps the code and message encapsulated on the instance, and the constructor intentionally performs no further validation or formatting — it simply persists the inputs so every module can raise UfException with an Errors entry and a message and rely on a uniform, inspectable error object downstream.
# file path: ultrafinance/lib/plotDateValueDict.py
from matplotlib import pyplot
from datetime import datetime
from ultrafinance.lib.errors import UfException, Errors
import logging
PlotDateValueDict pulls in matplotlib’s pyplot to perform the actual chart rendering work that PlotPortfolio.show relies on for visual output; datetime’s datetime is used to parse and normalize the mapping’s keys so the x-axis is a consistent sequence of date/time values. The module imports UfException and Errors from ultrafinance.lib.errors so any failures during plotting or date conversion are translated into the project’s unified error type and associated error codes, allowing the backtest pipeline (including PlotYearlyOutputer and tests) to handle visualization errors uniformly. logging is available to record diagnostic messages or warnings during rendering and conversion before the code raises a UfException. Earlier we saw urllib and traceback used by YahooFinance for network and error-trace capture; these imports focus on the visualization and error-wrapping responsibilities of PlotDateValueDict.
# file path: ultrafinance/lib/plotDateValueDict.py
def __init__(self, dateValueDict, dateFormat = '%Y%m%d', lowMargin = 0.05, upMargin = 0.05, rightMargin = 0.05, leftMargin = 0.05, betweenMargin = 0.05):
self.dateValueDict = dateValueDict
self.length = len(dateValueDict.keys())
self.lowMargin = lowMargin
self.upMargin = upMargin
self.leftMargin = leftMargin
self.rightMargin = rightMargin
self.dateFormat = dateFormat
self.rect = []
height = float(1 - self.lowMargin - self.upMargin - (self.length - 1) * betweenMargin) / self.length
pre = self.lowMargin
for _ in range(self.length):
self.rect.append([self.leftMargin, pre, 1 - self.leftMargin - self.rightMargin , height])
pre = pre + height + betweenMargin
pyplot.rc('axes', grid = True)
pyplot.rc('grid', color = '0.75', linestyle = '-', linewidth = 0.5)
PlotDateValueDict.init sets up a plotting helper that takes the caller-provided mapping of labels to date/value series and prepares the layout and configuration that PlotDateValueDict.plot will use to render those series for components like PlotYearlyOutputer and unit tests. It stores the incoming dateValueDict and the dateFormat string as instance attributes, records the number of labeled series as length, and keeps the margin parameters (lowMargin, upMargin, leftMargin, rightMargin) so the layout intent is preserved. It then computes the vertical size available to each subplot by subtracting the bottom and top margins and all inter-plot gaps from the total figure height and dividing that remainder by the number of series; the local pre variable is used as a running vertical offset that starts at the bottom margin and is advanced by the computed subplot height plus the betweenMargin on each iteration. For each series it appends a rectangle descriptor (containing left offset, vertical offset, width as the figure width minus left and right margins, and the computed height) to the rect list; those rectangle descriptors are the coordinates that PlotDateValueDict.plot later uses to add axes to the matplotlib figure in the correct stacked positions. Finally, it sets pyplot runtime configuration for axes and grid appearance so the subsequent plot call will use the intended grid style.
# file path: ultrafinance/model/__init__.py
import json
from collections import namedtuple
from ultrafinance.lib.errors import UfException, Errors
The three imports bring in the minimal building blocks this module needs to define, serialize and validate the domain types that sit between the DAMs, the TickFeeder and the persistence layer. The json module is used to convert model instances to and from a wire/storage-friendly representation so Quote, Tick and Order objects can be persisted by savers like HbaseSaver and sent/received by other components; collections.namedtuple is used to create the lightweight, field-named immutable records that implement those domain models (it keeps the model shapes compact and easy to convert into dict-like structures for serialization). ultrafinance.lib.errors exposes UfException and Errors so the model constructors and validation routines can raise consistent, project-wide application errors when business rules are violated (for example when Account, TradingCenter or strategies attempt to create or mutate orders), fitting into the same error-handling conventions used elsewhere in the pipeline.
# file path: ultrafinance/model/__init__.py
class Quote(object):
def __init__(self, time, open, high, low, close, volume, adjClose):
self.time = time
self.open = 0 if ("-" == open) else float(open)
self.high = 0 if ("-" == high) else float(high)
self.low = 0 if ("-" == low) else float(low)
self.close = 0 if ("-" == close) else float(close)
self.volume = int(volume)
self.adjClose = adjClose
def __str__(self):
return json.dumps({"time": self.time,
"open": self.open,
"high": self.high,
"low": self.low,
"close": self.close,
"volume": self.volume,
"adjClose": self.adjClose})
@staticmethod
def fromStr(string):
d = json.loads(string)
return Quote(d['time'], d['open'], d['high'],
d['low'], d['close'], d['volume'], d['adjClose'])
Quote represents the canonical market data record the pipeline passes around: it holds the seven fields time, open, high, low, close, volume and adjClose and is the normalized form that DAMs, the TickFeeder and downstream modules (Account, StockMeasurement, savers) rely on. Its constructor accepts external-stringy inputs (the form ExcelDAM, YahooFinance and GoogleFinance ultimately deliver) and normalizes them so numeric consumers can operate without defensive parsing: any “-” sentinel for price fields is converted to zero, the open/high/low/close fields are coerced to floats, volume is coerced to an integer, and adjClose is preserved as provided (often None). The str method produces a JSON representation of those seven attributes so Quote instances can be written into simple string-based persistence or logs, and the static fromStr method reverses that serialization by parsing JSON and instantiating a fresh Quote, which is what savers and read paths use when restoring rows or cells into runtime objects. Because HBaseDAM, SqlDAM and ExcelDAM all either create Quote instances from row values or persist Quote attribute values, Quote’s normalization is the single place that enforces the business expectation that price and volume fields are numeric, enabling Account.getHoldingValue, StockMeasurement.linearRegression and other calculations to assume consistent types.
# file path: deprecated/example/chinaReturn.py
class ChinaReturn():
def __init__(self, fullTest=False):
self.__workingDir = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
'dataSource',
'CHINA')
self.dayIntervals = [1, 3, 30, 90, 250, 500, 750]
if fullTest:
self.__stocklistFile = os.path.join(self.__workingDir, 'china544.list')
else:
self.__stocklistFile = os.path.join(self.__workingDir, 'china10.list')
def saveHistroyDataIntoExcel(self):
print 'Save HistroyData Into Excel'
storage = HistoricalDataStorage(os.path.join(self.__workingDir, 'china') )
storage.buildExlsFromFile(fileName=self.__stocklistFile, div=5)
def analyze(self):
print 'Start analyzing'
buildBenchmarkValues()
returnRates = [[], [], [], [], [], [], []]
alphas = [[], [], [], [], [], [], []]
relativeReturnRates = [[], [], [], [], [], [], []]
for fileName in filter( lambda f: f.endswith('.xls'), os.listdir(self.__workingDir) ):
excelFile = os.path.join(self.__workingDir, fileName)
sheetNames = ExcelLib.getSheetNames(excelFile)
for sheetName in sheetNames:
with ExcelLib(excelFile) as excel:
excel.openSheet(sheetName=sheetName)
contry = sheetName.split('.')[-1] if len( sheetName.split('.') ) != 1 else 'DEFAULT'
benchmark = benchmarks[contry]
print 'Processing %s with benchmark %s' % (sheetName, benchmark)
for index, duration in enumerate(self.dayIntervals):
data = []
broke = False
for i in range(1, duration + 1):
if broke:
break
try:
values = excel.readRow(i)
for j in range( len(values) ):
values[j] = float(values[j]) if j != 0 else values[j]
data.append( StockDailyType( *values ) )
except Exception:
print 'Analyzing %s break at %d' % (sheetName, i)
broke = True
break
if data:
dateValues = sorted(data, key=itemgetter(0))
stockMeasurement = StockMeasurement(dateValues, benchmark, benchmarkValues[benchmark])
stockMeasurement.linearRegression()
returnRates[index].append( stockMeasurement.returnRate() )
alphas[index].append( stockMeasurement.alpha() )
relativeReturnRates[index].append( stockMeasurement.relativeReturnRate() )
with open(os.path.join(self.__workingDir, 'output.txt'), 'w') as outputFile:
outputReturnRates = map(lambda x: sum(x)/len(x), returnRates)
outputAlphas = map(lambda x: sum(x)/len(x), alphas)
outputRelativeReturnRates = map(lambda x: sum(x)/len(x), relativeReturnRates)
print "Days since going public %s" % self.dayIntervals
print "returnRates: %s" % outputReturnRates
print "alphas: %s" % outputAlphas
print "relativeReturnRates: %s" % outputRelativeReturnRates
outputFile.write("outputReturnRates %s\n" % outputReturnRates)
outputFile.write("outputAlphas %s\n" % outputAlphas)
outputFile.write("outputRelativeReturnRates %s\n" % outputRelativeReturnRates)
outputFile.write("returnRates %s\n" % returnRates)
outputFile.write("alphas %s\n" % alphas)
outputFile.write("relativeReturnRates %s\n" % relativeReturnRates)
ChinaReturn is a deprecated example module that orchestrates assembling historical stock data, building benchmark series, persisting worksheets and then reading those sheets back to compute summary statistics used by the rest of the pipeline. Its constructor determines a working directory under the project’s dataSource/CHINA subtree and selects a small or full stock list file depending on the fullTest flag; it also defines the list of dayIntervals used by the analysis. saveHistroyDataIntoExcel instantiates HistoricalDataStorage with an output prefix inside that working directory and delegates to HistoricalDataStorage.buildExlsFromFile with the chosen stock list and a division factor of five, which causes HistoricalDataStorage to read the symbol list and split work across multiple Excel workbooks while fetching per-stock historical series via YahooFinance (which in turn uses the previously examined YahooFinance.__request) and writing worksheets. analyze begins by invoking buildBenchmarkValues to populate benchmarkValues by calling YahooFinance.getHistoricalPrices for each benchmark, then allocates three parallel arrays of lists to collect return rates, alphas and relative return rates for each interval. It then walks every .xls file found in the working directory, obtains sheet names via ExcelLib.getSheetNames, and for each sheet opens the workbook with ExcelLib as a context manager and opens the specific sheet; it extracts a country/benchmark key from the sheet name and looks up the benchmark symbol. For each configured duration it attempts to read sequential rows from the sheet (using ExcelLib.readRow, which delegates to ExcelRead.readRow), converts numeric columns into floats and builds a list of StockDailyType records; a try/except with a broke flag stops the inner loop early if a row read fails or data is missing. If any daily records were collected, the code sorts them by date, constructs a StockMeasurement with the aligned dateValues and the benchmark (passing any prebuilt benchmarkValues entry), calls StockMeasurement.linearRegression to align dates and compute regression parameters, and then appends returnRate, alpha and relativeReturnRate to the corresponding lists for that day interval. After all files and sheets are processed the module computes per-interval averages by taking the mean of each list and writes those averaged metrics plus the raw lists into an output.txt file. In short, ChinaReturn coordinates HistoricalDataStorage to create Excel inputs, uses ExcelLib/ExcelRead to pull per-stock time series back out, feeds those into StockMeasurement (which may again use YahooFinance to fill missing benchmark data), and aggregates regression-derived performance measures across the defined dayIntervals.
# file path: deprecated/ultrafinance/lib/dataType.py
from collections import namedtuple
The import pulls namedtuple from Python’s collections library so dataType can declare lightweight, named record types like Quote and the L2/L3 markers without defining full classes. Using namedtuple gives each domain marker a fixed set of named fields, tuple-like immutability and compact memory layout, which makes these records easy for the DAMs, crawlers and the TickFeeder to construct, unpack, compare and serialize when they normalize and pass market data through the pipeline. This keeps the module’s role as the centralized place for domain type declarations simple and dependency-light while providing convenient helpers (like converting instances to dictionaries) that the persistence and downstream modules rely on.
# file path: deprecated/ultrafinance/lib/dataType.py
StockDailyType = namedtuple('StockDailyType', 'date, open, high, low, close, volume, adjClose')
StockDailyType is a module-level domain marker defined as a namedtuple that represents a single daily market record with the fields date, open, high, low, close, volume and adjClose; it serves as the lightweight, immutable container that DAMs and crawlers like GoogleFinance, YahooFinance, ExcelDataFeeder and IndexDataFeeder produce and consume when normalizing daily price data. Remember the Quote we looked at earlier? StockDailyType mirrors that canonical shape (using date instead of time) so downstream code can treat historical price rows uniformly; for example, GoogleFinance.getHistoricalPrices and YahooFinance.getHistoricalPrices parse CSV responses into sequences of these records, HistoricalDataStorage.__buildExl iterates those sequences and accesses their attributes to write Excel sheets, and StockMeasurement expects lists of such objects when computing return rates and regressions. The choice of a namedtuple gives simple attribute access, tuple semantics (easy iteration/serialization) and immutability so the record can be passed through the TickFeeder, persistence and measurement layers without extra boilerplate or mutation.
# file path: ultrafinance/dam/excelLib.py
from xlrd import open_workbook
from xlwt import Workbook
from ultrafinance.lib.errors import UfException, Errors
from os import path
import logging
ExcelLib pulls in open_workbook from xlrd so it can read existing Excel workbooks and Workbook from xlwt so it can create and write new workbooks; together these two libraries provide the low-level Excel I/O that the adapter uses to implement its context-managed workbook lifecycle and the high-level sheet/row/column/cell operations that ExcelRead, ExcelWrite and ExcelOpertion rely on. UfException and Errors are imported from ultrafinance.lib.errors so ExcelLib can surface and standardize Excel-related failures as the pipeline’s unified exception types, allowing ExcelDAM, ExcelDataFeeder and IndexDataFeeder to handle errors consistently. The path utility from os is used to resolve and validate filesystem locations before attempting reads or writes, and logging is used to record operational events and diagnostic information during workbook lifecycle actions, which helps trace data flow and failures as ticks and sheets move through the ultrafinance-master_cleaned pipeline.
# file path: ultrafinance/dam/excelLib.py
class ExcelLib(object):
READ_MODE = 'r'
WRITE_MODE = 'w'
def __init__(self, fileName = None, mode = READ_MODE):
if ExcelLib.READ_MODE == mode:
self.__operation = ExcelRead(fileName)
elif ExcelLib.WRITE_MODE == mode:
self.__operation = ExcelWrite(fileName)
else:
raise UfException(Errors.INVALID_EXCEL_MODE,
"Invalid operation mode, only %s and %s are accepted"\
% (ExcelLib.READ_MODE, ExcelLib.WRITE_MODE))
def __enter__(self):
self.__operation.pre()
return self
def __exit__(self, type, value, traceback):
self.__operation.post()
return
def openSheet(self, name):
self.__operation.openSheet(name)
def readRow(self, row, startCol=0, endCol=-1):
return self.__operation.readRow(row, startCol, endCol)
def readCol(self, col, startRow=0, endRow=-1):
return self.__operation.readCol(col, startRow, endRow)
def readCell(self, row, col):
return self.__operation.readCell(row, col)
def writeRow(self, row, values):
self.__operation.writeRow(row, values)
def writeCell(self, row, col, value):
self.__operation.writeCell(row, col, value)
def getOperation(self):
return self.__operation
ExcelLib sits in the data-access layer as the thin, context-managed adapter that higher-level DAMs and feeders use to interact with Excel workbooks in a uniform way. Remember the ExcelRead and ExcelWrite classes we looked at earlier? ExcelLib’s constructor selects one of those concrete operation objects based on a READ or WRITE mode and raises a UfException if an invalid mode is provided, so callers never have to instantiate ExcelRead/ExcelWrite directly. When used as a context manager, ExcelLib ensures the operation lifecycle hooks are invoked: enter forwards to the operation’s pre method and returns the ExcelLib wrapper, and exit forwards to the operation’s post method so any necessary persistence or cleanup (for example, ExcelWrite writing the workbook to disk) always runs. For sheet and cell access ExcelLib simply delegates: openSheet, readRow, readCol, readCell, writeRow and writeCell forward to the underlying operation object and return or propagate its results or exceptions; getOperation exposes the underlying operation when callers (tests or callers like ExcelDataFeeder and TradeInWeek) need access to operation-specific APIs such as sheet count or sheet names. Conceptually ExcelLib acts as a facade/adapter that centralizes error handling and lifecycle management for Excel interactions so ExcelDAM, ExcelDataFeeder, IndexDataFeeder and other modules can read columns and rows and then transform those lists into domain objects like Quote, Tick and DateValueType without duplicating workbook setup or teardown logic. The control flow is straightforward: choose the correct Excel operation at construction, open and prepare the operation on enter, delegate all read/write calls to that operation during use, and ensure post-processing happens on exit; any invalid mode or underlying errors are surfaced as UfException for consistent upstream handling.
# file path: ultrafinance/dam/excelLib.py
def openSheet(self, name):
self.__operation.openSheet(name)
ExcelLib.openSheet simply forwards the requested sheet name down to whatever concrete ExcelOpertion instance ExcelLib.init selected (either ExcelRead or ExcelWrite) so the workbook-level operation can select or create the named sheet. Because ExcelLib is the context-managed adapter that callers like ExcelDataFeeder, TradeInWeek and ChinaReturn use inside a with block, calling openSheet here causes the underlying operation to set its internal current sheet (for ExcelRead, locating the named sheet in the opened workbook; for ExcelWrite, creating or retrieving and storing the writable sheet in its sheet dictionary). There is no branching in ExcelLib.openSheet itself; any validation or UfException error reporting comes from ExcelOpertion.openSheet or the concrete implementation beneath it, and subsequent calls to readRow/readCol/readCell or writeRow/writeCell rely on the sheet state that the delegated openSheet established.
# file path: ultrafinance/dam/excelLib.py
def openSheet(self, name):
raise UfException(Errors.UNDEFINED_METHOD, "openSheet function is not defined")
ExcelOpertion.openSheet is the abstract entry point that enforces the contract for opening a worksheet in the Excel adapter layer; when ExcelLib.openSheet delegates to the operation, a concrete operation class (ExcelRead or ExcelWrite) is expected to provide the real behavior. If no subclass has overridden openSheet, ExcelOpertion.openSheet signals that by raising a UfException with the UNDEFINED_METHOD code, which lets callers upstream detect a missing implementation in a uniform way. Conceptually this implements an interface/abstract-base pattern: ExcelLib relies on polymorphic dispatch to either ExcelRead.openSheet or ExcelWrite.openSheet for the actual work, and the base method exists only to make the omission explicit and consistent with the project’s centralized error handling via UfException. In normal data flow the happy path is that ExcelRead or ExcelWrite handle sheet selection and lifecycle; the error path triggered here prevents silent failures by producing a standardized UfException carrying a machine-readable error code and a descriptive message that other components in the pipeline can catch and log.
# file path: ultrafinance/dam/excelLib.py
def readRow(self, row, startCol=0, endCol=-1):
return self.__operation.readRow(row, startCol, endCol)
ExcelLib.readRow acts as the thin delegating entry point that higher-level modules use to fetch a horizontal record from an open workbook: it receives a row index plus an optional start and end column and forwards that request to the concrete operation object that ExcelLib.init created (either ExcelRead or ExcelWrite) so the actual I/O lives in the ExcelOpertion subclasses. Because ExcelLib is used inside the context-managed lifecycle where enter invokes the operation’s pre step and exit invokes post, calls to readRow happen with the workbook prepared and will return whatever the underlying operation’s readRow implementation produces (typically a list of cell values for the requested column slice, with the endCol default meaning “to the end”). The indirection also centralizes behavior and error semantics: if the concrete operation has not implemented readRow, the base ExcelOpertion.readRow surfaces a UfException with an UNDEFINED_METHOD code, so callers such as ExcelDAM.__readData, ExcelDataFeeder, IndexDataFeeder, TradeInWeek and the unit tests can rely on a consistent contract and error handling while converting those returned rows into Quote or Tick objects downstream.
# file path: ultrafinance/dam/excelLib.py
def readRow(self, row, startCol=0, endCol=-1):
raise UfException(Errors.UNDEFINED_METHOD, "readRow function is not defined")
ExcelOpertion.readRow is the abstract contract in the Excel adapter layer that defines how a caller asks the adapter to return a worksheet row; it does not attempt to read anything itself but immediately signals that the method is undefined by raising a UfException with the Errors.UNDEFINED_METHOD code and a message stating the readRow operation is not implemented. In the pipeline this method is the polymorphic hook that ExcelLib.readRow calls on the operation instance that ExcelLib creates at construction time, so when the operation is a concrete subclass like ExcelRead the subclass will provide the real row-reading behavior and return the list of cell values back up to ExcelLib and callers such as ExcelDAM.__readData. The explicit UfException here implements the abstract-base/template pattern for ExcelOpertion: it enforces a uniform interface and centralized error semantics for all Excel operation implementations and makes the error path explicit whenever a subclass has not provided the readRow implementation, while the happy path is handled by the overridden method in ExcelRead.
# file path: ultrafinance/dam/excelLib.py
def readCol(self, col, startRow=0, endRow=-1):
return self.__operation.readCol(col, startRow, endRow)
ExcelLib.readCol is a thin delegator that exposes a simple, mode-agnostic way for callers to fetch a vertical slice from a workbook: given a column index and optional start/end row bounds it forwards the request to the underlying Excel operation instance that ExcelLib.init created (either ExcelRead for read mode or ExcelWrite for write mode). Because ExcelOpertion defines a placeholder readCol that raises UfException, the actual work is performed by the concrete implementation (ExcelRead implements column extraction from an open workbook), and any missing implementation surfaces as a UfException so callers get a consistent error contract. In practice callers such as ExcelDAM.__findRange, ExcelDataFeeder.execute, IndexDataFeeder.execute and TradeInWeek.oneStrategy invoke readCol to pull date, open, close and other series; ExcelLib.readCol simply returns whatever the operation returns so the rest of the pipeline receives a list of column values ready for range searches, row-by-row construction (DateValueType/Quote/Tick) or further processing. Because ExcelLib is used as a context manager, pre/post lifecycle hooks on the operation are handled outside of readCol, leaving readCol focused solely on delegating the column-read request and preserving the unified interface the data-access layer relies on.
# file path: ultrafinance/dam/excelLib.py
def readCol(self, col, startRow=0, endRow=-1):
raise UfException(Errors.UNDEFINED_METHOD, "readCol function is not defined")
ExcelOpertion.readCol is the abstract declaration in the Excel adapter layer that specifies the contract for reading a column from a workbook; it expects a column identifier plus optional start and end row boundaries but does not implement any retrieval logic itself. When ExcelLib.readCol delegates to the operation object, the runtime dispatch will invoke ExcelOpertion.readCol on the base type only if a concrete subclass did not override it; in that case the method immediately signals a consistent failure by throwing a UfException with the Errors.UNDEFINED_METHOD code and an explanatory message. Conceptually this enforces a polymorphic contract: concrete operation implementations such as ExcelRead provide the real column-reading behavior and are invoked in the normal data-flow path, while the base method acts as a guard that makes missing implementations explicit to callers like ExcelDAM, ExcelDataFeeder and IndexDataFeeder. From a control-flow perspective there is no happy-path return from the base method — it always raises the UfException so the caller must rely on a subclass override to produce the expected list of cell values. The exception carries the standardized error code and message via UfException so the rest of the pipeline receives a uniform error object when a readCol implementation is absent.
# file path: ultrafinance/dam/excelLib.py
def readCell(self, row, col):
return self.__operation.readCell(row, col)
ExcelLib.readCell is a thin, mode-agnostic delegator that takes a row and column request from higher-level callers and forwards it to the concrete Excel operation instance that ExcelLib.init created (either ExcelRead or ExcelWrite). In the runtime flow a module such as ExcelDAM or ExcelDataFeeder will enter the ExcelLib context (which invokes the operation.pre hook), call readCell to fetch a single cell, and then exit the context (which invokes operation.post) so lifecycle concerns and any cleanup are handled centrally. The actual I/O work happens inside the operation’s readCell implementation: when the operation is ExcelRead that method ensures a sheet is open, validates indices and returns the cell value; if no concrete implementation exists ExcelOpertion.readCell will surface a UfException indicating the method is undefined. By delegating in this way, readCell keeps caller code simple and consistent with the existing readRow/readCol pattern and leverages the shared UfException-based error handling that the Excel adapter layer uses across read and write modes.
# file path: ultrafinance/dam/excelLib.py
def readCell(self, row, col):
raise UfException(Errors.UNDEFINED_METHOD, "readCell function is not defined")
ExcelOpertion.readCell is the abstract declaration in the Excel adapter layer that enforces that concrete operation classes must provide real cell-level read logic. When a higher-level caller routes a readCell request through ExcelLib.readCell, that call is forwarded to the operation instance created by ExcelLib; if that instance is still the abstract ExcelOpertion rather than a concrete implementation like ExcelRead, ExcelOpertion.readCell immediately signals a contract violation by raising a UfException initialized with the UNDEFINED_METHOD error code and an explanatory message. Functionally that means there is no happy-path behavior inside ExcelOpertion.readCell itself — the expected data flow is: ExcelLib.readCell -> concrete operation.readCell -> return a cell value. The control-flow enforced here is a guard: any attempt to use the base operation without an override results in a consistent UfException, which the rest of the pipeline (ExcelLib, ExcelDAM, feeders, etc.) can catch and handle uniformly. Conceptually this follows the abstract-base-class/template pattern used elsewhere in the adapter layer where unimplemented methods raise the same UfException to make missing implementations explicit and consistent.
# file path: ultrafinance/dam/excelLib.py
def writeRow(self, row, values):
self.__operation.writeRow(row, values)
ExcelLib.writeRow is a minimal delegating entry point that hands a horizontal write request to the concrete operation instance stored on the ExcelLib object. The operation instance is chosen by ExcelLib.init (either ExcelRead or ExcelWrite), so writeRow does not implement any I/O itself but forwards the requested row index and the list of values to the underlying ExcelOpertion implementation. In the normal write-mode path that ExcelDAM.__writeData uses, that call will reach ExcelWrite.writeRow which places the values into the active worksheet at the specified row; ExcelLib is typically used inside its context manager so operation.pre has already prepared the in-memory workbook and operation.post will be called on exit to persist the workbook to disk. If ExcelLib was created in read mode, the delegation hits the abstract ExcelOpertion.writeRow and a UfException is raised to indicate the undefined write operation, so the single-line forward enforces mode semantics and centralizes error behavior while leaving the concrete cell-level behavior to the ExcelWrite implementation.
# file path: ultrafinance/dam/excelLib.py
def writeRow(self, sheetName, row, values):
raise UfException(Errors.UNDEFINED_METHOD, "writeRow function is not defined")
ExcelOpertion.writeRow defines the abstract contract for writing a horizontal record into a worksheet and, when invoked on the base operation class, immediately signals that no concrete implementation exists by raising a UfException with the UNDEFINED_METHOD error. In the pipeline this method is the target that ExcelLib.writeRow delegates to after ExcelLib.init selected either ExcelRead or ExcelWrite as the concrete Excel operation; if ExcelWrite is in use it provides the real behavior for writing rows, but if a caller ends up hitting the base ExcelOpertion.writeRow the raised UfException (which captures an error code and message in its constructor) acts as a clear, centralized failure signal so higher layers like ExcelLib, ExcelDAM or any feeder/consumer will see a consistent error indicating the writeRow contract was not implemented. The pattern here is the abstract-base/interface style common to the adapter layer: declare the operation signature (sheetName, row, values) at the base and force subclasses to supply the actual I/O, otherwise produce a uniform UfException so the rest of the system can handle the missing implementation predictably.
# file path: ultrafinance/dam/excelLib.py
def writeCell(self, row, col, value):
self.__operation.writeCell(row, col, value)
ExcelLib.writeCell acts as a thin delegating method in the Excel adapter: when a caller asks to place a value into a specific row and column, writeCell forwards that request to the concrete operation instance stored on ExcelLib.__operation. That operation instance is created in ExcelLib.init and will be either ExcelWrite for write-mode workbooks or ExcelRead for read-mode workbooks, so the actual cell mutation logic lives in the ExcelWrite implementation while the abstract ExcelOpertion base declares the contract and raises a UfException for undefined implementations. Because ExcelLib is typically used as a context manager, the write lifecycle surrounding writeCell is managed by ExcelLib.enter and ExcelLib.exit, which call the underlying operation.pre and operation.post so writes can be prepared and then persisted (ExcelWrite.post handles the final workbook output). In practice higher-level components like ExcelDAM and test code use ExcelLib.writeRow/writeCell to build files: openSheet must be called earlier to select the target sheet and then writeCell delegates the cell-level write into ExcelWrite’s sheet bookkeeping; if the adapter was constructed in read mode, the delegation will hit the abstract contract and surface a UfException indicating the write operation is not implemented.
# file path: ultrafinance/dam/excelLib.py
def writeCell(self, sheetName, row, col, value):
raise UfException(Errors.UNDEFINED_METHOD, "readCell function is not defined")
ExcelOpertion.writeCell is the abstract contract in the Excel adapter layer that enforces the expectation that any concrete operation class must provide a concrete implementation for cell-level writes. When a caller such as ExcelLib.writeCell forwards a write request to the operation instance, control should normally land in a concrete subclass like ExcelWrite which implements the actual workbook mutation; if instead the base-class writeCell is invoked, it aborts by raising a UfException flagged with the UNDEFINED_METHOD error so the caller receives a normalized, descriptive failure. This follows the same pattern you saw with ExcelOpertion.openSheet and the other abstract ExcelOpertion methods: the base class does not attempt any I/O but explicitly signals “not implemented” using UfException, and UfException.init records the error code and message so the rest of the pipeline can handle the failure consistently. In terms of data flow, the happy path is ExcelLib delegating to ExcelWrite.writeCell to persist the value into the workbook; the error path is hitting ExcelOpertion.writeCell which immediately raises the UfException to halt the operation and bubble a consistent error up to callers like ExcelDAM or higher-level components.
# file path: ultrafinance/dam/excelLib.py
def post(self):
return
ExcelOpertion.post is the lifecycle hook that ExcelLib.exit invokes to let the concrete operation finish up when the context manager closes. By default it does nothing and returns immediately, which gives a safe no-op implementation so read-mode users (via ExcelRead) don’t need any cleanup behavior while higher-level callers like ExcelDAM, ExcelDataFeeder, IndexDataFeeder, TradeInWeek and ChinaReturn can always call the same exit path without special-casing modes. Conceptually it implements the hook part of a Template Method pattern: the base class provides the empty post hook and concrete subclasses override it when they need to perform end-of-life work (for example, ExcelWrite will override post to persist the in-memory workbook to disk or perform any required post actions). Because openSheet, readRow and readCol already delegate to the operation instance, post centralizes the finalization step for both read and write flows so the context-managed workbook lifecycle in ExcelLib remains consistent across modes.
# file path: ultrafinance/dam/excelLib.py
def pre(self):
return
ExcelOpertion.pre is the empty, overridable setup hook that the context manager calls when a caller enters an ExcelLib context: ExcelLib.enter invokes pre on whatever concrete operation instance ExcelLib.init created (either ExcelRead or ExcelWrite). In the base ExcelOpertion class pre performs no work and immediately returns, so there are no side effects or data flows at that layer and the context entry proceeds transparently. The purpose of having pre defined as a no-op is to provide a uniform lifecycle hook so ExcelLib can remain mode-agnostic while giving concrete subclasses a place to perform any necessary preparation (for example, file validation, sheet creation or other internal state setup that ExcelRead or ExcelWrite might implement) before the workbook is used; subclasses can override pre to do that work and surface problems via UfException if needed. Together with the corresponding post hook, pre implements a simple template-style lifecycle boundary that separates setup/teardown responsibilities from the higher-level ExcelLib context management.
# file path: ultrafinance/dam/excelLib.py
class ExcelWrite(ExcelOpertion):
def __init__(self, fileName):
if path.exists(fileName):
raise UfException(Errors.FILE_EXIST, "File already exist: %s" % fileName)
self.__fileName = fileName
self.__workbook = Workbook()
self.__sheetNameDict ={}
self.__sheet = None
def openSheet(self, name):
if name not in self.__sheetNameDict:
sheet = self.__workbook.add_sheet(name)
self.__sheetNameDict[name] = sheet
self.__sheet = self.__sheetNameDict[name]
def __getSheet(self, name):
if not self.sheetExsit(name):
raise UfException(Errors.SHEET_NAME_INVALID, "Can't find a sheet named %s" % name)
return self.__sheetNameDict[name]
def sheetExsit(self, name):
return name in self.__sheetNameDict
def writeCell(self, row, col, value):
if self.__sheet is None:
self.openSheet(super(ExcelWrite, self).DEFAULT_SHEET)
self.__sheet.write(row, col, value)
def writeRow(self, row, values):
if self.__sheet is None:
self.openSheet(super(ExcelWrite, self).DEFAULT_SHEET)
for index, value in enumerate(values):
self.__sheet.write(row, index, value)
def post(self):
self.__workbook.save(self.__fileName)
ExcelWrite is the write-mode concrete implementation of the ExcelOpertion adapter that ExcelLib picks when the data-access layer needs to create a new workbook and write rows and cells into it. On construction ExcelWrite enforces the pipeline’s safety rule that we never clobber an existing file by checking for the target file and raising a UfException with the FILE_EXIST error if it already exists; it then creates an in-memory workbook object, a map to track sheet name to sheet object, and a pointer for the currently active sheet. When ExcelLib.openSheet delegates to ExcelOpertion.openSheet (as explained earlier), ExcelWrite either creates a new worksheet and records it in the sheet name dictionary or switches the current sheet pointer to an existing one; sheets are created lazily so callers can start writing without manually pre-creating every sheet. The private helper __getSheet validates a requested sheet name and raises a UfException with SHEET_NAME_INVALID if the name is unknown, while sheetExsit performs the simple membership check against the internal sheet map. writeCell and writeRow are the row/column writing paths used by higher-level modules (for example ExcelDAM.__writeData via ExcelLib.writeRow); both ensure a worksheet is active by opening the default sheet defined by the parent ExcelOpertion when no sheet has been selected, then write values into the in-memory sheet sequentially (writeRow iterates the provided values and writes each cell). Persistence happens only when ExcelLib’s context manager exits: ExcelLib.exit calls post on the operation, which in ExcelWrite saves the in-memory workbook to the filename supplied at construction, completing the write-phase of the DAM pipeline. In pattern terms, ExcelWrite implements the ExcelOpertion contract for write operations, centralizes the same UfException-based error semantics used across the project, and keeps writes buffered in memory until the context-managed post step commits the file to disk.
# file path: ultrafinance/dam/excelLib.py
def sheetExsit(self, name):
return name in self.__sheetNameDict
ExcelWrite.sheetExsit is a very small, focused predicate that answers whether a named worksheet has already been registered in the writer instance’s in-memory sheet registry that ExcelWrite.init creates. When a caller hands a sheet name into the Excel adapter layer, sheetExsit performs a constant-time membership check against the internal __sheetNameDict populated by ExcelWrite.openSheet (recall ExcelOpertion.openSheet is the abstract contract and ExcelWrite.openSheet is the concrete method that adds entries). Because it only inspects that dictionary, it does no disk I/O and simply returns true or false, letting callers decide the next step: for example ExcelWrite.__getSheet queries sheetExsit and raises a UfException with Errors.SHEET_NAME_INVALID when the name is absent, while other flows may call openSheet to create a new sheet when the name is missing. In short, sheetExsit centralizes the existence-check logic used by the Excel adapter so higher-level components in the pipeline can consistently validate access to worksheets before reading, writing, or persisting workbooks.
# file path: ultrafinance/dam/excelLib.py
class ExcelRead(ExcelOpertion):
def __init__(self, fileName):
if not path.exists(fileName):
raise UfException(Errors.FILE_NOT_EXIST, "File doesn't exist: %s" % fileName)
self.__book = open_workbook(fileName)
self.__sheet = None
def openSheet(self, name):
self.__sheet = self.__book.sheet_by_name(name)
def getTotalSheetNumber(self):
return self.__book.nsheets
def getSheetNames(self):
return self.__book.sheet_names()
def readRow(self, row, startCol=0, endCol=-1):
if self.__sheet is None:
self.openSheet(super(ExcelRead, self).DEFAULT_SHEET)
if abs(row) >= self.__sheet.nrows:
raise UfException(Errors.INDEX_RANGE_ERROR,
"Excellib.readRow: row number too big: row %s, max %s" % (row, self.__sheet.nrows) )
if max(abs(startCol), abs(endCol)) > self.__sheet.ncols:
raise UfException(Errors.INDEX_RANGE_ERROR,
"Excellib.readRow: col number too big: col %s, max %s" % (max(abs(startCol), abs(endCol)), self.sheet.ncols) )
if -1 == endCol:
endCol = self.__sheet.ncols
return [self.readCell(row, i) for i in range(startCol, endCol)]
def readCol(self, col, startRow=0, endRow=-1):
if self.__sheet is None:
self.openSheet(super(ExcelRead, self).DEFAULT_SHEET)
if abs(col) > self.__sheet.ncols:
raise UfException(Errors.INDEX_RANGE_ERROR,
"Excellib.readCol: col number too big: col %s, max %s" % (col, self.sheet.ncols) )
if max(abs(startRow), abs(endRow)) > self.__sheet.nrows:
raise UfException(Errors.INDEX_RANGE_ERROR,
"Excellib.readCol: row number too big: row %s, max %s" % (max(abs(startRow), abs(endRow)), self.sheet.nrows) )
if -1 == endRow:
endRow = self.__sheet.nrows
return [self.readCell(i, col) for i in range(startRow, endRow)]
def readCell(self, row, col):
try:
if self.__sheet is None:
self.openSheet(super(ExcelRead, self).DEFAULT_SHEET)
return self.__sheet.cell(row, col).value
except BaseException as excp:
raise UfException(Errors.UNKNOWN_ERROR, "Unknown Error in Excellib.readCell %s" % excp)
ExcelRead is the read-mode concrete operation that ExcelLib uses to present a safe, high-level interface over an opened Excel workbook so higher layers like ExcelDAM, ExcelDataFeeder and IndexDataFeeder can request sheets, rows, columns and individual cell values and get consistent errors back. On construction ExcelRead verifies the file exists and then opens the workbook with the underlying workbook loader, keeping an internal book handle and a None current sheet until a sheet is selected; if the file is missing it raises UfException with the FILE_NOT_EXIST code so callers see the same error model used across the system. openSheet selects a worksheet by name and stores it as the current sheet so subsequent reads operate against it; getTotalSheetNumber and getSheetNames merely expose the workbook metadata so callers can discover available sheets. readRow and readCol are the row/column iterators: they ensure a sheet is opened (falling back to the DEFAULT_SHEET defined on the ExcelOpertion base type), validate the requested indices using absolute-value checks so negative indices are handled against the sheet size, treat a sentinel end index as “to the end”, and then build a list of values by delegating each element to readCell. readCell performs the single-cell access against the current sheet and returns the raw cell value, and any unexpected error during cell access is caught and rethrown as a UfException with the UNKNOWN_ERROR code so callers never see raw low-level exceptions. Because ExcelRead inherits the lifecycle hooks from ExcelOpertion, ExcelLib’s context-manager entry and exit will invoke the pre/post hooks around usage; ExcelRead’s responsibilities are focused on safe opening, sheet selection, bounds-checked row/column slicing, and centralizing error translation to UfException so the rest of the pipeline can rely on predictable behavior when reading Excel sources.
# file path: deprecated/ultrafinance/lib/stockMeasurement.py
import copy
import numpy
from ultrafinance.lib.yahooFinance import YahooFinance
from ultrafinance.lib.errors import UfException, Errors
import logging
The file pulls in a handful of utilities that support StockMeasurement’s role of fetching a historical price series and turning it into statistical and regression metrics for downstream processors. The copy module is brought in so the code can clone price arrays or metadata safely before doing in-place transformations or temporary derivatives (preserving the original series that other parts of the pipeline might still reference). Numpy provides the numerical foundation for the mean, standard deviation, returns calculations and the vectorized work behind the linear-regression, alpha and beta computations. YahooFinance is the data-access class the measurement uses to source historical prices from the market feeds the project supports; that external fetch is the entry point for the time-series that StockMeasurement normalizes and analyzes (contrast this with the Excel adapter classes we covered earlier, which handle workbook I/O rather than market data). UfException and Errors are the project’s domain-specific error types used to signal and classify failure modes like missing or malformed price data or a failed fetch, letting callers such as AvgDivProcessor and ChinaReturn react appropriately. Finally, logging is used to record operational information and error details during fetch and computation so the pipeline can be observed and debugged at runtime.
# file path: deprecated/ultrafinance/lib/stockMeasurement.py
class StockMeasurement():
def __init__(self, dateValues, benchmark='^GSPC', benchmarkValues=None):
self.__dateValues = dateValues
self.__benchmark = benchmark
self.__benchmarkValues = copy.deepcopy(benchmarkValues)
self.__alpha = None
self.__beta = None
self.__regressioned = False
def mean(self):
return numpy.mean([float(dateValues.close) for dateValues in self.__dateValues], axis=0)
def std(self):
return numpy.std([float(dateValues.close) for dateValues in self.__dateValues], axis=0)
def linearRegression(self):
if self.__regressioned:
return
if not self.__benchmarkValues:
self.__benchmarkValues = YahooFinance().getHistoricalPrices(self.__benchmark, self.__dateValues[0].date, self.__dateValues[-1].date)
tradeSuspended = False
if 0 in map(lambda x: float(x.adjClose), self.__dateValues):
tradeSuspended = True
dateSet = set([dateValue.date for dateValue in self.__dateValues]) & set([dateValue.date for dateValue in self.__benchmarkValues])
self.__dateValues = filter(lambda dateValue: dateValue.date in dateSet, self.__dateValues)
self.__benchmarkValues = filter(lambda dateValue: dateValue.date in dateSet, self.__benchmarkValues)
if len(self.__dateValues) <= 1 or tradeSuspended:
msg = "Not enought dateValues" if len(self.__dateValues) <= 1 else "trade suspended"
LOG.debug(msg)
self.__beta = 0
self.__alpha = 0
self.__regressioned = True
return
try:
x = [float(self.__benchmarkValues[index + 1].adjClose)/float(self.__benchmarkValues[index].adjClose) for index in range(len(self.__benchmarkValues) - 1)]
y = [float(self.__dateValues[index + 1].adjClose)/float(self.__dateValues[index].adjClose) for index in range(len(self.__dateValues) - 1)]
(self.__beta, self.__alpha) = numpy.polyfit(x, y, 1)
self.__regressioned = True
except BaseException as excep:
raise UfException(Errors.UNKNOWN_ERROR, "stockMeasurement.linearRegression got unknown error %s" % excep)
def marketReturnRate(self):
if not self.__regressioned:
self.linearRegression()
return (float(self.__benchmarkValues[-1].adjClose) - float(self.__benchmarkValues[0].adjClose)) / float(self.__benchmarkValues[0].adjClose) \
if self.__benchmarkValues and float(self.__benchmarkValues[0].adjClose) \
else 0
def returnRate(self):
return (float(self.__dateValues[-1].adjClose) - float(self.__dateValues[0].adjClose)) / float(self.__dateValues[0].adjClose) \
if self.__dateValues and float(self.__dateValues[0].adjClose) \
else 0
def relativeReturnRate(self):
return self.returnRate() - self.marketReturnRate()
def alpha(self):
if not self.__regressioned:
self.linearRegression()
return self.__alpha
def beta(self):
if not self.__regressioned:
self.linearRegression()
return self.__beta
StockMeasurement is the mid-level helper that takes a series of daily price records for one symbol and produces descriptive statistics and a simple regression-based attribution to a market benchmark so downstream modules like AvgDivProcessor and ChinaReturn can consume mean, volatility, alpha and beta. On instantiation StockMeasurement stores the supplied dateValues, a benchmark symbol (defaulting to the S&P proxy) and an optional copy of benchmarkValues, and initializes internal alpha, beta and a regression-completed flag. The mean and std methods compute the simple average and standard deviation over the close prices in dateValues by coercing values to floats and delegating to numpy. The core work happens in linearRegression: if regression has not already run it will ensure benchmark history is available (using the provided benchmarkValues or fetching historical prices from YahooFinance for the benchmark over the same date range), detect impossible trade periods by looking for zero adjusted-close entries, align the two series to the intersection of trading dates so the asset and benchmark are matched timestamp-by-timestamp, and short-circuit to set alpha and beta to zero when there is insufficient matched data or trading was suspended. For the normal path it builds series of period returns from consecutive adjusted-close ratios for benchmark (x) and asset (y) and fits a first-order polynomial with numpy to obtain beta and alpha, storing those results and flipping the regression flag. Any unexpected failure during fitting is re-thrown as a UfException. marketReturnRate and returnRate compute cumulative return across the matched period from adjusted-close endpoints (marketReturnRate will trigger linearRegression if needed), relativeReturnRate returns the asset return net of the market return, and alpha and beta accessors also ensure regression has been performed before returning the stored values. The method mutates instance attributes: it may replace the internal dateValues and benchmarkValues with their date-aligned subsets and permanently caches the regression results so repeated calls from AvgDivProcessor.execute or ChinaReturn.analyze are fast and consistent.
# file path: deprecated/ultrafinance/lib/stockMeasurement.py
def mean(self):
return numpy.mean([float(dateValues.close) for dateValues in self.__dateValues], axis=0)
StockMeasurement.mean computes the simple arithmetic average of the close prices for the time series that StockMeasurement.init stored in self.__dateValues: it walks the dateValues array, converts each entry’s close to a floating-point number, and delegates the aggregation to numpy.mean with axis set to collapse the sequence into a single average value. In the pipeline this average is the per-stock summary metric that AvgDivProcessor.execute reads after instantiating StockMeasurement and combines with the count of days, the standard deviation, and the regression-derived alpha and beta for downstream persistence and reporting. mean does not fetch benchmark data or trigger regression work itself; those concerns are handled by linearRegression, marketReturnRate, returnRate, alpha and beta when called elsewhere, so mean’s job is strictly to produce a numeric average of the provided close-price series for downstream consumers.
# file path: deprecated/ultrafinance/lib/stockMeasurement.py
def std(self):
return numpy.std([float(dateValues.close) for dateValues in self.__dateValues], axis=0)
StockMeasurement.std produces the per-stock volatility number that the pipeline and downstream processors use as the standard-deviation of the series of closing prices. It takes the historical tick list that StockMeasurement.init stored in the instance, walks that sequence and converts each DateValue object’s close field to a floating-point number, then asks NumPy to compute the standard deviation across that vector (axis zero) and returns that result. In the pipeline this value is consumed by AvgDivProcessor.execute when it constructs a StockMeasurement for a symbol and includes the returned standard deviation in the data payload that gets persisted or further processed, and it provides the scalar that metrics such as BasicMetric leverage (via their own stddev/sharpe calculations) to quantify volatility for risk and performance calculations.
# file path: deprecated/ultrafinance/lib/stockMeasurement.py
def linearRegression(self):
if self.__regressioned:
return
if not self.__benchmarkValues:
self.__benchmarkValues = YahooFinance().getHistoricalPrices(self.__benchmark, self.__dateValues[0].date, self.__dateValues[-1].date)
tradeSuspended = False
if 0 in map(lambda x: float(x.adjClose), self.__dateValues):
tradeSuspended = True
dateSet = set([dateValue.date for dateValue in self.__dateValues]) & set([dateValue.date for dateValue in self.__benchmarkValues])
self.__dateValues = filter(lambda dateValue: dateValue.date in dateSet, self.__dateValues)
self.__benchmarkValues = filter(lambda dateValue: dateValue.date in dateSet, self.__benchmarkValues)
if len(self.__dateValues) <= 1 or tradeSuspended:
msg = "Not enought dateValues" if len(self.__dateValues) <= 1 else "trade suspended"
LOG.debug(msg)
self.__beta = 0
self.__alpha = 0
self.__regressioned = True
return
try:
x = [float(self.__benchmarkValues[index + 1].adjClose)/float(self.__benchmarkValues[index].adjClose) for index in range(len(self.__benchmarkValues) - 1)]
y = [float(self.__dateValues[index + 1].adjClose)/float(self.__dateValues[index].adjClose) for index in range(len(self.__dateValues) - 1)]
(self.__beta, self.__alpha) = numpy.polyfit(x, y, 1)
self.__regressioned = True
except BaseException as excep:
raise UfException(Errors.UNKNOWN_ERROR, "stockMeasurement.linearRegression got unknown error %s" % excep)
StockMeasurement.linearRegression is the on-demand routine that aligns a stock’s series with its benchmark, guards against bad inputs, then computes a simple linear regression of the stock’s successive return ratios against the benchmark’s successive return ratios to produce beta and alpha for downstream modules like AvgDivProcessor and ChinaReturn. It first returns immediately if the instance already did regression (the cached __regressioned flag implements lazy evaluation/caching). If benchmark historical values were not supplied at construction, it asks YahooFinance.getHistoricalPrices for the benchmark over the date span defined by the first and last entries of self.__dateValues. It then checks for suspended trading by scanning adjusted-close values and flags trade suspension if any adjusted close equals zero. To ensure valid pairwise regression points it intersects the set of dates present in the stock series and the benchmark series and filters both series to that intersection so every point has a matching benchmark date. If, after that alignment, there is one or fewer dates or trading is suspended, it logs a debug message, sets alpha and beta to zero, marks regression done, and returns early. In the normal path it builds two arrays of successive return ratios — one for the benchmark (x) and one for the stock (y) — by taking each consecutive pair’s adjusted-close quotient, resulting in arrays of length n−1, and then calls numpy.polyfit to fit a first-degree line of y on x; the returned slope and intercept are stored into __beta and __alpha respectively and __regressioned is set to true. Any unexpected error during that process is caught and rethrown as a UfException with an unknown-error code. The method therefore updates instance attributes (__benchmarkValues and __dateValues may be overwritten by the alignment step, and __alpha, __beta, __regressioned are set) so that marketReturnRate, alpha, and beta can lazily call linearRegression and then return cached results.
# file path: deprecated/ultrafinance/lib/googleFinance.py
import urllib2
from BeautifulSoup import BeautifulSoup
import traceback
from operator import itemgetter
from ultrafinance.lib.util import convertGoogCSVDate, findPatthen
from ultrafinance.lib.dataType import StockDailyType
from ultrafinance.lib.errors import UfException, Errors
import copy
from ultrafinance.lib.util import string2EpochTime
import re
import logging
The imports set up the plumbing that lets GoogleFinance act as a Google-sourced data access module: urllib2 provides the HTTP client used to fetch pages and CSV endpoints from Google Finance, and BeautifulSoup is used to parse the returned HTML when the class needs to scrape symbol lists or financials. traceback is pulled in so the module can capture and include stack traces when turning low-level errors into the project’s normalized error objects; UfException and Errors are the domain-level error types the rest of the pipeline expects, so GoogleFinance wraps and rethrows problems using those. convertGoogCSVDate and string2EpochTime are the utility converters that normalize Google’s date strings into the epoch/timestamp and internal date formats the backtest TickFeeder and HistoricalDataStorage consume, while findPatthen and the regular expression library are used to extract specific tokens and values from messy HTML or CSV payloads. StockDailyType supplies the canonical data structure for a single daily record so parsed rows can be handed downstream in a uniform shape. operator.itemgetter is available for selecting and sorting fields (for example ordering records by date), copy is used when creating independent transformed copies of parsed records, and logging is used to emit runtime diagnostics and informational messages during fetch/parse flows. Together these imports support fetching, parsing, converting, error-normalizing, and packaging Google Finance data into the pipeline the project expects.
# file path: deprecated/ultrafinance/lib/googleFinance.py
def getHistoricalPrices(self, symbol, startdate, enddate):
try:
url = 'http://www.google.com/finance/historical?q=%s&startdate=%s&enddate=%s&output=csv' % (symbol, startdate, enddate)
try:
page = self.__request(url)
except UfException as ufExcep:
if Errors.NETWORK_400_ERROR == ufExcep.getCode:
raise UfException(Errors.STOCK_SYMBOL_ERROR, "Can find data for stock %s, symbol error?" % symbol)
raise ufExcep
days = page.readlines()
values = [day.split(',') for day in days]
data = []
for value in values[1:]:
date = convertGoogCSVDate(value[0])
data.append(StockDailyType(date, value[1], value[2], value[3], value[4], value[5], None))
dateValues = sorted(data, key=itemgetter(0))
return dateValues
except BaseException:
raise UfException(Errors.UNKNOWN_ERROR, "Unknown Error in GoogleFinance.getHistoricalPrices %s" % traceback.format_exc())
GoogleFinance.getHistoricalPrices is the adapter method that fetches historical price CSVs from Google Finance, normalizes the raw CSV into the pipeline’s canonical daily record type, and returns those records in chronological order so HistoricalDataStorage, TickFeeder and downstream analytics can consume them. It constructs a Google Finance CSV endpoint using the provided symbol, startdate and enddate and then delegates the HTTP fetch to GoogleFinance.__request; if __request raises a UfException and that exception represents a 400-level response the method translates that into a stock-symbol-specific UfException to signal a bad symbol, otherwise it propagates the original UfException. Once it has the response it reads all lines, splits each CSV row into fields, skips the header row, and for each data row converts Google’s date string into the codebase’s compact YYYYMMDD date format by calling convertGoogCSVDate, then wraps the date, open, high, low, close and volume fields into a StockDailyType record (leaving adjClose as None). Because Google CSVs are typically returned in reverse chronological order, the method sorts the list by the record’s date field into ascending order before returning it. Any unexpected error during the whole process is caught and re-raised as a UfException with an UNKNOWN_ERROR code and a traceback so callers see a normalized error type.
# file path: deprecated/ultrafinance/lib/googleFinance.py
def __request(self, url):
try:
return urllib2.urlopen(url)
except IOError:
raise UfException(Errors.NETWORK_ERROR, "Can't connect to Google server")
except urllib2.HTTPError:
raise UfException(Errors.NETWORK_400_ERROR, "400 error when connect to Google server")
except Exception:
raise UfException(Errors.UNKNOWN_ERROR, "Unknown Error in GoogleFinance.__request %s" % traceback.format_exc())
GoogleFinance.__request is the small network adapter that accepts a URL string, issues an HTTP GET via urllib2.urlopen and hands back the open response stream for the higher layers to parse (CSV lines, HTML for BeautifulSoup, or the raw bytes callers expect). Its primary responsibility is to normalize network-level failures into the project’s domain exception type, UfException, so the rest of the data-access pipeline gets consistent error objects instead of raw urllib2 exceptions. Concretely, it attempts to open the URL and return the response; if an IOError occurs it converts that into a UfException tagged as a network connectivity error, if an urllib2.HTTPError occurs it converts that into a UfException tagged as a 400-class network error, and any other unexpected exception is wrapped into a UfException flagged as an unknown error with the traceback attached for diagnostics. Callers such as GoogleFinance.getHistoricalPrices, getAll and getFinancials rely on this behavior: they consume the returned response stream on the happy path, and on error they inspect the UfException (via its code/getter methods) to decide higher-level actions (for example, treating a 400-class error as a possible bad symbol). This method therefore sits at the midpoint between raw HTTP I/O and the DAM’s parsing logic, centralizing the mapping from transport exceptions into the framework’s uniform error model.
# file path: ultrafinance/lib/util.py
import sys
from bs4 import BeautifulSoup
from datetime import datetime
from datetime import timedelta
from datetime import date
import time
from time import gmtime, strftime
import logging
sys is pulled in to let utility routines interact with the Python runtime — for example adjusting import resolution, inspecting or manipulating module state, or exiting early when dynamic class loading (used by PluginManager and dynamic importer helpers) needs to fail fast. BeautifulSoup provides the HTML parsing engine utilities rely on when GoogleFinance and other crawlers hand back raw pages so util can extract and normalize text and links for downstream parsers. The datetime family — datetime, timedelta and date — supplies the calendar and interval primitives the CSV/date parsing and TickFeeder normalization code uses to turn string timestamps into proper date/time objects, perform arithmetic windows, and canonicalize dates for storage and comparison. The time module together with the gmtime and strftime helpers is used for lower-level epoch handling and producing formatted timestamp strings for filenames, logs, and tick alignment. Finally, logging is imported so the helper functions can emit consistent diagnostics and error messages across the pipeline (similar to the lifecycle tracing we saw around ExcelOpertion and its concrete implementations), allowing callers like SqlDAM, PluginManager and the backtest runtime to record what the utilities are doing at runtime.
# file path: ultrafinance/lib/util.py
def convertGoogCSVDate(googCSVDate):
d = str(datetime.strptime(googCSVDate, googCSVDateformat).date())
return d.replace("-", "")
convertGoogCSVDate is a small normalization helper in ultrafinance/lib/util.py that takes a raw date token coming from a Google Finance CSV and turns it into the pipeline’s canonical date string (YYYYMMDD). It does this by parsing the incoming string according to the googCSVDateformat into a datetime date, converting that date to the usual ISO-style date text and then stripping the hyphens so the result matches other modules’ expectations. GoogleFinance.getQuotes and GoogleFinance.getHistoricalPrices call this helper while they transform Google CSV rows into the pipeline’s daily record types (Quote, Tick, StockDailyType), and downstream utilities like string2EpochTime and StockMeasurement expect the same compact YYYYMMDD representation; any parsing failure will propagate up to the caller.
# file path: deprecated/ultrafinance/lib/stockMeasurement.py
def marketReturnRate(self):
if not self.__regressioned:
self.linearRegression()
return (float(self.__benchmarkValues[-1].adjClose) - float(self.__benchmarkValues[0].adjClose)) / float(self.__benchmarkValues[0].adjClose) \
if self.__benchmarkValues and float(self.__benchmarkValues[0].adjClose) \
else 0
marketReturnRate is a small accessor-style routine on StockMeasurement that guarantees the benchmark alignment work has been done by invoking linearRegression if the instance has not yet been regressioned; remember linearRegression (which we covered earlier) may populate or refresh the internal benchmark series by calling YahooFinance to fetch historical prices and sets the __regressioned flag as a side effect. After ensuring those preconditions, marketReturnRate computes the benchmark’s cumulative return over the measurement window by taking the adjusted close at the end of the benchmark series, subtracting the adjusted close at the start, and dividing that difference by the starting adjusted close to produce a simple percentage return. It protects against missing data and divide-by-zero by returning zero when there are no benchmark values or the first adjusted close is falsy, and it coerces the price values to floating point to perform the arithmetic. The numeric result is then consumed by relativeReturnRate (which subtracts this market return from the stock’s own return) and by higher-level processors like AvgDivProcessor and ChinaReturn that need the market return for relative performance and regression-based metrics.
# file path: deprecated/ultrafinance/lib/stockMeasurement.py
def returnRate(self):
return (float(self.__dateValues[-1].adjClose) - float(self.__dateValues[0].adjClose)) / float(self.__dateValues[0].adjClose) \
if self.__dateValues and float(self.__dateValues[0].adjClose) \
else 0
StockMeasurement.returnRate computes the total empirical return for the stock over the measurement window and is the simple percentage change the rest of the pipeline uses to quantify performance. It reads the list of normalized daily records that StockMeasurement.init stores in self.__dateValues (these come from the GoogleFinance/YahooFinance adapters and are provided in chronological order), pulls the adjusted close from the earliest record and the adjusted close from the latest record, converts them to numeric form, and returns the change between them divided by the starting adjusted close. The method guards against missing data and division-by-zero by returning zero when the date list is empty or the starting adjusted close is zero. Using adjusted close ensures the computed return reflects corporate actions (dividends, splits) so downstream consumers such as AvgDivProcessor and ChinaReturn — and the relativeReturnRate method that subtracts marketReturnRate — get a total-return measure consistent with how benchmarkReturn is computed in marketReturnRate.
# file path: deprecated/ultrafinance/lib/historicalDataStorage.py
from datetime import date
from xlwt import Workbook
import time
import traceback
from ultrafinance.lib.errors import UfException, Errors
from ultrafinance.lib.yahooFinance import YahooFinance
import logging
HistoricalDataStorage pulls date from the datetime module to build and format the date ranges and timestamps it needs when requesting historical series and naming the output sheets, and it uses Workbook from xlwt to create the Excel workbooks that buildExls and buildExlsFromFile will populate and persist. The time module is available for simple timing tasks such as rate-limiting requests or stamping generated files, while traceback is used to capture and include stack traces when wrapping unexpected failures into the project’s error types. UfException and Errors from ultrafinance.lib.errors are the canonical error classes HistoricalDataStorage raises so callers like ChinaReturn.saveHistroyDataIntoExcel can handle failures consistently. YahooFinance provides the Yahoo-sourced data access adapter that HistoricalDataStorage calls to fetch normalized historical price series (used alongside the already-explained GoogleFinance.getHistoricalPrices), and logging is used throughout to record progress, informational messages, and error conditions during fetch-and-persist operations.
# file path: deprecated/ultrafinance/lib/historicalDataStorage.py
class HistoricalDataStorage():
def __init__(self, outputPrefix, startDate = '1900-01-01', endDate = date.today()):
self.__outputPrefix = outputPrefix
self.__startDate = startDate
self.__endDate = endDate
def buildExls(self, stocks, div=1):
print "BuildExls %s, div %s" % (stocks, div)
if div < 1:
raise UfException(Errors.INDEX_RANGE_ERROR, "div need to be at least 1, %s are given" % div)
for i in range(div):
workbook = Workbook()
stocksToProcess = stocks[i * len(stocks)/div: (i+1) * len(stocks)/div]
for stock in stocksToProcess:
try:
self.__buildExl(stock, workbook)
except Exception:
print "failed buildExl for stock %s: %s" % (stock, traceback.print_exc())
time.sleep(2)
fileName = '%s%d.xls' % (self.__outputPrefix, i)
print "Saved %s to %s" % (stocksToProcess, fileName)
workbook.save(fileName)
def buildExlsFromFile(self, fileName, div=1):
print "buildExlsFromFile %s, div %s" % (fileName, div)
f = open(fileName)
lines = [line.rstrip() for line in f]
self.buildExls(lines, div)
def __buildExl(self, stock, workbook):
try:
ws = workbook.add_sheet(stock)
yahooFinance = YahooFinance()
allData = yahooFinance.getHistoricalPrices(stock, self.__startDate, self.__endDate)
for col, field in enumerate(['date', 'open', 'high', 'low', 'close', 'volume', 'adjClose']):
ws.write(0, col, field)
for row, data in enumerate(allData):
for col, field in enumerate(['date', 'open', 'high', 'low', 'close', 'volume', 'adjClose']):
ws.write(row+1, col, getattr(data, field) )
except UfException as excp:
raise excp
except Exception:
raise UfException(Errors.UNKNOWN_ERROR, "historicalStorage.__buildExl got unknown error %s"
% traceback.print_exc())
HistoricalDataStorage is the mid-level utility that ChinaReturn uses to fetch and persist historical price series into Excel workbooks so the backtest pipeline has ready-made per-stock histories for downstream analytics like StockMeasurement.linearRegression. Its constructor records the target output file prefix and the start/end date bounds for all fetches. ChinaReturn.saveHistroyDataIntoExcel constructs a HistoricalDataStorage and calls buildExlsFromFile, which opens the provided stock list file, strips lines into a list of symbols, and delegates to buildExls. buildExls accepts a list of symbols and a division count and first validates that the division parameter is at least one, raising a UfException with Errors.INDEX_RANGE_ERROR on bad input. It then iterates over division slices of the symbol list (splitting the list into roughly equal chunks), creates a new Excel Workbook for each chunk, and processes each symbol in that chunk by calling the private __buildExl. buildExls prints progress messages, sleeps for two seconds between symbols to throttle requests, and saves each completed workbook to disk using the output prefix plus the chunk index. __buildExl is responsible for the per-symbol work: it creates a worksheet named after the symbol, calls YahooFinance.getHistoricalPrices to retrieve the canonical StockDailyType series between the stored start and end dates (the project’s network adapters such as GoogleFinance.__request and YahooFinance.getHistoricalPrices were covered earlier), writes a header row for the fields date, open, high, low, close, volume and adjClose, and then writes each record row from the returned series into the sheet. __buildExl propagates UfException unchanged, and converts any other unexpected exception into a UfException with Errors.UNKNOWN_ERROR and the traceback so callers see a uniform error type; buildExls, however, catches exceptions from __buildExl, logs a failure message with the traceback, and continues processing remaining symbols so a single symbol failure does not abort the whole batch. Overall, HistoricalDataStorage glues the DAM-level historical fetchers to the Excel persistence layer, producing .xls files that ChinaReturn and the rest of the backtest pipeline can consume.
# file path: deprecated/ultrafinance/lib/historicalDataStorage.py
def buildExls(self, stocks, div=1):
print "BuildExls %s, div %s" % (stocks, div)
if div < 1:
raise UfException(Errors.INDEX_RANGE_ERROR, "div need to be at least 1, %s are given" % div)
for i in range(div):
workbook = Workbook()
stocksToProcess = stocks[i * len(stocks)/div: (i+1) * len(stocks)/div]
for stock in stocksToProcess:
try:
self.__buildExl(stock, workbook)
except Exception:
print "failed buildExl for stock %s: %s" % (stock, traceback.print_exc())
time.sleep(2)
fileName = '%s%d.xls' % (self.__outputPrefix, i)
print "Saved %s to %s" % (stocksToProcess, fileName)
workbook.save(fileName)
HistoricalDataStorage.buildExls orchestrates taking a list of symbols and writing their historical series into one or more Excel workbooks so downstream analysis can consume a persisted snapshot. It first validates the div argument and raises UfException if div is less than one, then it partitions the stocks list into div roughly equal slices and processes each slice as a separate workbook. For each partition it creates a new Workbook, iterates the symbols in that partition and calls HistoricalDataStorage.__buildExl to add a worksheet containing the symbol’s historical rows (__buildExl in turn fetches series via the YahooFinance historical fetch routine you’ve already seen). Each per-symbol call is wrapped in a broad exception handler that logs the failure and continues with the remaining symbols, and the loop pauses between symbols to avoid aggressive request bursts. After finishing a partition it composes a filename using the instance’s output prefix set in HistoricalDataStorage.init and the partition index, saves the workbook to disk, and moves on to the next partition. The typical data flow is that buildExls is invoked from buildExlsFromFile (which reads a stocklist file) as in ChinaReturn.saveHistroyDataIntoExcel, producing Excel files that feed later stages of the pipeline.
# file path: deprecated/ultrafinance/lib/historicalDataStorage.py
def __buildExl(self, stock, workbook):
try:
ws = workbook.add_sheet(stock)
yahooFinance = YahooFinance()
allData = yahooFinance.getHistoricalPrices(stock, self.__startDate, self.__endDate)
for col, field in enumerate(['date', 'open', 'high', 'low', 'close', 'volume', 'adjClose']):
ws.write(0, col, field)
for row, data in enumerate(allData):
for col, field in enumerate(['date', 'open', 'high', 'low', 'close', 'volume', 'adjClose']):
ws.write(row+1, col, getattr(data, field) )
except UfException as excp:
raise excp
except Exception:
raise UfException(Errors.UNKNOWN_ERROR, "historicalStorage.__buildExl got unknown error %s"
% traceback.print_exc())
HistoricalDataStorage.__buildExl is the routine that turns a single symbol’s historical series into a sheet inside the Excel workbook produced by buildExls. It starts by asking the Workbook to create a new worksheet named for the stock, then instantiates YahooFinance and requests the symbol’s normalized daily records for the date window stored on the HistoricalDataStorage instance (self.__startDate to self.__endDate). Those returned records are the pipeline’s canonical daily type (the StockDailyType / Quote style objects used elsewhere), so __buildExl writes a header row of the canonical fields (date, open, high, low, close, volume, adjClose) and then iterates the returned sequence, writing each record’s field values into successive rows of the worksheet. For error handling it forwards any UfException it receives unchanged so upstream callers can react to known error conditions, and it converts any other unexpected exception into a UfException tagged with Errors.UNKNOWN_ERROR while including the traceback for diagnostics. In the larger pipeline this method is the point where externally sourced, normalized historical ticks from YahooFinance are persisted into an Excel sheet so later stages (TickFeeder, strategies, or offline analysis) can load consistent historical series.
# file path: deprecated/ultrafinance/lib/historicalDataStorage.py
def buildExlsFromFile(self, fileName, div=1):
print "buildExlsFromFile %s, div %s" % (fileName, div)
f = open(fileName)
lines = [line.rstrip() for line in f]
self.buildExls(lines, div)
buildExlsFromFile is a thin adaptor that turns a disk-resident stock list into the in-memory list that HistoricalDataStorage.buildExls expects and then delegates the heavy lifting to buildExls. When ChinaReturn.saveHistroyDataIntoExcel invokes storage.buildExlsFromFile it prints a status line, opens the supplied file, reads each line trimming trailing whitespace so you end up with a list of stock symbols, and then calls buildExls with that list and the div parameter. The subsequent work happens in buildExls: div is validated (and will raise a UfException if invalid), the symbol list is partitioned into div slices, a workbook is created for each slice, and for each symbol __buildExl is invoked to fetch canonical daily records (using the finance data adapter) and write date, open, high, low, close, volume and adjusted close into an Excel sheet. The net effect of buildExlsFromFile is to bootstrap that pipeline from a plain-text list file and trigger the file I/O and workbook creation performed by buildExls and __buildExl so the historical series are persisted for later consumers such as the TickFeeder and StockMeasurement routines (including the linearRegression path you saw earlier).
# file path: deprecated/ultrafinance/lib/stockMeasurement.py
def relativeReturnRate(self):
return self.returnRate() - self.marketReturnRate()
StockMeasurement.relativeReturnRate produces the stock’s excess return over its chosen benchmark by subtracting the benchmark return from the stock return; conceptually it asks StockMeasurement.returnRate what the cumulative return of the stock series is and asks StockMeasurement.marketReturnRate what the cumulative return of the benchmark is, then returns the numeric difference. returnRate computes a simple cumulative return using the adjusted-close on the first and last dates stored on the StockMeasurement instance and returns zero if the series is empty or the initial adjusted-close is zero, so relativeReturnRate inherits that guard behavior. marketReturnRate ensures the regression step has been performed (invoking StockMeasurement.linearRegression if necessary), which may fetch and align benchmarkValues with the stock’s dateValues and set up alpha/beta, then computes the benchmark’s cumulative return with the same emptiness/zero-price protection. The outcome is a single scalar that represents how much the stock out- or under-performed its market benchmark over the measurement window; ChinaReturn.analyze collects these relative return scalars into its relativeReturnRates buckets for downstream aggregation and Excel output.
# file path: deprecated/ultrafinance/lib/stockMeasurement.py
def alpha(self):
if not self.__regressioned:
self.linearRegression()
return self.__alpha
StockMeasurement.alpha is the lazy accessor for the regression intercept (alpha) of a single stock’s time series versus its benchmark. When a caller like AvgDivProcessor.execute or ChinaReturn.analyze asks for alpha, the method first checks the internal __regressioned flag; if the regression has not yet been performed it invokes StockMeasurement.linearRegression to align dates, fetch any missing benchmark prices, compute the linear regression of stock returns against benchmark returns, and write the computed __alpha and __beta back onto the instance (linearRegression also flips __regressioned to true as a side effect). After ensuring regression results exist, alpha simply returns the stored __alpha value. This lazy-evaluation pattern prevents unnecessary network requests and computation until a downstream module actually needs the alpha value, and guarantees callers always receive a regression result that was produced with the same aligned date set and data preprocessing performed inside linearRegression.
# file path: ultrafinance/dam/excelLib.py
def getSheetNames(self):
return self.__book.sheet_names()
ExcelRead.getSheetNames is the simple accessor on the Excel adapter that returns the workbook’s sheet name list; it reads from the workbook object that ExcelRead.init created when it opened the file and then delegates to the underlying workbook API to produce the sequence of sheet names. In the data-access layer role of ExcelLib, getSheetNames lets higher layers—ExcelLib and its callers such as testExcelLib.testReadExcel and ChinaReturn.analyze—discover which sheets to open and then drive the subsequent calls to openSheet, readRow, readCol and readCell. Because ExcelRead.init already validates the file and raises UfException on missing files, getSheetNames contains no additional branching or error handling itself: it assumes a valid book and simply adapts the underlying workbook’s sheet enumeration into the library’s high-level operation.
# file path: deprecated/example/chinaReturn.py
def __init__(self, fullTest=False):
self.__workingDir = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
'dataSource',
'CHINA')
self.dayIntervals = [1, 3, 30, 90, 250, 500, 750]
if fullTest:
self.__stocklistFile = os.path.join(self.__workingDir, 'china544.list')
else:
self.__stocklistFile = os.path.join(self.__workingDir, 'china10.list')
ChinaReturn.init constructs the object by setting the instance attribute __workingDir to the project’s dataSource CHINA folder (so subsequent steps know where to read and write the sheet files), initializing dayIntervals with the multi-horizon window sizes that analyze will use to bucket return, alpha and relative-return calculations, and selecting which disk-resident stock list to work from by assigning __stocklistFile based on the fullTest flag — the smaller sample list when fullTest is false and the full list when fullTest is true. That single conditional is the only branching here; everything else is straightforward initialization of these instance attributes which saveHistroyDataIntoExcel will pass into HistoricalDataStorage.buildExlsFromFile to persist historical series and which analyze will later read with ExcelLib/ExcelRead to compute returnRates, alphas and relativeReturnRates across the configured dayIntervals.
# file path: deprecated/example/chinaReturn.py
def saveHistroyDataIntoExcel(self):
print 'Save HistroyData Into Excel'
storage = HistoricalDataStorage(os.path.join(self.__workingDir, 'china') )
storage.buildExlsFromFile(fileName=self.__stocklistFile, div=5)
saveHistroyDataIntoExcel is the ChinaReturn method that triggers the persistence step of the data acquisition layer so the analysis stage can operate on disk-resident snapshots. When invoked it logs a brief status message, instantiates a HistoricalDataStorage object pointed at a subfolder named china under ChinaReturn’s working directory, and then delegates to HistoricalDataStorage.buildExlsFromFile with the stock list file that ChinaReturn.init selected and a div value of five. HistoricalDataStorage.buildExlsFromFile reads the stock list from disk and hands the in-memory list to HistoricalDataStorage.buildExls, which enforces the div guard (div must be at least one) and then slices the symbol list into five partitions, creating one workbook per partition. For each symbol in each partition HistoricalDataStorage.__buildExl is called to fetch historical prices (via YahooFinance/GoogleFinance) and write a sheet containing the time-series columns such as date, open, high, low, close, volume and adjusted close. The net effect is a set of Excel workbooks written under the storage prefix containing the historical series for the requested symbols; this produces the files that ChinaReturn.analyze will later discover, load with ExcelLib, and run benchmark-relative computations on. The call chain therefore moves data from a disk-resident symbol list through HistoricalDataStorage into persisted Excel sheets, with network requests and file I/O as side effects and with a div-based partitioning to spread symbols across multiple workbooks.
# file path: deprecated/example/chinaReturn.py
def analyze(self):
print 'Start analyzing'
buildBenchmarkValues()
returnRates = [[], [], [], [], [], [], []]
alphas = [[], [], [], [], [], [], []]
relativeReturnRates = [[], [], [], [], [], [], []]
for fileName in filter( lambda f: f.endswith('.xls'), os.listdir(self.__workingDir) ):
excelFile = os.path.join(self.__workingDir, fileName)
sheetNames = ExcelLib.getSheetNames(excelFile)
for sheetName in sheetNames:
with ExcelLib(excelFile) as excel:
excel.openSheet(sheetName=sheetName)
contry = sheetName.split('.')[-1] if len( sheetName.split('.') ) != 1 else 'DEFAULT'
benchmark = benchmarks[contry]
print 'Processing %s with benchmark %s' % (sheetName, benchmark)
for index, duration in enumerate(self.dayIntervals):
data = []
broke = False
for i in range(1, duration + 1):
if broke:
break
try:
values = excel.readRow(i)
for j in range( len(values) ):
values[j] = float(values[j]) if j != 0 else values[j]
data.append( StockDailyType( *values ) )
except Exception:
print 'Analyzing %s break at %d' % (sheetName, i)
broke = True
break
if data:
dateValues = sorted(data, key=itemgetter(0))
stockMeasurement = StockMeasurement(dateValues, benchmark, benchmarkValues[benchmark])
stockMeasurement.linearRegression()
returnRates[index].append( stockMeasurement.returnRate() )
alphas[index].append( stockMeasurement.alpha() )
relativeReturnRates[index].append( stockMeasurement.relativeReturnRate() )
with open(os.path.join(self.__workingDir, 'output.txt'), 'w') as outputFile:
outputReturnRates = map(lambda x: sum(x)/len(x), returnRates)
outputAlphas = map(lambda x: sum(x)/len(x), alphas)
outputRelativeReturnRates = map(lambda x: sum(x)/len(x), relativeReturnRates)
print "Days since going public %s" % self.dayIntervals
print "returnRates: %s" % outputReturnRates
print "alphas: %s" % outputAlphas
print "relativeReturnRates: %s" % outputRelativeReturnRates
outputFile.write("outputReturnRates %s\n" % outputReturnRates)
outputFile.write("outputAlphas %s\n" % outputAlphas)
outputFile.write("outputRelativeReturnRates %s\n" % outputRelativeReturnRates)
outputFile.write("returnRates %s\n" % returnRates)
outputFile.write("alphas %s\n" % alphas)
outputFile.write("relativeReturnRates %s\n" % relativeReturnRates)
ChinaReturn.analyze is the orchestration routine that turns the persisted Excel snapshots produced earlier into aggregated performance metrics across a set of holding intervals. It first triggers buildBenchmarkValues so the global benchmarkValues store is populated via YahooFinance (network I/O), then prepares three parallel collectors for raw returnRates, alphas and relativeReturnRates indexed by the dayIntervals list defined on ChinaReturn. The method walks the working directory looking for Excel workbooks, asks ExcelLib for each workbook’s sheet names, and then opens each sheet with ExcelLib as a context manager so the ExcelRead/ExcelWrite abstraction is used for file I/O. For each sheet it determines which benchmark to use by parsing the sheet name and looking it up in the benchmarks mapping, logs that choice, and then iterates over the configured durations (the seven dayIntervals). For each duration it attempts to read up to that many rows from the sheet via ExcelLib.readRow, converting numeric strings into floats and assembling a list of StockDailyType entries; if any read fails an exception is caught, the loop for that duration is aborted and processing moves on. When at least one row was read, the collected rows are sorted by date and handed into a StockMeasurement constructed with the chosen benchmark and the prebuilt benchmarkValues slice; StockMeasurement.linearRegression is invoked (which ensures benchmark series alignment and computes regression stats), and the routine appends the stock’s cumulative return, its regression alpha and its relative return versus market into the corresponding collector for that duration index. After all files, sheets and durations are processed, analyze computes the per-interval averages from the collected lists, prints summary lines showing the dayIntervals and the averaged returnRates, alphas and relativeReturnRates, and writes both the aggregated summaries and the raw per-stock lists into an output.txt file in the working directory. The method therefore connects the persisted historical data layer (produced by HistoricalDataStorage.buildExlsFromFile as invoked by saveHistroyDataIntoExcel) to the metric computation layer (StockMeasurement), handling I/O, per-sheet benchmark selection, per-duration aggregation, and final persistence of the analysis results.
# file path: deprecated/example/chinaReturn.py
if __name__ == '__main__':
app = ChinaReturn()
app.saveHistroyDataIntoExcel()
app.analyze()
When the module is executed as a program, the runtime creates a ChinaReturn instance which triggers ChinaReturn.init to assemble the symbol list, build benchmark time series (via buildBenchmarkValues and the YahooFinance data adapter) and set up any internal state needed for the persistence and analysis steps. After construction it runs saveHistroyDataIntoExcel, which delegates to HistoricalDataStorage.buildExlsFromFile to write the assembled historical and benchmark series into Excel workbooks so downstream tools can consume a snapshot. Finally it calls analyze, which reads the generated sheets using ExcelRead and ExcelLib, computes per-stock metrics (for example using StockMeasurement.relativeReturnRate and StockMeasurement.alpha against the benchmark) and produces the reported analysis. The conditional guard ensures these orchestration steps execute only when the file is run directly and not when the module is imported elsewhere in the pipeline.
# file path: ultrafinance/dam/yahooFinance.py
LOG = logging.getLogger()
At the top of the module a module-level logger named LOG is created by invoking the logging facility to obtain the logger instance that the rest of YahooFinance will use. Because YahooFinance is responsible for making HTTP calls, consulting HbaseSaver.read via its helper routines and surfacing getQuotes or raising UfException on failures, LOG provides a single, consistent channel for emitting debug, info and error messages from those operations so they integrate with the application’s global logging configuration and handlers. Having a module-level logger means functions like __request, getAll and getQuotes can reference the same logger instance to record request lifecycle events, HBase lookup results and error conditions in a predictable, centralized way.
# file path: ultrafinance/dam/yahooFinance.py
def getAll(self, symbol):
values = self.__request(symbol, 'l1c1va2xj1b4j4dyekjm3m4rr5p5p6s7').split(',')
data = {}
data['price'] = values[0]
data['change'] = values[1]
data['volume'] = values[2]
data['avg_daily_volume'] = values[3]
data['stock_exchange'] = values[4]
data['market_cap'] = values[5]
data['book_value'] = values[6]
data['ebitda'] = values[7]
data['dividend_per_share'] = values[8]
data['dividend_yield'] = values[9]
data['earnings_per_share'] = values[10]
data['52_week_high'] = values[11]
data['52_week_low'] = values[12]
data['50day_moving_avg'] = values[13]
data['200day_moving_avg'] = values[14]
data['price_earnings_ratio'] = values[15]
data['price_earnings_growth_ratio'] = values[16]
data['price_sales_ratio'] = values[17]
data['price_book_ratio'] = values[18]
data['short_ratio'] = values[19]
return data
YahooFinance.getAll is the thin adapter that turns a single symbol into a labeled map of quote-level statistics for
# file path: ultrafinance/dam/yahooFinance.py
def getQuotes(self, symbol, start, end):
try:
start = str(start).replace('-', '')
end = str(end).replace('-', '')
url = 'http://ichart.yahoo.com/table.csv?s=%s&' % symbol + \
'd=%s&' % str(int(end[4:6]) - 1) + \
'e=%s&' % str(int(end[6:8])) + \
'f=%s&' % str(int(end[0:4])) + \
'g=d&' + \
'a=%s&' % str(int(start[4:6]) - 1) + \
'b=%s&' % str(int(start[6:8])) + \
'c=%s&' % str(int(start[0:4])) + \
'ignore=.csv'
days = urllib.urlopen(url).readlines()
values = [day[:-2].split(',') for day in days]
data = []
for value in values[1:]:
data.append(Quote(value[0], value[1], value[2], value[3], value[4], value[5], value[6]))
dateValues = sorted(data, key = lambda q: q.time)
return dateValues
except IOError:
raise UfException(Errors.NETWORK_ERROR, "Can't connect to Yahoo server")
except BaseException:
raise UfException(Errors.UNKNOWN_ERROR, "Unknown Error in YahooFinance.getHistoricalPrices %s" % traceback.format_exc())
YahooFinance.getQuotes implements the Yahoo-specific retrieval and normalization step of the data-access layer: it turns a date range and a ticker into a chronologically ordered list of Quote objects that higher layers such as YahooDAM and the TickFeeder can consume. The function first normalizes the input dates by removing any dashes so the strings match the numeric form Yahoo expects, then it composes an ichart.yahoo.com historical CSV URL by extracting the year, month and day pieces from the start and end strings and adjusting the month parameters to the zero-based form the Yahoo CSV endpoint requires. It issues an HTTP GET to that URL (a network call using urllib.urlopen) and reads the returned CSV lines. It trims trailing characters from each CSV line to avoid newline/quote artifacts, splits each line by commas, skips the header row, and for each data row constructs a Quote instance with the CSV fields mapped to time, open, high, low, close, volume and adjClose; the Quote constructor performs the numeric conversions and handles placeholder values. The function collects those Quote objects, sorts them by the Quote.time field to guarantee ascending chronological order, and returns the sorted list. Network IO and any other unexpected failures are caught and reported as UfException instances using the appropriate error codes: an IOError is converted to a NETWORK_ERROR UfException and any other exception is converted to an UNKNOWN_ERROR UfException with the traceback included for diagnostics.
# file path: deprecated/ultrafinance/lib/historicalDataStorage.py
LOG = logging.getLogger(__name__)
This line creates a module-level logger by asking the standard logging facility for a logger named after the module (via the module name variable) so that HistoricalDataStorage and its routines can record informational messages, progress updates and error conditions in a consistent, hierarchical way. That logger will be used across buildExls, __buildExl and buildExlsFromFile to report events like successful downloads, write operations to Excel, and to surface failures that lead to UfException, while leaving handler configuration and formatting to the global logging setup used by the rest of the pipeline.
# file path: deprecated/ultrafinance/lib/historicalDataStorage.py
def __init__(self, outputPrefix, startDate = '1900-01-01', endDate = date.today()):
self.__outputPrefix = outputPrefix
self.__startDate = startDate
self.__endDate = endDate
HistoricalDataStorage.init initializes a HistoricalDataStorage instance by capturing and storing the configuration that downstream methods will use: it assigns the provided outputPrefix to __outputPrefix, the provided startDate to __startDate (defaulting to the long-past string so callers get a full-history baseline when they omit it), and the provided endDate to __endDate (defaulting to the current date at instantiation so exported snapshots end at “now” unless overridden). Those three private attributes are the state that buildExls, __buildExl and buildExlsFromFile later read and pass along to the YahooFinance and GoogleFinance historical-price callers and to the Excel-writing routines, so the initializer’s role is simply to centralize the file-name prefix and the date-range for every historical-data fetch and persistence operation invoked from this object.
# file path: ultrafinance/dam/excelLib.py
LOG = logging.getLogger()
The line instantiates a module-level logger called LOG by invoking the standard logging facility (logging.getLogger), following the same pattern used in YahooFinance and HistoricalDataStorage so this module emits messages into the project’s hierarchical logging system. Having LOG available lets ExcelLib record informational messages, lifecycle progress and error conditions consistently as it manages workbook contexts and delegates operations, which is important because ExcelLib centralizes UfException-based error handling for callers such as ExcelDAM, ExcelDataFeeder and IndexDataFeeder.
# file path: ultrafinance/dam/excelLib.py
READ_MODE = 'r'
READ_MODE is a module-level constant in ExcelLib that encodes the read-only file-open mode used throughout the Excel adapter; ExcelRead, ExcelOpertion and the higher-level context-managed workbook lifecycle consult this constant when they open workbooks so callers like ExcelDAM, ExcelDataFeeder and IndexDataFeeder get a consistent, explicit read-only handle for operations that must not modify a spreadsheet. By centralizing the mode value under the name READ_MODE, the Excel adapter expresses intent clearly to downstream code and to the UfException-based error handling around workbook access, ensuring the data-access layer enforces a safe, uniform policy for read versus write access across the pipeline.
# file path: ultrafinance/dam/excelLib.py
WRITE_MODE = 'w'
The module declares a constant named WRITE_MODE whose value is the single-character flag used to signal write semantics when the adapter opens or creates an Excel workbook. Within ExcelLib’s context-managed workbook lifecycle and its delegations to ExcelWrite and ExcelOpertion, WRITE_MODE is the canonical indicator the code uses whenever it needs to perform mutating operations on a workbook — for example when ExcelWrite commits changes or when ExcelOpertion chooses to open a file for updates rather than read-only access. Using a shared WRITE_MODE constant ensures ExcelDAM, ExcelDataFeeder, and IndexDataFeeder encounter a consistent mode indicator when they ask ExcelLib to save, overwrite, or otherwise modify sheets, and it integrates with the module’s centralized UfException-based error handling for write failures.
# file path: ultrafinance/dam/excelLib.py
def __init__(self, fileName = None, mode = READ_MODE):
if ExcelLib.READ_MODE == mode:
self.__operation = ExcelRead(fileName)
elif ExcelLib.WRITE_MODE == mode:
self.__operation = ExcelWrite(fileName)
else:
raise UfException(Errors.INVALID_EXCEL_MODE,
"Invalid operation mode, only %s and %s are accepted"\
% (ExcelLib.READ_MODE, ExcelLib.WRITE_MODE))
ExcelLib.init decides which concrete Excel operation to use for the lifetime of the ExcelLib instance by inspecting the provided mode and creating either an ExcelRead or an ExcelWrite instance and storing it on the private __operation attribute. When the mode matches ExcelLib.READ_MODE, it constructs an ExcelRead so subsequent context-manager entry will call the read implementation’s pre/post lifecycle and sheet/row/col/cell readers; when the mode matches ExcelLib.WRITE_MODE, it constructs an ExcelWrite so later calls will route to the writer implementation that enforces file-creation semantics. If the caller supplies any other mode, ExcelLib.init signals a configuration error by raising a UfException tagged with Errors.INVALID_EXCEL_MODE and a message indicating only the read and write modes are accepted. This simple mode-based selection is the delegation/factory-like step that binds ExcelLib to a specific ExcelOpertion subclass up front; after construction ExcelLib.enter, exit and the high-level methods (openSheet, readRow, writeRow, etc.) delegate to the chosen __operation, allowing ExcelDataFeeder, IndexDataFeeder and ExcelDAM to work against a consistent context-managed API regardless of whether the underlying operation is reading or writing.
# file path: ultrafinance/dam/excelLib.py
def __enter__(self):
self.__operation.pre()
return self
ExcelLib.enter is the context-manager entry point that activates the workbook operation object created during ExcelLib.init; it calls ExcelOpertion.pre on that operation (the operation was set to either ExcelRead or ExcelWrite by ExcelLib.init) and then returns the ExcelLib instance so the caller can use the adapter’s high-level sheet, row and cell methods. Conceptually this wires the resource-lifecycle pattern into the data-access pipeline: construction chooses the concrete operation, enter gives the operation a chance to perform any pre-use setup (ExcelOpertion.pre is currently a no-op), and the returned ExcelLib instance lets upstream modules such as ExcelDataFeeder and IndexDataFeeder stream reads or writes through the operation proxies. The matching exit will later call the operation’s post step to finalize or flush state, so enter simply serves to hand control back to the caller after preparing the chosen Excel operation.
# file path: ultrafinance/dam/excelLib.py
def __exit__(self, type, value, traceback):
self.__operation.post()
return
ExcelLib.exit is the context-manager tear-down for the Excel adapter: when a with-block using ExcelLib ends it calls the post step on the operation object that ExcelLib.init previously set to either ExcelRead or ExcelWrite, then returns None. Because ExcelLib.init wires __operation to the concrete ExcelRead/ExcelWrite instance, the exit invocation delegates finalization and any persistence or external side effects to ExcelOpertion.post (the operation-level post hook that the writer path is expected to implement). The method signature accepts the standard context-manager exception triple so callers may pass exception information, but exit itself does not transform exceptions — it simply ensures the operation-level post runs before control returns. In the pipeline this provides a consistent lifecycle pairing with ExcelLib.enter (which invoked the operation pre step) so higher-level modules like ExcelDataFeeder and IndexDataFeeder get a single, predictable place where workbook finalization and any post/flush actions occur.
# file path: ultrafinance/dam/excelLib.py
def getOperation(self):
return self.__operation
ExcelLib.getOperation simply hands back the internal operation object that ExcelLib.init created and stored — the ExcelRead instance when the instance was opened in read mode or the ExcelWrite instance when opened in write mode. Because ExcelLib is the adapter that provides a context-managed workbook lifecycle and high-level sheet/row/column/cell helpers, getOperation gives callers direct access to the low-level ExcelOpertion implementation so they can call methods not wrapped by the thin facade (for example the methods implemented on ExcelRead and ExcelWrite such as obtaining sheet counts or names). In practice testExcelLib.testReadExcel and downstream modules like ExcelDataFeeder and IndexDataFeeder use getOperation while inside the ExcelLib context so that ExcelLib.enter has already invoked the operation pre-step and ExcelLib.exit will run the operation post-step when the with block ends. Returning the underlying operation preserves the unified UfException-based error semantics and lets higher layers perform more detailed or implementation-specific work when necessary while the rest of ExcelLib continues to present the simplified, context-managed interface.
# file path: ultrafinance/dam/excelLib.py
class ExcelOpertion(object):
DEFAULT_SHEET = "sheet0"
def openSheet(self, name):
raise UfException(Errors.UNDEFINED_METHOD, "openSheet function is not defined")
def readRow(self, row, startCol=0, endCol=-1):
raise UfException(Errors.UNDEFINED_METHOD, "readRow function is not defined")
def readCol(self, col, startRow=0, endRow=-1):
raise UfException(Errors.UNDEFINED_METHOD, "readCol function is not defined")
def readCell(self, row, col):
raise UfException(Errors.UNDEFINED_METHOD, "readCell function is not defined")
def writeRow(self, sheetName, row, values):
raise UfException(Errors.UNDEFINED_METHOD, "writeRow function is not defined")
def writeCell(self, sheetName, row, col, value):
raise UfException(Errors.UNDEFINED_METHOD, "readCell function is not defined")
def post(self):
return
def pre(self):
return
ExcelOpertion defines the abstract contract that the Excel adapter layer in the data-access pipeline implements: it exposes the common operations a caller expects when working with a workbook—opening a sheet, reading a row, column or cell, writing a row or cell, and optional lifecycle hooks before and after work. It provides a shared DEFAULT_SHEET name that implementing classes can use as the fallback sheet when no sheet has been explicitly opened; ExcelRead uses that constant when callers request reads without first opening a sheet. Each of the I/O methods in ExcelOpertion intentionally raises a UfException reporting an undefined method so that any direct call to the base class fails fast and consistently; the concrete subclasses ExcelRead and ExcelWrite override these methods to perform the actual file operations, and ExcelLib chooses and delegates to the appropriate subclass at runtime. The two lifecycle hooks pre and post are implemented as no-ops in ExcelOpertion so subclasses can optionally perform setup or teardown (for example committing or flushing data) without forcing every implementation to define them. Conceptually ExcelOpertion is an interface/abstract base class that centralizes the method contract and standardizes error behavior via UfException, enabling higher-level components like ExcelLib, ExcelDAM, ExcelDataFeeder and the rest of the pipeline to rely on a uniform Excel operation API and predictable exception semantics.
# file path: ultrafinance/dam/excelLib.py
DEFAULT_SHEET = "sheet0"
DEFAULT_SHEET is a module-level configuration constant that provides a fallback worksheet name (sheet0) used throughout ExcelLib when callers omit an explicit sheet. In the Excel adapter role—where ExcelRead, ExcelWrite and ExcelOpertion perform low-level work and the higher-level ExcelDAM, ExcelDataFeeder and IndexDataFeeder rely on a context-managed workbook lifecycle—having DEFAULT_SHEET ensures every high-level sheet/row/column/cell operation has a predictable target, so methods can assume a valid sheet name, unify error handling via UfException, and avoid scattering hardcoded literals across the data-access layer. Like the module-level logger defined earlier, DEFAULT_SHEET centralizes a small but important convention for consistent behavior across the Excel integration points.
# file path: ultrafinance/dam/excelLib.py
def __init__(self, fileName):
if path.exists(fileName):
raise UfException(Errors.FILE_EXIST, "File already exist: %s" % fileName)
self.__fileName = fileName
self.__workbook = Workbook()
self.__sheetNameDict ={}
self.__sheet = None
ExcelWrite.init is the constructor that prepares an in-memory Excel writer whenever ExcelLib is opened in write mode. It first enforces a guard: if the target file path already exists it raises a UfException with the FILE_EXIST error so callers cannot accidentally overwrite an existing workbook. After that check it records the provided fileName for later persistence, allocates a fresh Workbook instance as the in-memory container for sheets and cells, and initializes an empty mapping to track sheet objects by name. It also leaves the current sheet pointer set to None so that writeRow and writeCell can lazily create or switch to a sheet via openSheet when the first write happens. The fileName stored here is the same value later used by post to save the accumulated Workbook to disk, and the sheet name dictionary is the structure used by openSheet, __getSheet and sheetExsit to manage sheet-level operations.
# file path: ultrafinance/dam/excelLib.py
def openSheet(self, name):
if name not in self.__sheetNameDict:
sheet = self.__workbook.add_sheet(name)
self.__sheetNameDict[name] = sheet
self.__sheet = self.__sheetNameDict[name]
openSheet makes sure there is an actual writable worksheet object inside the in-memory Workbook that ExcelWrite.init created so subsequent writes have a target. It checks the internal sheetNameDict cache for the requested sheet name and, if that name is not already present, asks the Workbook to create a new sheet and stores that sheet object in sheetNameDict. After that existence-or-creation step it assigns the cached sheet object to the internal __sheet reference so writeRow and writeCell will operate against it. Because writeRow and writeCell will call openSheet with the DEFAULT_SHEET when __sheet is None, openSheet provides the lazy creation/lookup behavior that prepares the workbook for later persistence by ExcelWrite.post when the ExcelLib context is closed.
# file path: ultrafinance/dam/excelLib.py
def __getSheet(self, name):
if not self.sheetExsit(name):
raise UfException(Errors.SHEET_NAME_INVALID, "Can't find a sheet named %s" % name)
return self.__sheetNameDict[name]
ExcelWrite.__getSheet acts as the safe accessor for worksheet objects inside the ExcelWrite writer instance: when higher-level write operations need the sheet object they call __getSheet with a sheet name, and __getSheet consults the internal mapping that ExcelWrite.init created to see if that name is present. It delegates the membership test to sheetExsit, which simply checks the name against the __sheetNameDict, and if the name is not found __getSheet raises a UfException with the SHEET_NAME_INVALID error so callers and upstream components receive a consistent, descriptive failure. If the name is present, __getSheet returns the actual sheet object from __sheetNameDict for downstream methods like writeRow, writeCell or post to use. This enforces the invariant that sheet retrieval either yields a valid writable sheet or a standardized UfException, while leaving sheet creation to openSheet which will add a sheet when callers explicitly request it.
# file path: ultrafinance/dam/excelLib.py
def writeCell(self, row, col, value):
if self.__sheet is None:
self.openSheet(super(ExcelWrite, self).DEFAULT_SHEET)
self.__sheet.write(row, col, value)
ExcelWrite.writeCell is the small write-time implementation that actually places a single value into the in-memory workbook the adapter manages. Because ExcelWrite.init initializes the Workbook and leaves the current sheet reference as None, writeCell first ensures there is an active sheet: if the internal __sheet is unset it asks the operation to open the DEFAULT_SHEET (delegating through the class’s openSheet behavior, which will create a new sheet via the Workbook and register it in __sheetNameDict). After guaranteeing an active sheet, writeCell delegates to the worksheet object’s write method to store the given value at the specified row and column. The method therefore keeps writes in-memory; persistence to disk is handled later by ExcelWrite.post, which ExcelLib.exit invokes when the ExcelLib context manager closes. This implementation follows the ExcelOpertion contract for write-mode operations and mirrors the same lazy-sheet-open pattern that writeRow uses.
# file path: ultrafinance/dam/excelLib.py
def writeRow(self, row, values):
if self.__sheet is None:
self.openSheet(super(ExcelWrite, self).DEFAULT_SHEET)
for index, value in enumerate(values):
self.__sheet.write(row, index, value)
ExcelWrite.writeRow is the small, row-level writer used by the Excel adapter to push an ordered sequence of values into the in-memory workbook that ExcelWrite.init created. When called it first ensures there is an active worksheet to write into; if no sheet has been selected it asks for the DEFAULT_SHEET as a fallback, which flows through ExcelLib.openSheet and ultimately invokes the ExcelWrite.openSheet implementation to create or select a sheet on the Workbook that ExcelWrite.init prepared. After guaranteeing a target sheet exists, it iterates over the provided values and writes each one into successive columns on the specified row, starting from column zero. The control flow is a single guard clause to ensure a sheet is present followed by a simple indexed loop that places values cell-by-cell. The actual persistence to disk is handled later when the operation’s post hook runs at context exit (the ExcelLib.exit behavior you already saw), so writeRow only mutates the in-memory sheet state; openSheet and post implement the sheet lifecycle and final save respectively, and error handling for invalid operations is delegated to the ExcelOpertion contract and UfException used elsewhere in the adapter.
# file path: ultrafinance/dam/excelLib.py
def post(self):
self.__workbook.save(self.__fileName)
ExcelWrite.post is the after-work lifecycle hook that actually persists the in-memory workbook prepared by ExcelWrite.init. When a caller has opened the adapter in write mode and used openSheet, writeRow or writeCell to populate sheets, post is invoked by ExcelLib.exit to flush that Workbook instance to the filesystem using the file name ExcelWrite.init stored. In the pipeline this completes the Excel adapter’s responsibility for write operations—turning the transient Workbook state into a concrete file so downstream consumers can read it—and it implements the post step defined by ExcelOpertion to centralize the write-side lifecycle behavior of the Excel adapter.
# file path: ultrafinance/dam/excelLib.py
def __init__(self, fileName):
if not path.exists(fileName):
raise UfException(Errors.FILE_NOT_EXIST, "File doesn't exist: %s" % fileName)
self.__book = open_workbook(fileName)
self.__sheet = None
ExcelRead.init is the constructor the Excel adapter uses when ExcelLib is instantiated for reading; its job is to validate and open the workbook so all higher-level read operations can work against a normalized book object. It first performs a guard check against the filesystem and raises a UfException with Errors.FILE_NOT_EXIST when the supplied fileName does not exist, ensuring the rest of the pipeline gets a consistent, domain-level error rather than a raw OS exception. Assuming the path check passes, it opens the workbook using the underlying Excel opener and stores that workbook in the instance attribute self.__book for downstream use by methods like getTotalSheetNumber and getSheetNames. It also initializes self.__sheet to None so sheet selection is deferred: subsequent calls such as openSheet, readRow, readCol and readCell will either set the explicit sheet or lazily open the DEFAULT_SHEET if no sheet has been chosen. This places ExcelRead.init as the entry point that both enforces file existence and prepares the in-memory workbook state that the rest of the Excel read operations rely on.
# file path: ultrafinance/dam/excelLib.py
def openSheet(self, name):
self.__sheet = self.__book.sheet_by_name(name)
openSheet on ExcelRead simply tells the already-opened workbook which worksheet to use for subsequent read operations by looking up the worksheet by the provided name and storing that sheet object on the ExcelRead instance under __sheet. ExcelRead.init previously created __book by calling the workbook loader and left __sheet as None, so openSheet binds the runtime cursor to a concrete worksheet; later calls like readRow, readCol and readCell rely on that bound __sheet (and their guards will call openSheet with DEFAULT_SHEET if no sheet has been bound yet). Because openSheet delegates the lookup to the underlying workbook object created by open_workbook, an invalid name will surface from that library and the surrounding ExcelRead read methods and the adapter layer translate or wrap such errors into UfException as part of the Excel adapter’s consistent error handling used by ExcelDAM, ExcelDataFeeder and IndexDataFeeder.
# file path: ultrafinance/dam/excelLib.py
def getTotalSheetNumber(self):
return self.__book.nsheets
ExcelRead.getTotalSheetNumber is a tiny read-side accessor that returns the worksheet count from the workbook object ExcelRead.init opened and stored on the instance. Because ExcelRead.init already validated the file and placed the open_workbook into the private __book attribute, getTotalSheetNumber simply surfaces the workbook’s internal sheet count so callers can discover how many worksheets exist before they call openSheet or start reading rows, columns or cells. In the adapter layer this supports the ExcelOpertion contract by providing a direct way for consumers—accessed via ExcelLib.getOperation as we covered earlier—to enumerate or validate sheets; testExcelLib.testReadExcel uses that pathway to print and confirm the number of sheets when ExcelLib is opened in read mode.
# file path: ultrafinance/dam/excelLib.py
def readRow(self, row, startCol=0, endCol=-1):
if self.__sheet is None:
self.openSheet(super(ExcelRead, self).DEFAULT_SHEET)
if abs(row) >= self.__sheet.nrows:
raise UfException(Errors.INDEX_RANGE_ERROR,
"Excellib.readRow: row number too big: row %s, max %s" % (row, self.__sheet.nrows) )
if max(abs(startCol), abs(endCol)) > self.__sheet.ncols:
raise UfException(Errors.INDEX_RANGE_ERROR,
"Excellib.readRow: col number too big: col %s, max %s" % (max(abs(startCol), abs(endCol)), self.sheet.ncols) )
if -1 == endCol:
endCol = self.__sheet.ncols
return [self.readCell(row, i) for i in range(startCol, endCol)]
ExcelRead.readRow is the read-side implementation that returns a sequence of cell values for a horizontal slice of the currently opened worksheet, and it enforces bounds and error semantics expected by the data-access pipeline. When called it first ensures there is an active sheet; if none is set it opens the fallback sheet name (DEFAULT_SHEET, as covered earlier) so callers who forgot to select a sheet still get a predictable target. Next it validates the requested row index against the sheet’s reported row count and raises a UfException with an index-range error if the index would be out of bounds. It then validates the requested start and end column indexes against the sheet’s column count and raises the same kind of UfException on an out-of-range column. If the caller passed the sentinel end value (-1) it replaces that sentinel with the sheet’s column count so the request means “to the end of the row.” Finally, it builds and returns a Python list by iterating the requested column positions and delegating each cell read to the adapter readCell method; those lower-level reads are themselves wrapped to convert unexpected failures into UfException instances. In short, readRow coordinates sheet selection, defensive index checks, sentinel expansion, and per-cell reads so higher layers like ExcelDAM and the TickFeeder receive a clean, consistently error-handled row of values.
# file path: ultrafinance/dam/excelLib.py
def readCol(self, col, startRow=0, endRow=-1):
if self.__sheet is None:
self.openSheet(super(ExcelRead, self).DEFAULT_SHEET)
if abs(col) > self.__sheet.ncols:
raise UfException(Errors.INDEX_RANGE_ERROR,
"Excellib.readCol: col number too big: col %s, max %s" % (col, self.sheet.ncols) )
if max(abs(startRow), abs(endRow)) > self.__sheet.nrows:
raise UfException(Errors.INDEX_RANGE_ERROR,
"Excellib.readCol: row number too big: row %s, max %s" % (max(abs(startRow), abs(endRow)), self.sheet.nrows) )
if -1 == endRow:
endRow = self.__sheet.nrows
return [self.readCell(i, col) for i in range(startRow, endRow)]
ExcelRead.readCol is the Excel adapter method that exposes a safe, high-level way to pull an entire column of cell values from the currently opened worksheet so higher-level data-access layers (for example ExcelDAM, ExcelDataFeeder and IndexDataFeeder) can consume column data without touching xlrd directly. It first ensures a sheet is available by lazily opening the default worksheet name if the internal sheet attribute is None; that open flows through ExcelLib.openSheet and into the ExcelOpertion implementation on ExcelRead which binds the workbook sheet. It then validates bounds: it checks the column index (using absolute value checks so negative indexing is treated consistently against the sheet width) and raises a UfException tagged with Errors.INDEX_RANGE_ERROR if the column is outside the sheet’s column count; it performs a similar absolute-value test for startRow and endRow against the sheet’s row count and raises the same kind of UfException for out-of-range rows. If the caller passed the sentinel endRow of -1 the method normalizes that to the sheet’s total number of rows. On the happy path it iterates the row range and delegates each individual lookup to the adapter’s cell reader, collecting and returning a list of cell values; because those single-cell reads funnel through ExcelLib.readCell and ExcelRead.readCell, any lower-level errors are converted into UfException instances so callers get consistent error semantics.
# file path: ultrafinance/dam/excelLib.py
def readCell(self, row, col):
try:
if self.__sheet is None:
self.openSheet(super(ExcelRead, self).DEFAULT_SHEET)
return self.__sheet.cell(row, col).value
except BaseException as excp:
raise UfException(Errors.UNKNOWN_ERROR, "Unknown Error in Excellib.readCell %s" % excp)
ExcelRead.readCell is the read-side primitive the Excel adapter uses to return the raw value at a given row and column from the workbook that ExcelRead.init opened. When called with row and col it first guarantees there is an active worksheet object on the ExcelRead instance by opening the inherited default sheet if __sheet is not set (that happens via ExcelRead.openSheet, which looks up the worksheet on the workbook opened by ExcelRead.init). Once a sheet is available it asks the underlying sheet object for the cell at the requested coordinates and returns that cell’s stored value to the caller. All runtime exceptions are caught and normalized into a UfException with the UNKNOWN_ERROR code so callers higher in the data-access pipeline (for example ExcelRead.readRow, ExcelRead.readCol and the ExcelLib wrappers that iterate over cells) get a consistent, project-wide error type rather than raw xlrd or Python exceptions. The normal (happy) path simply returns the cell contents; the error path wraps any unexpected failure and re-raises it as a UfException with the original exception text included in the message.
# file path: deprecated/ultrafinance/lib/dataType.py
DateValueType = namedtuple('DateValue', 'date, value')
L2 defines DateValueType as a small, immutable value object with two named fields: date and value; ExcelDataFeeder and IndexDataFeeder use it as the canonical container for the time-stamped numeric rows they read from spreadsheets. In the feeders’ execute methods, columns are pulled via ExcelLib.readCol (which delegates to ExcelRead.readCol as you already reviewed) and each row is turned into a DateValueType instance (the year coerced to a string and paired with the corresponding numeric cell) and appended to the feeder’s stockData/hoursingData/interestData lists. By centralizing this two-field marker, downstream pieces in the pipeline — the normalizers, strategies, and persistence layers — get a consistent, attribute-addressable representation for L2 time/value pairs rather than raw tuples or loose dictionaries, which simplifies consumption and enforces a stable, memory-light value-object shape across data-access modules.
# file path: deprecated/ultrafinance/lib/stockMeasurement.py
LOG = logging.getLogger(__name__)
The module defines a module-level logger named LOG by obtaining a logger object from the logging subsystem using the module’s name; this establishes a dedicated, namespaced logger that all functions and methods in StockMeasurement will use to emit debug, info, warning and error messages. Using logging.getLogger in this way follows the common module-level logger pattern so messages from the statistical and regression computations, from the GoogleFinance and YahooFinance data fetches, and from interactions with downstream consumers like AvgDivProcessor and ChinaReturn can be correlated to this specific component in the pipeline. Because the logging subsystem returns a single logger instance per name, LOG becomes the consistent, centralized channel for observational output and error reporting for everything that happens inside StockMeasurement.
# file path: deprecated/ultrafinance/lib/stockMeasurement.py
def __init__(self, dateValues, benchmark='^GSPC', benchmarkValues=None):
self.__dateValues = dateValues
self.__benchmark = benchmark
self.__benchmarkValues = copy.deepcopy(benchmarkValues)
self.__alpha = None
self.__beta = None
self.__regressioned = False
The StockMeasurement constructor takes the time-series passed in as dateValues along with an optional benchmark symbol and an optional pre-fetched benchmarkValues, then stores those on the instance and prepares the object for on-demand measurements: it assigns the provided dateValues to an internal attribute, records the benchmark identifier, and creates a deep copy of any supplied benchmarkValues so later internal alignment or modifications won’t mutate the caller’s data. It also initializes alpha and beta as unset (None) and flips a regressioned flag to False to indicate that the regression-based calculations have not yet been performed. This design intentionally defers the expensive work of fetching benchmark history and computing linear regression until one of the regression-dependent methods (alpha, beta, marketReturnRate, linearRegression) is called, while allowing cheap aggregate methods like mean and std to operate immediately on the stored dateValues. In the pipeline this enables modules such as AvgDivProcessor to instantiate StockMeasurement with date series produced upstream (for example by the Excel/DAM layers) and obtain simple statistics quickly, with benchmark alignment and regression only triggered when needed.
# file path: deprecated/ultrafinance/lib/stockMeasurement.py
def beta(self):
if not self.__regressioned:
self.linearRegression()
return self.__beta
When a caller asks a StockMeasurement instance for beta, the beta method first checks the instance flag __regressioned that StockMeasurement.init set to False. If the regression has not yet been performed it invokes linearRegression, which is responsible for pulling or aligning benchmark time series (using YahooFinance/GoogleFinance as needed), detecting suspended trading, filtering dates, computing the regression coefficients and writing the instance attributes __beta and __alpha and flipping __regressioned to True. After ensuring that work has been done, beta returns the stored __beta value. This implements a lazy-evaluation/memoization behavior so expensive operations (network calls and the regression) occur at most once per StockMeasurement instance and subsequent calls from consumers like AvgDivProcessor and ChinaReturn get the cached numeric beta immediately.
# file path: ultrafinance/backTest/stateSaver/hbaseSaver.py
LOG = logging.getLogger()
LOG is a module-level logger obtained from Python’s logging module. In the context of HbaseSaver — the StateSaver implementation that uses HBaseLib and coordinates resetCols and commit — LOG is the single logging handle the file’s functions and methods use to record informational, debug and error messages during table lifecycle operations, row read/write activity, resets, commits and when errors are wrapped as UfException. Because the logger is acquired without a specific logger name it resolves to the root logging configuration and therefore inherits the global logging settings (for example those driven by PyConfig for the backtest pipeline), ensuring HbaseSaver’s messages are formatted and routed consistently with the rest of the application. Declaring LOG at module scope makes it immediately available to all parts of HbaseSaver so the persistence flow can be observed and diagnosed during backtest runs.
# file path: ultrafinance/backTest/stateSaver/hbaseSaver.py
class HbaseSaver(StateSaver):
def __init__(self, ip = "localhost", port = 9090):
super(HbaseSaver, self).__init__()
self.__hbase = HBaseLib(ip, port)
self.__writeCache = {}
def resetCols(self, cols):
if self.tableName in self.__hbase.getTableNames():
self.__hbase.deleteTable(self.tableName)
LOG.debug("create table %s with cols %s" % (self.tableName, cols))
self.__hbase.createTable(self.tableName, [ColumnDescriptor(name = str(col), maxVersions = 5) for col in cols])
def read(self, row, col):
oneRow = self.__hbase.getRow(self.tableName, row)
keyValues = oneRow.columns
key = "%s:" % col
if key in keyValues:
return keyValues[key].value
else:
return None
def write(self, row, col, value):
self.__writeCache[(row, col)] = value
def commit(self):
if not self.tableName:
raise UfException(Errors.TABLENAME_NOT_SET,
"Table name not set")
cols = set()
for (row, col) in self.__writeCache.iterkeys():
cols.add(col)
self.resetCols(cols)
for (row, col), value in self.__writeCache.iteritems():
self.__hbase.updateRow(self.tableName,
row,
[Mutation(column = "%s:" % col, value = str(value))])
def setup(self, setting):
pass
HbaseSaver is a StateSaver implementation that persists backtest runtime state into HBase and therefore provides the persistence endpoint for the pipeline: its constructor delegates to StateSaver.init to inherit the table-name handling, constructs an HBaseLib instance to open a Thrift client connection to HBase, and initializes an in-memory write cache. During runtime higher-level components call write to stage a state update; write stores values in the write cache keyed by the (row, col) pair rather than touching HBase immediately. read retrieves a single row from HBase through HBaseLib.getRow, inspects the returned columns map using a column key built from the column name and a trailing colon, and returns the stored value or None if absent. resetCols enforces a fresh schema for the table: it asks HBaseLib for existing table names, deletes the named table if present via HBaseLib.deleteTable, then creates the table with a ColumnDescriptor for each supplied column (each descriptor is created with five max versions) through HBaseLib.createTable. commit is the flush operation: it first verifies a table name has been set (using the StateSaver contract and raising UfException if not), computes the set of columns present in the write cache, calls resetCols to recreate the table schema for those columns, then iterates the cached (row, col) entries and issues row mutations to HBase via HBaseLib.updateRow, packaging each value as a Mutation with the column formatted the same way read expects. setup is a no-op placeholder. Throughout, HbaseSaver delegates lifecycle and row-level behavior to HBaseLib and surfaces errors as UfException, and it intentionally buffers writes in memory until commit recreates the column family layout and pushes all mutations to HBase.
# file path: ultrafinance/backTest/stateSaver/__init__.py
from ultrafinance.lib.errors import Errors, UfException
import abc
import logging
The file brings in ultrafinance.lib.errors.Errors and ultrafinance.lib.errors.UfException so the persistence layer can use the project’s centralized error codes/messages and raise the domain-specific UfException when validations fail (for example validating table names before writes), which aligns with StateSaver’s role of enforcing consistency for HbaseSaver and SqlSaver. It imports Python’s abc support so StateSaver can be declared as an abstract interface with required lifecycle methods (getStates, write, commit, setup, getTableName, setTableName) that concrete savers must implement, enforcing the contract the rest of the backtest pipeline expects. Finally, it imports the standard logging module so StateSaver implementations can emit runtime diagnostics and lifecycle-level logs during persistence operations that StateSaverFactory and the backtest runtime can consume for observability.
# file path: ultrafinance/backTest/stateSaver/__init__.py
def __init__(self):
self.__tableName = None
StateSaver.init simply initializes the table-name slot on the StateSaver instance by setting the internal table-name attribute to a null value, establishing the saver as not yet bound to any persistence table; this tiny initialization is what concrete savers rely on when they call the StateSaver constructor (for example HbaseSaver.init and SqlSaver.init invoke the parent constructor to get that initialized state before they attach their own database clients and caches). The null table-name is important for the persistence lifecycle because getTableName will later return that attribute and setTableName enforces a one-time assignment policy and raises UfException if a caller attempts to overwrite an already-set table name, and other lifecycle operations like resetCols, commit, read and write in HbaseSaver and the table construction/commit flows in SqlSaver depend on that table-name being present or explicitly absent to decide whether to proceed or raise errors. In short, StateSaver.init performs a single, deliberate side effect—clearing the table-name—so the rest of the state-saving pipeline has a well-defined starting point for table-name management and validation.
# file path: ultrafinance/dam/hbaseLib.py
class HBaseLib:
def __init__(self, ip='localhost', port=9090, timeout = 10):
transport = TSocket.TSocket(ip, int(port))
transport.setTimeout(timeout * 1000)
transport = TTransport.TBufferedTransport(transport)
protocol = TBinaryProtocol.TBinaryProtocol(transport)
self.__client = Client(protocol)
transport.open()
def getTableNames(self):
return self.__client.getTableNames()
def deleteTable(self, tName):
if self.__client.isTableEnabled(tName):
self.__client.disableTable(tName)
self.__client.deleteTable(tName)
def disableTable(self, tName):
self.__client.disableTable(tName)
def enableTable(self, tName):
self.__client.enableTable(tName)
def createTable(self, tName, ColumnDescriptors):
try:
self.__client.createTable(tName, ColumnDescriptors)
except ttypes.AlreadyExists as excp:
raise UfException(Errors.HBASE_CREATE_ERROR,
"AlreadyExists Error when creating table %s with cols: %s): %s" % \
(tName, [col.name for col in ColumnDescriptors], excp.message))
def getColumnDescriptors(self, tName):
try:
return self.__client.getColumnDescriptors(tName)
except:
raise UfException(Errors.UNKNOWN_ERROR,
"Error when getting column descriptors table %s" % tName)
def updateRow(self, tName, rowName, mutations, timestamp=None):
try:
if timestamp is None:
self.__client.mutateRow(tName, rowName, mutations)
else:
self.__client.mutateRowTs(tName, rowName, mutations, timestamp)
except Exception as excp:
raise UfException(Errors.HBASE_UPDATE_ERROR,
"Error when updating table %s - rowName %s - mutations %s: %s" % \
(tName, rowName, mutations, excp))
def getRow(self, tName, rowName):
result = self.__client.getRow(tName, rowName)
if not result:
return result
else:
return result[0]
def scanTable(self, tName, columns, startRow="", endRow=None):
if endRow is None:
scanner = self.__client.scannerOpen(tName, str(startRow), columns)
else:
scanner = self.__client.scannerOpenWithStop(tName, startRow, str(endRow), columns)
ret = []
row = self.__client.scannerGet(scanner)
while row:
ret.append(row[0])
row = self.__client.scannerGet(scanner)
return ret
def getClient(self):
return self.__client
HBaseLib provides a centralized, thin persistence adapter that wraps the Thrift HBase client so higher-level components such as HbaseSaver and HBaseDAM can manage table lifecycle and row-level operations without dealing with socket, transport and protocol mechanics or Thrift error types. On construction HBaseLib opens a Thrift socket to the configured host and port, sets a timeout in milliseconds, layers a buffered transport and binary protocol, instantiates the HBase Client and opens the transport so the instance is ready for use by callers. For schema and lifecycle management HBaseLib exposes getTableNames, createTable, enableTable, disableTable and deleteTable; deleteTable first checks and disables an enabled table before deleting it, and createTable translates the Thrift AlreadyExists exception into a UfException so callers receive the project-standard error model. For metadata HBaseLib exposes getColumnDescriptors and maps any failure to a UfException. For writes HBaseLib.updateRow chooses the plain mutateRow or the timestamped mutateRowTs depending on whether a timestamp is provided and converts any exception into a UfException with an HBASE_UPDATE_ERROR code, which lets callers like HbaseSaver handle commit semantics uniformly. For reads HBaseLib.getRow asks the client for rows and returns the first result or the empty result to match how HbaseSaver.read expects a single-row response; scanTable opens either a scanner without a stop row or with a stop row depending on whether an endRow was supplied, repeatedly calls scannerGet until no more rows and returns the collected row objects for HBaseDAM to convert into Tick or Quote objects. HBaseLib also exposes getClient so code that needs direct Thrift access can retrieve the underlying client. Conceptually HBaseLib functions as an adapter/facade over the Thrift client: it centralizes connection lifecycle, normalizes table and row operations, and maps Thrift errors into UfException so the rest of the backtest pipeline (for example the way ExcelRead normalized workbook access for Excel-based DAMs) can interact with HBase through a small, consistent API.
# file path: ultrafinance/dam/hbaseLib.py
def getTableNames(self):
return self.__client.getTableNames()
HBaseLib.getTableNames simply asks the Thrift HBase client that HBaseLib.init opened for the current list of tables and returns that list to the caller. Because HBaseLib centralizes the low-level Thrift socket/protocol handling, this method acts as a tiny adapter façade: it hides the raw client call behind the HBaseLib API so higher layers like HbaseSaver and HBaseDAM can check whether a particular table exists without touching the Thrift client directly. There is no additional processing or branching inside getTableNames itself — it reads from the self.__client established in HBaseLib.init and hands the returned table names back to callers, who then drive lifecycle decisions such as deleting, creating or updating tables (for example HbaseSaver.resetCols and HBaseDAM.writeQuotes/writeTicks use the returned list to decide whether to create or delete a table). This straightforward delegation fits the file’s role of providing a centralized persistence adapter that keeps Thrift details out of the saver/DAM layers and participates in the consistent error/operation surface that HBaseLib exposes.
# file path: ultrafinance/dam/hbaseLib.py
def deleteTable(self, tName):
if self.__client.isTableEnabled(tName):
self.__client.disableTable(tName)
self.__client.deleteTable(tName)
HBaseLib.deleteTable asks the Thrift HBase client (the client instance that HBaseLib.init opened) whether the named table is currently enabled; if it is enabled, it explicitly disables the table using the same client before issuing the table deletion call, otherwise it proceeds straight to deleting the table. This sequence reflects HBase’s lifecycle requirement that a table be disabled prior to deletion, so deleteTable enforces that precondition on behalf of higher-level callers. In the project pipeline this method is the low-level operation that HbaseSaver.resetCols invokes when it wants to drop an existing state table before creating a fresh one, and HBaseLib.disableTable is the thin wrapper used to perform the explicit disable step. The observable control flow is a single branch: check enabled → disable if needed → delete, resulting in the table being removed from the HBase cluster so callers can recreate or reinitialize it.
# file path: ultrafinance/dam/hbaseLib.py
def disableTable(self, tName):
self.__client.disableTable(tName)
disableTable is a single-purpose lifecycle helper on HBaseLib that uses the Thrift client the instance opened during HBaseLib.init to tell HBase to transition the named table into the disabled state. Conceptually it performs a direct delegation: it takes the table name the caller supplies, invokes the underlying HBase client operation to disable that table, and returns control to the caller. In the pipeline this operation is part of table lifecycle management so higher-level components like HBaseLib.deleteTable and HBaseClient.deleteTable can safely disable a table before removing it, and HbaseSaver.resetCols can rely on deleteTable (which may invoke disableTable) when recreating schema. There is no branching or local error handling inside disableTable itself; it relies on the centralized Thrift client held on the HBaseLib instance (created in HBaseLib.init) and lets any lower-level exceptions propagate to callers that map them into UfException or take corrective action. This keeps disableTable as a thin adapter-style delegation that fits HBaseLib’s role of centralizing HBase connection and lifecycle calls for the rest of the persistence layer.
# file path: ultrafinance/dam/hbaseLib.py
def enableTable(self, tName):
self.__client.enableTable(tName)
Remember that HBaseLib.init created and opened a Thrift HBase client and stored it on the instance; HBaseLib.enableTable simply forwards the enable-table request to that underlying client using the table name passed in. In the persistence layer of the pipeline this method is the thin adapter that brings a named HBase table online so higher-level components like HbaseSaver and HBaseDAM can perform subsequent reads or writes; it therefore participates in HBaseLib’s centralized table lifecycle responsibilities alongside methods such as createTable, disableTable and deleteTable. enableTable contains no additional validation or transformation itself — it delegates the lifecycle operation to the Thrift client so the call flows directly from the saver/DAM through HBaseLib to the HBase server, with error behavior consistent with HBaseLib’s overall error-handling conventions used by callers.
# file path: ultrafinance/dam/hbaseLib.py
def createTable(self, tName, ColumnDescriptors):
try:
self.__client.createTable(tName, ColumnDescriptors)
except ttypes.AlreadyExists as excp:
raise UfException(Errors.HBASE_CREATE_ERROR,
"AlreadyExists Error when creating table %s with cols: %s): %s" % \
(tName, [col.name for col in ColumnDescriptors], excp.message))
HBaseLib.createTable is the HBase lifecycle operation that higher-level persistence components like HbaseSaver and HBaseDAM call when they need a column-family schema created for a persistence table; it delegates the actual work to the Thrift client that HBaseLib opened during HBaseLib.init, attempting to instruct HBase to create the named table with the provided ColumnDescriptor list. On the normal (happy) path it simply hands the table name and descriptors down to the Thrift client and returns, leaving callers such as HbaseSaver.resetCols or HBaseDAM.writeQuotes/writeTicks to proceed with writes once the table exists. If the Thrift layer reports that the table already exists via the ttypes.AlreadyExists exception, createTable catches that Thrift-specific error and translates it into the project’s domain-level UfException with the Errors.HBASE_CREATE_ERROR code, packaging a message that identifies the table and the column-family names along with the original exception text; this keeps error handling uniform for callers that expect UfException rather than raw Thrift errors.
# file path: ultrafinance/dam/hbaseLib.py
def getColumnDescriptors(self, tName):
try:
return self.__client.getColumnDescriptors(tName)
except:
raise UfException(Errors.UNKNOWN_ERROR,
"Error when getting column descriptors table %s" % tName)
HBaseLib.getColumnDescriptors is the thin adapter method that asks the underlying Thrift HBase client for the column-family descriptors for the table named by tName and returns whatever descriptor structure the client produces back to the caller (HbaseSaver, HBaseDAM, or other HBase-using components). It relies on the Thrift client instance that HBaseLib.init established and opened, so the data flow is: caller supplies a table name, the method delegates the request to the internal client, and the returned column-descriptor information flows back to the caller for schema decisions (for example HbaseSaver’s table reset/commit logic or HBaseDAM’s table creation). For error handling it does not try to interpret specific Thrift exceptions; it catches any exception and converts it into a domain-level UfException using the project error catalog, tagging it with Errors.UNKNOWN_ERROR and a message that includes the table name so higher layers can uniformly handle failures. In short, getColumnDescriptors is a simple adapter/facade call that centralizes the client invocation and normalizes exceptions into UfException for callers.
# file path: ultrafinance/backTest/stateSaver/hbaseSaver.py
def resetCols(self, cols):
if self.tableName in self.__hbase.getTableNames():
self.__hbase.deleteTable(self.tableName)
LOG.debug("create table %s with cols %s" % (self.tableName, cols))
self.__hbase.createTable(self.tableName, [ColumnDescriptor(name = str(col), maxVersions = 5) for col in cols])
HbaseSaver.resetCols is the routine that ensures the HBase table backing the StateSaver matches the set of column families the backtest is about to persist; commit builds the set of cols from the in-memory write cache and passes them here so the persistence layer can establish the correct schema before any row updates. The method first asks HBaseLib for the current table names and, if the StateSaver has already been bound to an existing table, instructs HBaseLib to delete that table (HBaseLib.deleteTable will disable the table if needed and then remove it via the Thrift client). It then emits a debug-level message via LOG so the operation is visible in diagnostics, and finally asks HBaseLib to create a new table using a ColumnDescriptor for each column name found in cols, converting each column identifier to a string and setting the column descriptor to retain up to five historical versions. In control-flow terms there is a single conditional: remove the old table if it exists, then always create the new one; semantically this guarantees a fresh table whose column families align with the write cache and that stores multiple versions per column as configured.
# file path: ultrafinance/dam/hbaseLib.py
def updateRow(self, tName, rowName, mutations, timestamp=None):
try:
if timestamp is None:
self.__client.mutateRow(tName, rowName, mutations)
else:
self.__client.mutateRowTs(tName, rowName, mutations, timestamp)
except Exception as excp:
raise UfException(Errors.HBASE_UPDATE_ERROR,
"Error when updating table %s - rowName %s - mutations %s: %s" % \
(tName, rowName, mutations, excp))
HBaseLib.updateRow is the thin adapter that takes a target table name, a row key, a list of Thrift Mutation objects and an optional timestamp, and pushes that mutation to HBase over the Thrift client that HBaseLib.init opened. On the normal path it delegates to the Thrift client’s row-mutation method; when a timestamp is supplied it uses the timestamp-aware mutation call so callers can write a specific cell version, otherwise it issues a standard mutate call. Any exception coming from the Thrift layer is caught and translated into a domain UfException using Errors.HBASE_UPDATE_ERROR with a message that includes the table, row, mutations and the original exception text; that keeps higher layers such as HbaseSaver.commit and HBaseDAM.writeQuotes/writeTicks from having to handle raw Thrift errors and enforces the project’s unified error model. Functionally this method acts as an adapter/wrapper around the Thrift client: it centralizes the transport call, encodes the timestamp decision, and maps transport-level failures into the project-level UfException so callers can manage schema resets, commits and reads consistently.
# file path: ultrafinance/backTest/stateSaver/hbaseSaver.py
def __init__(self, ip = "localhost", port = 9090):
super(HbaseSaver, self).__init__()
self.__hbase = HBaseLib(ip, port)
self.__writeCache = {}
HbaseSaver.init initializes an HbaseSaver instance by first invoking the StateSaver constructor so the internal table-name slot starts out unset (remember StateSaver.init sets the table-name to None), then it constructs an HBaseLib client using the provided ip and port so the saver has an open Thrift-backed adapter for all subsequent table lifecycle and row-level operations, and finally it creates an empty in-memory write cache to stage row/column writes before they are persisted. By wiring an HBaseLib instance and a write cache into the object, the HbaseSaver constructor prepares the persistence endpoint that resetCols, read, write and commit will use: resetCols and commit will call into HBaseLib to manage tables and update rows, read will retrieve rows through HBaseLib, and write will populate the in-memory cache that commit will flush to HBase. The ip and port parameters let the backtest runtime target a specific HBase host and Thrift port, and the method has the side effect of assigning those client and cache attributes on the HbaseSaver instance.
# file path: ultrafinance/backTest/stateSaver/hbaseSaver.py
def write(self, row, col, value):
self.__writeCache[(row, col)] = value
As the HBase-backed StateSaver responsible for persisting backtest runtime state, HbaseSaver.write functions as the in-memory staging step that captures a single cell update without touching HBase. When called it records the provided row, column and value into the instance write cache that HbaseSaver.init created, using a tuple of row and column as the dictionary key and the provided value as the dictionary value. That cached mapping lets the persistence layer batch many writes and defer table lifecycle and row mutations until commit runs; commit will inspect the cached keys to build the set of columns, call resetCols to recreate the HBase table schema via HBaseLib, and then push the cached entries into HBase with updateRow, while read uses HBaseLib.getRow to fetch data back. write itself performs no table validation or I/O—its sole job is to stage the update so later steps in the saver pipeline can manage schema, validation (where commit will raise UfException if the table name is not set), and the actual Thrift-backed operations.
# file path: ultrafinance/backTest/stateSaver/hbaseSaver.py
def commit(self):
if not self.tableName:
raise UfException(Errors.TABLENAME_NOT_SET,
"Table name not set")
cols = set()
for (row, col) in self.__writeCache.iterkeys():
cols.add(col)
self.resetCols(cols)
for (row, col), value in self.__writeCache.iteritems():
self.__hbase.updateRow(self.tableName,
row,
[Mutation(column = "%s:" % col, value = str(value))])
HbaseSaver.commit is the persistence step that flushes the in-memory state captured by HbaseSaver.write into HBase so the backtest runtime state is durable. It first enforces that a target table name has been set on the StateSaver; if not, it raises a UfException indicating the table name is not set so callers cannot accidentally persist into nowhere. Next it scans the write staging map that HbaseSaver.init created and that HbaseSaver.write populates, collecting the unique column families present in the cached (row, column) keys; it passes that set into HbaseSaver.resetCols so the HBase table schema will match the families the backtest intends to persist (resetCols was explained earlier as the routine that re-creates the table with the required ColumnDescriptor entries). After ensuring the table schema, commit iterates over every cached cell entry and issues a row-level mutation through HBaseLib.updateRow for each one: for each cached (row, column) pair it constructs a single Mutation that encodes the column family/qualifier and the cell value (coerced to a string) and asks HBaseLib to apply it to the named table and row. HBaseLib.updateRow is responsible for talking to the Thrift client and will raise a UfException on transport or server errors, so commit relies on those exceptions to propagate update failures. Control flow is simple: a guard-on-table-name, one pass to collect column families, then one pass to push each cached cell; the function does not supply timestamps to updateRow and does not batch beyond per-cell mutations, and any error paths are surfaced as UfException instances so the backtest pipeline can handle persistence failures.
# file path: ultrafinance/backTest/stateSaver/hbaseSaver.py
def setup(self, setting):
pass
HbaseSaver.setup reads the provided PyConfig-style setting and turns that configuration into a live, ready-to-use persistence context for the backtest runtime. It takes the table name and any HBase-specific options out of setting and stores the table name on the StateSaver slot that HbaseSaver.init left unset, then uses the HBaseLib client that HbaseSaver.init opened to ensure the HBase table is present and its column-family schema aligns with what the backtest will persist. To do that it asks HBase about the current column-family descriptors via HBaseLib.getColumnDescriptors, computes the set of column families required for the saver (either from configuration or from the saver’s in-memory write cache), and invokes resetCols to create or alter the table schema so future commits and row updates can succeed. After the schema is reconciled, setup optionally reads existing rows from the table to populate the in-memory staging cache so the runtime can be restored to a previous state; any row-level writes after that continue to be staged by HbaseSaver.write and later pushed by HBaseLib.updateRow when commit runs. Any protocol or configuration failures are reported through the project’s UfException error mechanism so callers know setup failed. The method returns after the table name, schema alignment, and optional restore step complete, leaving the HbaseSaver ready to accept staged writes and commits.
# file path: ultrafinance/backTest/stateSaver/hbaseSaver.py
if __name__ == '__main__':
h = HbaseSaver()
h.tableName = 'unittest_outputSaver'
for i in range(5):
h.write('time1', 'accountValue', 10000)
h.commit()
accountValue = h.read('time1', 'accountValue')
print(accountValue)
assert str(10000) == accountValue
assert None == h.read('time1', 'EDFASNONdafs')
When the file is executed directly the script creates an HbaseSaver instance (which, as you already know from HbaseSaver.init, has opened an HBaseLib client and initialized an empty in-memory write cache) and assigns a concrete table name so subsequent operations target a real HBase table. It then runs five iterations that exercise the saver: each iteration stages a single cell update via HbaseSaver.write into the in-memory write cache, calls commit which builds the set of column families from that cache and invokes resetCols and HBaseLib.updateRow to ensure the table schema and push the mutation into HBase, and then retrieves the cell back using HbaseSaver.read to verify persistence. The retrieved value is printed and an assertion checks that the stored numeric value comes back as the expected string representation, and a second assertion verifies that reading a non-existent column returns None. This loop therefore serves as a small end-to-end verification that staging, schema synchronization, row mutation, and readback all work together correctly across repeated commits.
# file path: ultrafinance/lib/errors.py
class Errors(object):
UNKNOWN_ERROR = 1
FILE_NOT_EXIST = 2
FILE_EXIST = 3
UNDEFINED_METHOD = 4
NETWORK_ERROR = 100
NETWORK_400_ERROR = 101
INDEX_RANGE_ERROR = 200
INVALID_DAM_TYPE = 201
STOCK_SYMBOL_ERROR = 300
STOCK_PARSING_ERROR = 301
HBASE_CREATE_ERROR = 401
HBASE_UPDATE_ERROR = 402
SIDE_TYPE_ERROR = 500
ORDER_TYPE_ERROR = 501
TRANSITION_TYPE_ERROR = 502
FEEDER_INVALID_ERROR = 600
SYMBOL_EXIST = 601
INVALID_TYPE = 602
SYMBOL_NOT_IN_SOURCE = 604
FEEDER_TIMEOUT = 605
ORDER_INVALID_ERROR = 700
MISSING_SYMBOL = 701
SHEET_NAME_EXIST = 800
SHEET_NAME_INVALID = 801
INVALID_EXCEL_MODE = 802
INVALID_ACCOUNT = 901
INVALID_METRIC_NAME = 1001
ACCOUNT_ALEADY_SET = 1002
ACCOUNT_NOT_SET = 1003
INVALID_RISK_FREE_RETURN = 1004
INVALID_STRATEGY_NAME = 1200
NONE_ACCOUNT_ID = 1201
NONE_TRADING_CENTER = 1202
INVALID_SYMBOLS = 1203
TABLENAME_NOT_SET = 1300
TABLENAME_ALREADY_SET = 1301
INVALID_SAVER_NAME = 1302
Errors is a centralized registry of numeric error codes used across the pipeline so components can raise and inspect standardized failure conditions via UfException and make programmatic decisions instead of parsing free-form messages. The class simply exposes a set of named integer constants whose names indicate the subsystem and fault type—file-system and unknown errors, network errors, index and parsing errors for data access, stock- and symbol-related errors, HBase-related create/update errors, trading-side and order-type errors, feeder and timeout conditions, Excel/sheet validation issues, account and metric validation errors, strategy and symbol configuration problems, and table-name/saver configuration problems. Grouping the codes into contiguous numeric ranges makes it straightforward for callers such as DAMs, TickFeeder, HbaseSaver and configuration code like PyConfig to pick an appropriate code when constructing a UfException, and for the rest of the backtest pipeline to handle or log errors consistently and inspectably without coupling to textual messages.
# file path: ultrafinance/lib/errors.py
UNKNOWN_ERROR = 1
UNKNOWN_ERROR is a named error code constant defined in errors.py and given the numeric value 1; it serves as the canonical, generic error category that the Errors mapping and UfException use when no more specific error code applies. Because errors.py centralizes the project’s error codes and messages, UNKNOWN_ERROR provides a stable integer key that code such as PyConfig can place into a UfException during configuration validation so callers elsewhere in the pipeline can inspect the exception’s code, map it to a human-readable message via the Errors mapping, and handle an unspecified or fallback failure case consistently across the system.
# file path: ultrafinance/lib/errors.py
FILE_NOT_EXIST = 2
errors.py defines a stable, numeric identifier for the situation where a required file cannot be found by naming a module-level constant FILE_NOT_EXIST and giving it the value two. That numeric code is the canonical key used alongside the Errors mapping and carried inside UfException instances so PyConfig can raise a uniform, machine-readable error when configuration loading or validation fails; downstream callers can inspect the exception’s code to decide how to handle a missing file. Centralizing the code here keeps the backtest pipeline — including configuration, TickFeeder setup, and persistence components — from scattering magic numbers and allows logs, conditionals, and any error-to-message lookup in Errors to operate consistently across ultrafinance-master_cleaned.
# file path: ultrafinance/lib/errors.py
FILE_EXIST = 3
The constant named FILE_EXIST is the predefined numeric code used by the Errors mapping to identify the specific error condition that a file already exists; here it is assigned the integer value three so that callers and exception handlers can refer to a compact, machine-friendly identifier instead of relying on free-form text. Within errors.py this numeric code is paired with a human-readable message inside Errors and is carried by UfException instances so that PyConfig (which raises UfException during configuration setting and validation) and any other component can catch the exception, read the code via UfException accessors, and make a deterministic decision based on that exact error class rather than parsing a string. This approach keeps error signaling consistent across the pipeline, lets logging and telemetry record a stable enum-like value, and allows downstream logic to implement distinct handling for the file-exists condition by matching against the FILE_EXIST code.
# file path: ultrafinance/lib/errors.py
UNDEFINED_METHOD = 4
UNDEFINED_METHOD is an integer constant that serves as one of the canonical error codes in the Errors registry; it labels the specific failure case where a requested or configured method is not defined so that PyConfig and other components can raise a UfException with that code and allow downstream logic to detect and handle the undefined-method error in a consistent, inspectable way.
# file path: ultrafinance/lib/errors.py
NETWORK_ERROR = 100
NETWORK_ERROR is the named numeric error code declared in errors.py that represents network-level failures across the pipeline; errors.py centralizes these codes and their human-readable messages in the Errors mapping and the UfException wrapper so all layers use the same identifiers when signaling problems. When a DAM or an HBase-facing component such as HBaseLib (for example during the Thrift client lifecycle opened in HbaseSaver.init or while performing an update via HBaseLib.updateRow) encounters a connectivity or transport failure, the runtime will construct a UfException carrying this NETWORK_ERROR code (the code’s numeric value of one hundred) and the associated message from Errors, allowing callers up the stack — TickFeeder, state persistence, or strategy subscribers — to detect the condition programmatically, surface a consistent error message in logs, and handle the failure according to the pipeline’s error-handling logic.
# file path: ultrafinance/lib/errors.py
NETWORK_400_ERROR = 101
The line defines a named error code constant called NETWORK_400_ERROR and gives it the integer value 101, making a machine-readable identifier the rest of the system can use to represent a client-side/network 400-style failure. In the context of errors.py, that numeric constant is one of the standardized codes that the Errors mapping and UfException rely on so callers can raise or inspect a consistent, programmatic error value instead of free-form text. Because PyConfig uses UfException for configuration validation and other components in the pipeline (for example, modules that open networked clients such as HBaseLib used by HbaseSaver) need to signal network-related failures, having NETWORK_400_ERROR centralized here lets those layers communicate the same semantic condition (a 400-like network error) and allows higher-level code to branch on the numeric code and look up the canonical message from the Errors mapping.
# file path: ultrafinance/lib/errors.py
INDEX_RANGE_ERROR = 200
The line registers a named error code called INDEX_RANGE_ERROR and assigns it the numeric identifier 200 so the rest of the system can refer to an out-of-range index condition with a compact, consistent token; like FILE_NOT_EXIST, FILE_EXIST, UNDEFINED_METHOD, NETWORK_ERROR and NETWORK_400_ERROR, this constant will be looked up in the Errors mapping and used to instantiate a UfException when an index/slice or time-range access fails (for example during TickFeeder normalization, strategy windowing, indicator warm-up, or configuration date-range validation), allowing callers and exception handlers across the data acquisition, strategy and execution layers to detect and respond to index-range failures programmatically rather than relying on free-form text.
# file path: ultrafinance/lib/errors.py
INVALID_DAM_TYPE = 201
The constant INVALID_DAM_TYPE is declared as a numeric error identifier (two hundred and one) that the Errors registry and UfException use to represent the specific failure where a configured data access module (DAM) is not of an expected or supported type. In the ultrafinance pipeline this lets PyConfig and other validation code detect misconfigured or wrong-class DAM entries before they reach the TickFeeder, raise a UfException carrying a stable machine-readable code plus a human message from the Errors mapping, and allow callers or runtime components to handle that situation deterministically. It sits alongside the other canonical codes you already saw (like FILE_NOT_EXIST, FILE_EXIST, UNDEFINED_METHOD, NETWORK_ERROR, and NETWORK_400_ERROR) so the system uses a single, consistent identifier whenever a DAM type validation error occurs.
# file path: ultrafinance/lib/errors.py
STOCK_SYMBOL_ERROR = 300
STOCK_SYMBOL_ERROR is the named integer constant used as the canonical error code for situations where a stock symbol is invalid, unknown, or cannot be resolved; by assigning that distinct numeric identifier the Errors registry and the UfException wrapper can represent symbol-related failures consistently across the pipeline. In practice, data acquisition modules and crawlers, the TickFeeder normalization stage, and configuration validation in PyConfig will reference STOCK_SYMBOL_ERROR when they detect a bad symbol and then raise or encapsulate the problem in a UfException so downstream components—strategy subscribers, execution simulation, and metrics/persistence—can inspect the numeric code and handle the condition programmatically rather than relying on free-form text. Because errors.py centralizes these codes alongside others like NETWORK_ERROR, using STOCK_SYMBOL_ERROR ensures a stable, machine-friendly way to classify and propagate symbol resolution errors throughout ultrafinance-master_cleaned.
# file path: ultrafinance/lib/errors.py
STOCK_PARSING_ERROR = 301
STOCK_PARSING_ERROR is the named numeric identifier used to signal failures that occur while transforming raw market input into the normalized tick/quote form the runtime expects; it is given the integer value 301 and is registered in the Errors mapping so code across the project can refer to a compact, machine-friendly token instead of free-form text. In the layered pipeline, data originates in the DAMs and crawlers and then flows into the TickFeeder for normalization, and when those components encounter malformed rows, unexpected CSV/JSON layouts, bad type conversions, or missing fields they use STOCK_PARSING_ERROR (wrapped in a UfException) to announce the problem upstream. Like FILE_NOT_EXIST, FILE_EXIST, UNDEFINED_METHOD, and NETWORK_400_ERROR, STOCK_PARSING_ERROR provides a consistent code that exception handlers and logging can inspect; the UfException wrapper carries that code and a human-readable message so configuration validators such as PyConfig and the rest of the runtime can uniformly detect, log, and react to parsing failures during data ingestion.
# file path: ultrafinance/lib/errors.py
HBASE_CREATE_ERROR = 401
Like FILE_NOT_EXIST and NETWORK_400_ERROR described earlier, HBASE_CREATE_ERROR is a stable numeric identifier (value 401) that the Errors registry and UfException use to denote a specific failure mode: an error encountered when the HBase data access module attempts to create HBase resources (for example tables or namespaces) during configuration or runtime. Centralizing this code in errors.py lets PyConfig and other layers raise a UfException carrying that machine-friendly code so the rest of the pipeline — including the DAMs, TickFeeder, and strategy/error-handling logic — can programmatically recognize and respond to an HBase-create failure while the Errors mapping provides the corresponding human-readable message for logging and user feedback.
# file path: ultrafinance/lib/errors.py
HBASE_UPDATE_ERROR = 402
Within the centralized Errors registry and the UfException wrapper, HBASE_UPDATE_ERROR names the specific failure mode for an inability to update records in the HBase persistence backend and assigns it the numeric identifier 402. In the pipeline this corresponds to the point where normalized ticks, trade executions or aggregated results are being written into the HBase storage; when an HBase write/update operation fails, the persistence layer will signal that condition using the HBASE_UPDATE_ERROR code (wrapped in a UfException and paired with the human-readable message from Errors) so upstream callers and error handlers can detect and respond programmatically. This follows the same pattern used by other constants like FILE_NOT_EXIST, FILE_EXIST and NETWORK_ERROR: provide a compact, machine-friendly key the rest of the system can inspect to branch logic, log consistently, or surface a clear message to operators.
# file path: ultrafinance/lib/errors.py
SIDE_TYPE_ERROR = 500
The constant named SIDE_TYPE_ERROR is assigned the numeric identifier 500 to serve as the canonical error code for an invalid or unsupported trade side within the system. In errors.py, that numeric label becomes the key that the Errors mapping and the UfException wrapper use to represent situations where a side value (for example an unexpected buy/sell indicator coming from configuration, a strategy, or an order request) is not acceptable. When code paths such as PyConfig validation or a strategy/order creation detect that the side type is wrong, they raise a UfException carrying this code so callers can look up the human-readable message in Errors and handle the failure programmatically by inspecting the numeric identifier rather than parsing text. This keeps side-type validation failures consistent with other centralized error conditions like the network and file errors already defined.
# file path: ultrafinance/lib/errors.py
ORDER_TYPE_ERROR = 501
ORDER_TYPE_ERROR is defined as the canonical numeric identifier the project uses to signal an invalid or unsupported order type; within errors.py it becomes one of the entries the Errors mapping can reference and it is the code that UfException will carry when components need to throw a structured error about order types. Because errors.py centralizes these codes and messages, having ORDER_TYPE_ERROR lets strategy implementations and the execution/simulation layer raise a consistent, machine-friendly signal when a trade request uses the wrong order kind, and lets callers and error handlers inspect the numeric code (rather than parsing free-form text) to decide how to respond. PyConfig and other modules that validate inputs can likewise use ORDER_TYPE_ERROR via UfException so configuration and runtime checks propagate the same identifiable condition across the pipeline.
# file path: ultrafinance/lib/errors.py
TRANSITION_TYPE_ERROR = 502
As part of the centralized error registry in errors.py, TRANSITION_TYPE_ERROR provides a stable numeric identifier (502) that labels the specific failure mode where a transition type is invalid or inappropriate for the operation being attempted. Like the previously discussed FILE_NOT_EXIST and NETWORK_400_ERROR constants, this constant is intended to be used as a machine-friendly key in the Errors mapping and as the code carried by a UfException so callers can programmatically detect and handle transition-type failures. In practice PyConfig or other modules that validate configuration or state transitions will raise a UfException using this identifier so the rest of the layered pipeline can inspect the code, map it to a human-readable message from Errors, and follow the appropriate error-handling path.
# file path: ultrafinance/lib/errors.py
FEEDER_INVALID_ERROR = 600
errors.py adds a named, machine-friendly error code called FEEDER_INVALID_ERROR and assigns it the numeric value 600 so the rest of the pipeline can unambiguously signal problems coming from the TickFeeder layer; in practice this constant is entered into the Errors mapping and is the identifier that UfException instances will carry when PyConfig or runtime code detects that a feeder is misconfigured, unable to produce normalized ticks, or otherwise invalid. By giving feeder failures their own canonical numeric code alongside the other codes you already saw such as NETWORK_400_ERROR and FILE_NOT_EXIST, the project makes it straightforward for subscribers, strategy logic, and higher-level orchestration to programmatically distinguish feeder-level faults from network, file, or undefined-method errors and handle them consistently.
# file path: ultrafinance/lib/errors.py
SYMBOL_EXIST = 601
Within the centralized Errors registry that UfException uses to carry structured failures through the pipeline, SYMBOL_EXIST is defined with the numeric identifier 601 to represent the specific failure mode when an instrument or ticker being registered already exists in the system. In practice, when PyConfig or one of the data acquisition components (for example during symbol registration with a DAM or when the TickFeeder is being wired up) detects a duplicate symbol entry, the code will raise a UfException that carries the SYMBOL_EXIST code and the corresponding human-readable message from the Errors mapping. Assigning the explicit 601 code and grouping it near FEEDER_INVALID_ERROR (600) makes the duplicate-symbol condition programmatically distinguishable from feeder, order, side, transition, or persistence errors, so callers higher in the backtest runtime or configuration validation can inspect the exception and decide whether to surface the problem to the operator, ignore a duplicate, or take other domain-appropriate recovery steps.
# file path: ultrafinance/lib/errors.py
INVALID_TYPE = 602
INVALID_TYPE introduces a dedicated, machine-friendly error code (602) into the centralized Errors registry so the system can unambiguously signal situations where a value’s type is invalid or unexpected. In the context of this layered pipeline, that means configuration validation in PyConfig, data normalization in TickFeeder, or type checks inside strategy logic can raise a UfException that carries INVALID_TYPE so upstream code and monitoring can inspect the exact failure mode. Because UfException wraps an error code and message with accessor methods, using INVALID_TYPE ensures the same structured error shape is produced wherever a type-mismatch or unsupported type condition is detected, allowing the rest of the runtime and persistence/metrics layers to handle type-related failures consistently alongside the other codes like FEEDER_INVALID_ERROR.
# file path: ultrafinance/lib/errors.py
SYMBOL_NOT_IN_SOURCE = 604
SYMBOL_NOT_IN_SOURCE establishes a specific machine-readable error code that the centralized Errors registry and the UfException wrapper use to signal that a requested trading symbol could not be found in the upstream data source. Within the project’s layered pipeline this error is the precise way PyConfig validation, a TickFeeder instance, or one of the data access modules will communicate a missing symbol condition to callers and subscribers: the constant is entered into Errors and becomes the numeric identifier carried by a UfException when configuration-time checks or runtime lookups fail to locate the symbol in the DAM or feeder. This makes the missing-symbol case distinguishable from the broader feeder problems already captured by FEEDER_INVALID_ERROR and allows downstream components—strategies, execution simulation, and persistence—to detect and handle a symbol-not-found failure path explicitly rather than treating it as a generic feeder or configuration error.
# file path: ultrafinance/lib/errors.py
FEEDER_TIMEOUT = 605
FEEDER_TIMEOUT defines a specific machine-friendly error identifier (605) used to signal that the TickFeeder layer failed to produce normalized ticks within an expected time window. Placed in the centralized Errors registry, it complements the previously defined FEEDER_INVALID_ERROR by representing the distinct failure mode of an unresponsive or stalled feeder rather than a misconfiguration; when the feeder runtime or PyConfig detects this timeout condition they raise a UfException carrying FEEDER_TIMEOUT so downstream systems can inspect the code and handle the timeout deterministically as the tick stream is being normalized and handed off to strategy subscribers.
# file path: ultrafinance/lib/errors.py
ORDER_INVALID_ERROR = 700
ORDER_INVALID_ERROR defines a canonical numeric identifier (700) that labels the specific failure mode where an order object is malformed, fails validation, or otherwise cannot be accepted by the pipeline. Because errors.py centralizes error codes in the Errors mapping and uses UfException to carry a code and message for consistent, inspectable error handling, ORDER_INVALID_ERROR becomes the machine-friendly code that UfException instances will carry when PyConfig or runtime components—such as strategy validation logic, the execution simulator, the TickFeeder-driven order flow, or persistence layers—need to signal an invalid order. As we saw earlier with HBASE_UPDATE_ERROR, SIDE_TYPE_ERROR, ORDER_TYPE_ERROR, TRANSITION_TYPE_ERROR and FEEDER_INVALID_ERROR, ORDER_INVALID_ERROR complements that registry by providing a single, stable numeric identifier that callers and orchestrators can test against to route error handling and metrics collection for order-validation failures.
# file path: ultrafinance/lib/errors.py
MISSING_SYMBOL = 701
The file adds a canonical error code named MISSING_SYMBOL with the numeric identifier 701 to the central Errors registry so the rest of the pipeline can signal and inspect the specific failure mode when a requested market symbol is absent or cannot be resolved. In practice, components such as the configurator PyConfig, the data access modules that query Yahoo/SQL/HBase, and the TickFeeder that must normalize ticks will use UfException carrying this code when a required ticker is not provided in configuration, cannot be found in a data source, or is not present in the normalized feed that strategies subscribe to. Because Errors and UfException centralize codes like FEEDER_INVALID_ERROR and HBASE_UPDATE_ERROR, assigning 701 to MISSING_SYMBOL lets downstream code and monitoring map that precise condition to a reproducible identifier and human-readable message for logging, error handling, and diagnostics during backtests and live runs.
# file path: ultrafinance/lib/errors.py
SHEET_NAME_EXIST = 800
SHEET_NAME_EXIST is added to the centralized Errors registry as the canonical, machine-friendly code (800) used to signal that an attempted operation encountered a spreadsheet sheet name that already exists; in practice the Excel data access module or PyConfig during configuration validation will trigger a UfException carrying this code and its associated message so callers can programmatically detect and handle a duplicate-sheet condition in the pipeline. This follows the same centralized-error pattern used by other codes like HBASE_UPDATE_ERROR and SIDE_TYPE_ERROR, enabling uniform error inspection across components such as the Excel DAM, PyConfig, and any part of the TickFeeder or strategy setup that must validate or create sheet entries.
# file path: ultrafinance/lib/errors.py
SHEET_NAME_INVALID = 801
SHEET_NAME_INVALID is the project’s machine-friendly error identifier for an invalid Excel sheet name, assigned the numeric value 801 so it can be referenced consistently across the system. It lives in the Errors registry alongside other domain-specific codes like HBASE_UPDATE_ERROR and FEEDER_INVALID_ERROR, and is the code that UfException will carry when configuration validation in PyConfig or the Excel DAM detects that a configured sheet name is missing, misspelled, or otherwise unacceptable. When raised, that UfException makes the failure explicit and inspectable to upstream components (for example preventing the TickFeeder from receiving normalized ticks from an Excel source) so callers and monitoring can branch on the specific 801 condition instead of relying on freeform text.
# file path: ultrafinance/lib/errors.py
INVALID_EXCEL_MODE = 802
INVALID_EXCEL_MODE is the project-level, machine-friendly error identifier (802) that marks the specific failure mode where an Excel-based data access module or its configuration has been given an unsupported or otherwise invalid mode. In errors.py this name becomes an entry in the Errors registry and is the code that UfException will carry when PyConfig detects a bad Excel mode during configuration validation or when an Excel DAM or the TickFeeder discovers that incoming Excel-sourced data does not conform to the expected mode. When raised as a UfException, INVALID_EXCEL_MODE lets the rest of the pipeline inspect the numeric code and handle the condition consistently—logging, failing fast, or routing to a different data source—just as other centralized codes like HBASE_UPDATE_ERROR and FEEDER_INVALID_ERROR do for their domains.
# file path: ultrafinance/lib/errors.py
INVALID_ACCOUNT = 901
INVALID_ACCOUNT is a numeric error identifier (901) that gets added into the centralized Errors registry so the rest of the pipeline can signal a canonical failure when an account is malformed, missing, or otherwise unusable. Within the project’s error-handling model the UfException wrapper carries one of these numeric codes plus its message, so when PyConfig detects an account configuration problem during setup it raises a UfException with the INVALID_ACCOUNT code and the rest of the system — the execution/simulation adapters, order routing, persistence layers, and monitoring — can inspect that code to handle the failure consistently. Like the previously covered HBASE_UPDATE_ERROR and FEEDER_INVALID_ERROR entries, INVALID_ACCOUNT provides a machine-friendly token that lets diagnostics, logs and higher-level control flow distinguish account validation failures from other error classes and decide the appropriate path (for example, aborting strategy startup or preventing order placement).
# file path: ultrafinance/lib/errors.py
INVALID_METRIC_NAME = 1001
The file registers INVALID_METRIC_NAME in the Errors mapping with the numeric identifier 1001 so the project has a canonical, machine-readable code to represent the failure mode where a metric name is malformed, missing, reserved, or otherwise unacceptable to the metrics and persistence subsystems. In the pipeline that normalizes ticks and drives strategy subscribers, metric names are used to look up computed indicators, map persistence columns, and wire metric definitions supplied via configuration; by giving that failure mode a stable Errors code, components like PyConfig during configuration validation or the metrics registry during runtime can raise a UfException carrying that code and message. Callers anywhere in the backtest or execution simulation can then catch UfException and inspect its code rather than parsing free-form text to determine that the problem was an invalid metric name, just as other canonical errors such as MISSING_SYMBOL, the SHEET_NAME_* codes, and INVALID_ACCOUNT are handled uniformly across the system.
# file path: ultrafinance/lib/errors.py
ACCOUNT_ALEADY_SET = 1002
ACCOUNT_ALEADY_SET registers a canonical, machine-readable error code with the numeric value 1002 in the Errors mapping to represent the specific failure mode where an account has already been established and a second set attempt is attempted. In the context of the ultrafinance pipeline and the centralized error strategy used by Errors and UfException, this code gives PyConfig and other configuration or setup logic a predictable way to signal a duplicate-account condition: when PyConfig’s configuration setting or validation detects that an account is already present it raises a UfException carrying the ACCOUNT_ALEADY_SET code and its human-friendly message so callers (for example callers of configuration APIs, initialization routines, or higher-level orchestration) can inspect the exception, distinguish this precise failure from other conditions like MISSING_SYMBOL or SHEET_NAME_EXIST, and handle the duplicate-account case programmatically and consistently across the system.
# file path: ultrafinance/lib/errors.py
ACCOUNT_NOT_SET = 1003
Errors maintains a central registry of machine-friendly error identifiers and numeric codes; the entry ACCOUNT_NOT_SET with numeric value 1003 adds a canonical signal that there is no account configured or assigned where one is required. In the pipeline this code gives a specific, inspectable failure mode distinct from INVALID_ACCOUNT (901) — which you already saw represents a malformed or unusable account — by representing the absence of any account setting at all. PyConfig will surface this condition by raising a UfException that carries the ACCOUNT_NOT_SET code during configuration setting or validation, and any caller in the data acquisition, TickFeeder-driven runtime, strategy, or execution simulation layers can catch the UfException and programmatically detect the ACCOUNT_NOT_SET failure mode to decide the next control flow. Because errors.py is a standalone canonical source of codes, ACCOUNT_NOT_SET becomes the consistent identifier used across the system whenever an operation fails due to a missing account configuration.
# file path: ultrafinance/lib/errors.py
INVALID_RISK_FREE_RETURN = 1004
INVALID_RISK_FREE_RETURN is registered in Errors with the numeric identifier 1004 to serve as the canonical, machine-readable signal that a supplied risk-free return value is invalid. Within the pipeline this lets PyConfig or any configuration/validation layer raise a UfException carrying that specific code when a configured risk-free rate is malformed, out of expected bounds, or otherwise unusable; downstream pieces such as the metrics calculator that computes Sharpe-like statistics can catch or inspect that UfException to respond programmatically. By centralizing this identifier in Errors the system ensures all modules use the same token to represent that precise failure mode, making error handling and automated tests deterministic across the data acquisition, strategy, and metrics layers.
# file path: ultrafinance/lib/errors.py
INVALID_STRATEGY_NAME = 1200
The Errors registry is extended with a canonical entry named INVALID_STRATEGY_NAME assigned the numeric identifier 1200 so the rest of the system has a single, machine-friendly way to signal and inspect the specific failure mode where a configured strategy name is unacceptable. In practice PyConfig’s configuration setting and validation logic — which already raises UfException for other configuration problems — will use INVALID_STRATEGY_NAME when a strategy name is missing, unknown, misspelled, or otherwise not resolvable to a strategy implementation; likewise the strategy loader that wires strategy implementations as subscribers to the TickFeeder can surface the same error code if it cannot map a configuration value to an actual strategy class. By giving this failure mode a dedicated Errors entry, components across the pipeline (configuration, the loader, and any calling code that constructs backtests) can programmatically detect, log, or present a consistent error state for an invalid strategy name just as they do for the other canonical errors like the symbol- and Excel-related codes already registered.
# file path: ultrafinance/lib/errors.py
NONE_ACCOUNT_ID = 1201
The Errors registry is extended with a canonical error code named NONE_ACCOUNT_ID mapped to the numeric identifier 1201; within ultrafinance-master_cleaned this represents the specific failure mode where an account identifier is missing or set to None during configuration or runtime account handling. Code paths that validate configuration or construct account objects — notably PyConfig during its setting and validation phase and any account-management or execution components that require a concrete account id — will raise a UfException carrying the NONE_ACCOUNT_ID code so callers can programmatically detect and distinguish the “no account id provided” condition. Like the previously-added identifiers such as MISSING_SYMBOL and the Excel-related SHEET_NAME_* and INVALID_EXCEL_MODE errors, NONE_ACCOUNT_ID gives the pipeline a stable, machine-friendly key that the rest of the system can inspect, log, or map to user-facing messages for consistent error handling.
# file path: ultrafinance/lib/errors.py
NONE_TRADING_CENTER = 1202
The Errors registry is extended with a canonical error named NONE_TRADING_CENTER assigned the numeric identifier 1202 so the rest of the system can signal the specific failure mode where no trading center (exchange/venue) is provided or resolvable. In practice PyConfig will use this code when validating configuration and discovering that a required trading-center setting is missing or unspecified, by raising a UfException that carries the NONE_TRADING_CENTER identifier and its human-readable message. Downstream pieces—data access modules, the TickFeeder normalization layer, and strategy initialization—can catch that UfException and inspect the error code to deterministically distinguish a missing trading-center condition from other failures (similar to how the earlier Excel and account error codes are centralized), because trading-center information is needed to interpret symbols, trading hours, tick rules and other exchange-specific behavior in the backtest runtime.
# file path: ultrafinance/lib/errors.py
INVALID_SYMBOLS = 1203
This line registers the canonical error identifier INVALID_SYMBOLS with the numeric code 1203 in the centralized Errors mapping so the rest of the ultrafinance-master_cleaned pipeline can signal and detect the specific failure mode where a provided set of market symbols is invalid (for example malformed entries, unsupported symbol formats, or a batch that fails validation). In practice, components that validate configuration or input—such as PyConfig during configuration setting and validation, the Excel or other data access modules when parsing symbol lists, or the TickFeeder when normalizing incoming tick subscriptions—will raise a UfException carrying this code so callers can programmatically distinguish an invalid-symbols condition from other symbol-related errors (remember MISSING_SYMBOL is used for a single unresolved symbol). By centralizing INVALID_SYMBOLS here, the pipeline gains a consistent, inspectable signal that upstream logic, error handlers, and logging can use to make deterministic decisions about failing fast, skipping entries, or surfacing user-facing errors.
# file path: ultrafinance/lib/errors.py
TABLENAME_NOT_SET = 1300
TABLENAME_NOT_SET is added to the Errors registry as the canonical, machine-friendly identifier for the failure mode where a persistence or storage configuration lacks a required table name; it is assigned the numeric code 1300 so callers can programmatically detect this precise condition. In the layered pipeline this maps to the persistence/storage layer and any DAMs that require a target table (for example SQL or HBase backends): when PyConfig performs configuration setting and validation or when a storage module initializes and finds no table name provided, code elsewhere will raise a UfException carrying TABLENAME_NOT_SET so upstream components can inspect the error code and handle the missing-table-name case consistently. This follows the same centralized Errors pattern used for MISSING_SYMBOL and the Excel-related error codes, enabling uniform error handling and clear, inspectable control flow during startup and configuration validation.
# file path: ultrafinance/lib/errors.py
TABLENAME_ALREADY_SET = 1301
In the Errors registry, TABLENAME_ALREADY_SET is added as the canonical machine-friendly identifier for the specific failure mode where a table name has already been established and someone attempts to set or override it later; it is registered with the numeric value 1301. Like MISSING_SYMBOL and the sheet-name errors you already saw, TABLENAME_ALREADY_SET exists so PyConfig or any persistence layer can raise a UfException carrying a consistent code and human-facing message when configuration validation or runtime checks detect an attempt to redefine a table name. In terms of data flow, the detection happens in configuration or storage setup code, an UfException is constructed with TABLENAME_ALREADY_SET and propagated up to callers so orchestration code can inspect the numeric code and decide how to handle the duplicate-table-name condition; on the control-flow side this serves as a guard that prevents changing table identity after it has been set.
# file path: ultrafinance/lib/errors.py
INVALID_SAVER_NAME = 1302
Line 39 registers INVALID_SAVER_NAME with the numeric code 1302 in the Errors mapping so the pipeline has a dedicated, machine-identifiable signal for the failure mode where a configured persistence “saver” identifier cannot be resolved. In the ultrafinance-master_cleaned architecture the persistence layer exposes different saver implementations (HBase, SQL, Excel, etc.) that are selected by name in configuration; when PyConfig validates configuration or when the persistence factory attempts to instantiate a saver and the provided name does not match any known saver, the system will raise a UfException carrying this error code. That lets callers and error handlers distinguish a missing or mistyped saver name from the related table-name errors already registered (TABLENAME_NOT_SET and TABLENAME_ALREADY_SET) and allows downstream components — such as the persistence factory or repository code that consumes the configuration — to programmatically inspect the error and react accordingly.
# file path: ultrafinance/lib/errors.py
def __str__(self):
return repr(self.__errorMsg)
UfException.str returns the programmer-oriented representation of the private error message that UfException.init stored when the exception was created. In practice that means when the exception instance is coerced to text for logging or printing, the string returned is the repr-style form of the internal error message attribute rather than a formatted, multi-line diagnostic. This keeps the textual output concise and unambiguous while the numeric error code remains accessible via getCode and the more verbose getMsg (which includes traceback information) remains available for consumers that need it. Because errors.py centralizes error semantics and PyConfig raises UfException for configuration problems, this implementation ensures that the pipeline can both display a stable, inspectable message and programmatically examine the canonical error codes you previously saw registered (for example NONE_ACCOUNT_ID, NONE_TRADING_CENTER, INVALID_SYMBOLS, TABLENAME_NOT_SET, TABLENAME_ALREADY_SET).
# file path: ultrafinance/lib/errors.py
def getCode(self):
return self.__error
UfException.getCode is the simple accessor that returns the numeric error identifier held by the UfException instance. When UfException is constructed, init stores a machine-friendly code into a private attribute and a human-facing message into another attribute; getCode exposes that stored numeric code so callers can make programmatic decisions based on the centralized Errors registry entries (for example the canonical identifiers like NONE_ACCOUNT_ID or TABLENAME_NOT_SET that we already reviewed). getCode performs no formatting, no side effects and no lookup — it hands back the raw error code that PyConfig or other pipeline components set when they raised the exception. Where str and getMsg supply the readable message and traceback for logging or display, getCode supplies the complementary, inspectable integer that the rest of the ultrafinance-master_cleaned pipeline uses to route error handling deterministically.
# file path: ultrafinance/lib/errors.py
def getMsg(self):
return "%s: %s" % (self.__errorMsg, traceback.format_exc(5))
UfException.getMsg builds a human-readable, single-line representation of the exception by reading the error message that was saved when UfException was constructed and appending a short, formatted traceback to give context about where the failure originated. Because errors.py is the centralized registry and UfException is the exception object used throughout the pipeline (for example when PyConfig raises configuration failures identified by codes like NONE_ACCOUNT_ID or TABLENAME_NOT_SET), getMsg provides a compact payload that callers, logs, or monitoring components can consume: the stored error message from UfException.init combined with a traceback snippet (the formatted exception stack limited to a few frames) so downstream code can both programmatically inspect the error via getCode or str and also surface a contextual diagnostic via getMsg. There is no branching or additional state mutation in getMsg; it simply reads the instance message and produces the combined string for reporting.
# file path: ultrafinance/model/__init__.py
TICK_FIELDS = ['time', 'open', 'high', 'low', 'close', 'volume']
TICK_FIELDS is a module-level constant that declares the canonical attributes every Tick instance is expected to carry for the rest of the pipeline to interoperate: a timestamp plus the standard bar prices (open, high, low, close) and volume. By naming that schema in ultrafinance.model.init, the Tick and Quote constructors, their serialization/validation routines, and the TickFeeder normalization logic all have a single authoritative contract to map incoming DAM records (Yahoo/Google, Excel, HBase, SQL) into the backtest runtime. Strategies that use pyTaLib and the execution simulation, as well as persistence layers and components like Account and TradingCenter when they create or evaluate orders, rely on those fields being present and consistently ordered, so TICK_FIELDS enforces a uniform data shape across adapters and savers and makes validation and downstream indicator calculations predictable.
# file path: ultrafinance/model/__init__.py
QUOTE_FIELDS = ['time', 'close', 'volume', 'low', 'high']
QUOTE_FIELDS is the canonical list of attribute names the Quote model expects and uses when normalizing, validating, and serializing market quote data. Downstream components like the DAMs and the TickFeeder map incoming market data into these attributes — time, close, volume, low, and high — so the Quote constructor and its serializers can rely on a consistent shape and ordering for persistence, metrics computation, and strategy consumption. By centralizing those attribute names in QUOTE_FIELDS, the model layer enforces a uniform contract between external data sources, the backtest runtime, and savers so that creation of Quote instances, presence checks during validation, and column selection for storage all behave predictably across the pipeline.
# file path: ultrafinance/model/__init__.py
class Tick(object):
def __init__(self, time, open, high, low, close, volume):
self.time = time
self.open = float(open)
self.high = float(high)
self.low = float(low)
self.close = float(close)
self.volume = int(volume)
def __str__(self):
return json.dumps({"time": self.time,
"open": self.open,
"high": self.high,
"low": self.low,
"close": self.close,
"volume": self.volume})
@staticmethod
def fromStr(string):
d = json.loads(string)
return Tick(d['time'], d['open'], d['high'],
d['low'], d['close'], d['volume'], d['adjClose'])
Within the pipeline, Tick is the canonical in-memory representation of a single time-series observation (a bar/tick) that DAMs produce and the backtest runtime consumes. DAM implementations and network parsers create Tick instances from raw inputs, Tick normalizes those inputs into consistent numeric types, and other runtime components (TickFeeder, TradingCenter, Account, strategy subscribers) read those normalized fields to make timing and execution decisions.
When a Tick is constructed via its constructor, it stores the time as-is and coerces open, high, low, and close into floating point numbers and volume into an integer. The purpose of those conversions is normalization: Excel/CSV/HTTP sources often supply numeric values as strings, and Tick makes sure downstream logic—indicator calculations, order matching in TradingCenter.isOrderMet, account value computations—can rely on numeric operations without repeated parsing. Tests and DAMs exercise that normalization by instantiating Tick with string-valued fields so the backtest sees consistent numeric types.
Tick provides a string serialization method that emits the Tick as JSON containing time, open, high, low, close, and volume. That serialization is used when the system needs to persist or transmit a tick as text and later reconstruct it. The static deserializer parses a JSON string and reconstructs a Tick by extracting the time, open, high, low, close, volume and adjClose entries from the parsed object and creating a new Tick instance from those values; callers such as savers and DAMs use that path to rebuild Tick instances coming from stored text blobs or external services.
Data flows into Tick in several places: ExcelDAM.readTicks and ExcelDAM.__readData produce row arrays that get turned into Tick instances; HBaseDAM and SqlDAM translate database rows into Tick objects so the runtime always operates on the same Tick shape regardless of source; GoogleFinance/GoogleDAM parsing ultimately yields the same Tick representation for the TickFeeder to publish. Downstream, TradingCenter and Account consume Tick fields (for example close and low) when validating and executing orders, and tests like test
# file path: ultrafinance/model/__init__.py
def __init__(self, time, open, high, low, close, volume):
self.time = time
self.open = float(open)
self.high = float(high)
self.low = float(low)
self.close = float(close)
self.volume = int(volume)
As the constructor for the Tick domain model, Tick.init receives the raw fields supplied by data access modules and normalizes them into the runtime representation the pipeline expects: it stores the supplied time value as-is and converts each price field to a floating‑point number and the volume to an integer. Converting open, high, low and close into floats and volume into an int ensures that downstream components—technical indicators, strategy logic subscribed to the TickFeeder, persistence adaptors such as SqlDAM and HBaseDAM, and the JSON serializer exposed by Tick.str—always operate on consistent numeric types rather than raw strings. Because tests and DAM implementations construct Tick instances with textual numeric inputs, these conversions are the gateway that turns external DAM output into validated, arithmetic‑ready data for the rest of the ultrafinance-master_cleaned pipeline.
# file path: ultrafinance/model/__init__.py
def __str__(self):
return json.dumps({"time": self.time,
"open": self.open,
"high": self.high,
"low": self.low,
"close": self.close,
"volume": self.volume})
Tick.str produces a compact JSON-formatted text snapshot of a Tick instance by serializing the canonical fields time, open, high, low, close, and volume into a single string. Because Tick.init already normalized the numeric fields (open/high/low/close to floats and volume to an integer), the string representation is deterministic and ready for downstream uses like logging, persistence, or simple transport between pipeline components. By exposing those exact keys, Tick.str gives a predictable textual contract that ExcelDAM, HBaseDAM, SqlDAM, the TickFeeder, or debugging facilities can rely on when they need a human- or machine-readable snapshot of a tick without pulling individual attributes.
# file path: ultrafinance/model/__init__.py
@staticmethod
def fromStr(string):
d = json.loads(string)
return Tick(d['time'], d['open'], d['high'],
d['low'], d['close'], d['volume'], d['adjClose'])
Tick.fromStr is a static factory that converts a JSON-encoded tick back into the Tick domain object the backtest runtime expects. It parses the incoming string into a dictionary, pulls out the standard tick fields (time, open, high, low, close, volume and adjusted close), and constructs a Tick instance so the rest of the pipeline receives a normalized object rather than raw JSON. It relies on Tick.init to coerce numeric fields into the proper float/int types, so type normalization and basic validation happen at construction time rather than in the deserializer. On the happy path it returns the freshly constructed Tick for use by TickFeeder, strategies and persistence layers; malformed JSON or missing keys will raise exceptions up to the caller since
# file path: ultrafinance/model/__init__.py
"""
class Quote(object):
''' tick class '''
def __init__(self, time, open, high, low, close, volume, adjClose):
''' constructor '''
self.time = time
self.close = float(close)
def __str__(self):
''' convert to string '''
return json.dumps({"time": self.time,
"close": self.close})
@staticmethod
def fromStr(string):
''' convert from string'''
d = json.loads(string)
return Quote(d['time'], d.get('open'), d.get('high'),
d.get('low'), d['close'], d.get('volume'), d.get('adjClose'))
"""
Quote implements a compact domain model for a market quote that mirrors how Tick normalizes incoming market data: its constructor takes the usual time and price fields but in this implementation only stores the time value as supplied and converts the close price into a floating‑point number for the runtime. The instance stringifier produces a minimal JSON snapshot containing just time and close so other parts of the pipeline can persist or transmit a small, consistent representation. The static factory fromStr reverses that serialization by parsing a JSON string into a dictionary and rebuilding a Quote instance using the parsed time and close as required fields while extracting open, high, low, volume, and adjClose via optional dictionary lookups so a Quote can be reconstituted even when some ancillary fields are missing. Overall, Quote here serves the same normalization and serialization role in the pipeline that Tick does for bar observations: normalize external DAM inputs into a predictable in‑memory object and provide a compact, reversible serialized form for persistence and messaging.
# file path: ultrafinance/model/__init__.py
def __init__(self, time, open, high, low, close, volume, adjClose):
self.time = time
self.open = 0 if ("-" == open) else float(open)
self.high = 0 if ("-" == high) else float(high)
self.low = 0 if ("-" == low) else float(low)
self.close = 0 if ("-" == close) else float(close)
self.volume = int(volume)
self.adjClose = adjClose
Quote.init is the constructor that takes the raw market fields supplied by data access modules or persistence layers and normalizes them into the Quote domain model the backtest runtime expects: it stores the provided time value directly on the instance and converts each price field into a numeric value while treating the external placeholder “-” as a missing-price signal by mapping it to zero, converts volume into an integer to preserve integer volume semantics, and preserves adjClose as given so downstream adjustment logic can use it. This mirrors the normalization behavior you saw in Tick.init but adds the explicit “-” → 0 handling for sources that emit a dash for missing prices. The normalized attributes written to the Quote instance are then what Quote.str serializes into a compact JSON snapshot for the rest of the pipeline to consume.
# file path: ultrafinance/model/__init__.py
def __str__(self):
return json.dumps({"time": self.time,
"open": self.open,
"high": self.high,
"low": self.low,
"close": self.close,
"volume": self.volume,
"adjClose": self.adjClose})
Quote.str produces a compact JSON string that represents the canonical state of a Quote instance by serializing the key quote attributes — time, open, high, low, close, volume, and adjClose — into a single text snapshot. Because Quote.init has already normalized those attributes (converting placeholder values and coercing numeric fields), the string contains the cleaned runtime values that other layers expect. That serialized representation mirrors the behavior of Tick.str, providing a consistent, portable format that the persistence, logging, and backtest runtime components can consume when they need a stable textual view of a Quote for storage, comparison, or debug output. It simply returns that JSON snapshot.
# file path: ultrafinance/model/__init__.py
@staticmethod
def fromStr(string):
d = json.loads(string)
return Quote(d['time'], d['open'], d['high'],
d['low'], d['close'], d['volume'], d['adjClose'])
Quote.fromStr is a static factory that turns a JSON-encoded quote string back into the canonical Quote domain object the pipeline expects. It parses the incoming string into a dictionary and then extracts the canonical quote fields time, open, high, low, close, volume and adjClose and hands them to the Quote constructor, which enforces the model’s normalization and typing rules (for example handling missing values and converting numeric strings to floats/ints). Its role mirrors Tick.fromStr: enabling persistence and data-access layers to deserialize stored or transmitted quote payloads into in-memory Quote instances so strategies, Account, TradingCenter and other pipeline components can consume normalized market data.
# file path: ultrafinance/model/__init__.py
TupleQuote = namedtuple('Quote', ' '.join(QUOTE_FIELDS))
TupleQuote creates a lightweight, immutable record type by turning the canonical QUOTE_FIELDS list into a namedtuple called Quote so the data access layer can emit plain, ordered quote records cheaply. By deriving its field names from QUOTE_FIELDS it guarantees the same canonical field order and attribute names that the rest of the pipeline expects, while giving tuple semantics (indexing/iteration) alongside attribute access. SqlDAM uses TupleQuote in places like __sqlToTupleQuote and the readTupleQuotes/readBatchTupleQuotes paths to convert database rows into these compact DTOs instead of instantiating the richer Quote domain object; downstream code can either consume the tuple form directly for simple reads or convert to the full Quote/Tick models when it needs the runtime normalization and validation provided by Quote and Tick.init.
# file path: ultrafinance/model/__init__.py
class Action(object):
SELL = 'sell'
BUY = 'buy'
SELL_SHORT = 'sell_short'
BUY_TO_COVER = 'buy_to_cover'
@staticmethod
def validate(action):
if action not in [Action.BUY, Action.SELL, Action.SELL_SHORT, Action.BUY_TO_COVER]:
raise UfException(Errors.SIDE_TYPE_ERROR, 'Action error: %s is not accepted' % action)
return action
Action enumerates the allowed trade sides and provides a single validation gate so the rest of the pipeline always sees a canonical action value. It defines four named constants for the possible sides: BUY, SELL, SELL_SHORT, and BUY_TO_COVER, and the static validate method checks an incoming action against that fixed set. When Order.init asks for an action, it routes the supplied value through Action.validate; if the value is one of the four accepted names the same value is returned and stored on the Order, otherwise an UfException is raised with the SIDE_TYPE_ERROR code and a descriptive message. That simple membership check is how the model layer normalizes and enforces the business rule that only those four sides are legal, preventing invalid side values from propagating into TradingCenter, Account execution, savers, and strategies downstream. Conceptually Action acts like a lightweight enum and a centralized validation point for action-related control flow in the backtest runtime.
# file path: ultrafinance/model/__init__.py
SELL = 'sell'
SELL is the canonical domain-level identifier for the sell side of an Action or Order defined in ultrafinance.model.init. As a simple string constant it standardizes how the pipeline names and compares sell orders when strategies construct orders, when Account and TradingCenter apply business rules and fills, and when serializers and persistence layers normalize or store order records. By providing a single authoritative token for the sell direction (paired with the matching buy token elsewhere in the model), SELL ensures consistent validation, comparison, and mapping between external DAM representations and the backtest runtime’s order semantics.
# file path: ultrafinance/model/__init__.py
BUY = 'buy'
Within ultrafinance.model.init, the Action model exposes a constant named BUY that serves as the canonical label for a buy-side instruction. Defining BUY creates a single, normalized token that Order constructors, validators, the trading strategy layer, the Account and TradingCenter components, the execution simulator, and persistence code all use when they need to represent, compare, serialize, or validate a buy order. Because external DAMs and savers can provide many textual variants, having Action.BUY as the agreed-upon value ensures the pipeline consistently recognizes a buy intent as orders flow from strategy creation through validation, execution simulation, metric calculation, and storage.
# file path: ultrafinance/model/__init__.py
SELL_SHORT = 'sell_short'
SELL_SHORT is a module-level constant in ultrafinance.model.init that provides the canonical name for a short-selling order type used by Order, Action, and Type. Remember the Tick model we covered earlier represents market observations; SELL_SHORT complements that by representing a trading intent to enter a short position. Strategies set an order’s type to this constant when they want to short, and the TradingCenter, Account, and persistence layers rely on that standardized label for validation, enforcement of business rules (for example margin/shorting permissions), execution routing, and serialization. Practically it participates in the enum-like set of order-type constants that normalizes short-sale semantics across DAMs, the TickFeeder-driven runtime, and storage backends so all components compare against a single, consistent identifier rather than ad-hoc strings.
# file path: ultrafinance/model/__init__.py
BUY_TO_COVER = 'buy_to_cover'
BUY_TO_COVER is the canonical action-type constant the domain model exposes to represent the operation of closing a short position by buying shares back. Placed in ultrafinance.model.init, BUY_TO_COVER serves the same normalization role that QUOTE_FIELDS and Tick play for market data: it gives strategies, Action/Type/Order construction code, Account, TradingCenter, and persistence layers a single, well‑known token to use when an order is intended to cover a short. When a strategy decides to exit a short, it will create an Order that uses BUY_TO_COVER as its type so the model’s validation and serialization logic, and downstream execution/simulation, treat that order according to the buy‑to‑cover semantics rather than relying on ad hoc strings.
# file path: ultrafinance/model/__init__.py
@staticmethod
def validate(action):
if action not in [Action.BUY, Action.SELL, Action.SELL_SHORT, Action.BUY_TO_COVER]:
raise UfException(Errors.SIDE_TYPE_ERROR, 'Action error: %s is not accepted' % action)
return action
Action.validate is the single, centralized gate that guarantees any trade side entering the backtest runtime is one of the domain’s canonical actions (remember the canonical labels BUY, SELL, SELL_SHORT and BUY_TO_COVER). It takes an incoming action value from callers like Order.init, Order.setAction, TradingCenter and Account and checks it against that allowed set; if the value is valid it simply returns it so the caller stores a canonical action, and if the value is not allowed it aborts the flow by raising a UfException populated with Errors.SIDE_TYPE_ERROR and a message that includes the offending value. Conceptually this enforces a business rule early in order construction/validation so downstream components (Account.execute, TradingCenter.validateOrder, strategies placing orders, persistence, etc.) always see a normalized action value and can rely on that invariant when making branching decisions about execution, matching and accounting. The method is implemented as a protective guard clause: the happy path returns the validated action, the error path raises UfException to signal a domain-level side-type violation.
# file path: ultrafinance/model/__init__.py
class Type(object):
MARKET = 'market'
STOP = 'stop'
LIMIT = 'limit'
@staticmethod
def validate(type):
if type not in [Type.MARKET, Type.STOP, Type.LIMIT]:
raise UfException(Errors.SIDE_TYPE_ERROR, 'Type error: %s is not accepted' % type)
return type
ultrafinance.model.init uses Type to represent the domain-level order kinds used across the pipeline; Type declares the canonical labels MARKET, STOP and LIMIT and exposes a static validator so callers can enforce those labels early. when strategies, config loaders or persistence produce an order type, code calls Type.validate to confirm the value is one of the accepted kinds; on the happy path the supplied type value is returned unchanged so downstream code works with a canonical token, and on the error path a UfException is raised using the Errors.SIDE_TYPE_ERROR code to stop invalid types from propagating. because Order.init invokes Type.validate and Account.execute treats market orders specially by replacing the order price with the tick close, Type.validate acts as the gatekeeper that keeps order creation, trading-center validation and execution logic consistent; its static-validator plus constant-holder pattern mirrors Action.validate and provides a lightweight enum-like contract for the rest of the system.
# file path: ultrafinance/model/__init__.py
MARKET = 'market'
The module defines a module-level constant named MARKET that provides the canonical label for market-style order execution within the domain model. Order and Type validation, serialization, and the execution path in TradingCenter, Account and strategy code use this label to normalize incoming orders from DAMs and savers and to drive the backtest behavior associated with immediate, at-market execution across the pipeline.
# file path: ultrafinance/model/__init__.py
STOP = 'stop'
STOP is the domain-level constant that names the stop order category within ultrafinance.model.init; alongside the already-defined BUY, SELL, SELL_SHORT and BUY_TO_COVER labels, STOP provides a single canonical identifier the rest of the pipeline uses when creating, validating and serializing stop-style orders. Its purpose is to ensure that inputs from the data access modules and savers are normalized into a consistent stop order concept that Order and Type instances can rely on, and that Account, TradingCenter and strategy code see the same unambiguous label when enforcing business rules or making execution decisions.
# file path: ultrafinance/model/__init__.py
LIMIT = 'limit'
The LIMIT constant in ultrafinance.model.init is the canonical module-level label for a limit-type order used by the domain models; much like the Action constants (BUY, SELL, SELL_SHORT, BUY_TO_COVER) standardize trade sides, LIMIT standardizes how order type is represented across Order, Type, the TradingCenter, Account logic, strategy code, and persistence. When an Order is constructed or validated the code compares its type against LIMIT to decide type-specific rules — for example, requiring a limit price to be present and driving the matching/execution behavior in the backtest runtime — and when savers or DAMs serialize or normalize external order representations they map their native type strings to this same LIMIT identifier so every layer of the pipeline interprets a limit order consistently.
# file path: ultrafinance/model/__init__.py
@staticmethod
def validate(type):
if type not in [Type.MARKET, Type.STOP, Type.LIMIT]:
raise UfException(Errors.SIDE_TYPE_ERROR, 'Type error: %s is not accepted' % type)
return type
In ultrafinance.model.init, Type.validate is the small canonical gate that ensures any order type entering the runtime is one of the three supported domain-level constants: MARKET, STOP, or LIMIT. When Order.init or Order.setType ask for a type to be accepted, Type.validate checks membership against those constants and if the value is acceptable simply returns it so the caller can persist the canonical value; if the value is not one of the allowed constants it raises a UfException carrying the SIDE_TYPE_ERROR code and an explanatory message. This mirrors the same validation pattern used by Action.validate you reviewed earlier: it centralizes the allowed values for a domain concept so downstream components such as TradingCenter, Account.execute (which compares against Type.MARKET) and order lifecycle methods can rely on a single, validated set of type values rather than defensive checks scattered throughout the pipeline.
# file path: ultrafinance/model/__init__.py
class Order(object):
OPEN = 'open'
FILLED = 'filled'
CANCELED = 'canceled'
def __init__(self, accountId, action, type, symbol, share, price = None, orderId = None,
status = OPEN, filledTime = None, executedTime = None):
self.__action = Action.validate(action)
self.__type = Type.validate(type)
self.__orderId = None
self.__status = None
self.accountId = accountId
self.symbol = symbol
self.price = price
self.share = share
self.filledTime = filledTime
self.executedTime = executedTime
self.setOrderId(orderId)
self.setStatus(status)
def setStatus(self, status):
if status not in [Order.OPEN, Order.FILLED, Order.CANCELED]:
raise UfException(Errors.ORDER_TYPE_ERROR, 'Order status error: %s is not accepted' % status)
self.__status = status
def setOrderId(self, orderId):
if self.__orderId is not None:
raise UfException(Errors.ORDER_TYPE_ERROR, 'OrderId already set: %s' % self.orderId)
self.__orderId = orderId
def getOrderId(self):
return self.__orderId
def getStatus(self):
return self.__status
def getAction(self):
return self.__action
def setAction(self, action):
self.__side = Action.validate(action)
def getType(self):
return self.__type
def setType(self, type):
self.__side = Type.validate(type)
def __str__(self):
return json.dumps({'accountId': str(self.accountId), 'action': self.__action, 'type': self.__type, 'symbol': self.symbol,
'price': self.price, 'share': self.share, 'orderId': str(self.orderId), 'status': self.status})
@staticmethod
def fromStr(string):
d = json.loads(string)
return Order(d['accountId'], d['action'], d['type'], d['symbol'], d['share'], d.get('price'),
d.get('orderId'), d.get('status'), d.get('filledTime'), d.get('executedTime'))
type = property(getType, setType)
action = property(getAction, setAction)
orderId = property(getOrderId, setOrderId)
status = property(getStatus, setStatus)
Order is the domain object that represents a trade instruction as it flows from strategies into the trading engine and persistence layers. When an Order is instantiated it immediately normalizes the requested trade side and order kind by calling Action.validate and Type.validate so downstream code always sees the canonical Action and Type labels (recall Action and the module-level BUY/SELL/SELL_SHORT/BUY_TO_COVER constants). The constructor records the accountId, symbol, price, share count and optional timestamps, then delegates to setOrderId and setStatus so the same validation rules are enforced at creation time: setOrderId refuses to overwrite an existing identifier and setStatus only accepts the domain status values OPEN, FILLED or CANCELED and raises UfException on invalid inputs. Order exposes explicit getters and setters for action, type, orderId and status and also maps those accessors to Python properties named action, type, orderId and status so callers like BaseStrategy.placeOrder and TradingCenter can read and assign them through a simple attribute API; the setters call the same Action.validate/Type.validate gates and store the validated value on a private attribute. TradingCenter.placeOrder is the normal place that assigns a persistent identifier to an Order (it creates a UUID and then stores the Order in the open-orders table), and Order.setOrderId prevents a strategy or later code from accidentally clobbering that identifier. Order implements a JSON serialization path: its string conversion emits a JSON document containing account, action, type, symbol, price, share, orderId and status, and Order.fromStr reverses that process by parsing a JSON string and constructing a new Order with the parsed fields, which is used by tests and by any persistence/communication layer that needs to materialize orders. Throughout the pipeline strategies such as PeriodStrategy, OneTraker and SMAStrategy construct Order instances and hand them to the trading engine; Order’s validators and property accessors ensure the trading center and Account receive a normalized, validated instruction.
# file path: ultrafinance/model/__init__.py
def setStatus(self, status):
if status not in [Order.OPEN, Order.FILLED, Order.CANCELED]:
raise UfException(Errors.ORDER_TYPE_ERROR, 'Order status error: %s is not accepted' % status)
self.__status = status
Order.setStatus enforces the domain-level invariants for an Order’s lifecycle inside the core domain models defined in ultrafinance.model.init, so callers across the pipeline (for example when an Order is instantiated or when the TradingCenter/Account mark an order filled or cancelled) cannot put the Order into an unexpected state. When a caller hands a new status value to setStatus, the method checks that the value is one of the canonical lifecycle constants the model accepts (OPEN, FILLED, or CANCELED). If the supplied status is not one of those three, setStatus raises a UfException carrying the ORDER_TYPE_ERROR code and a human-readable message describing the invalid status; UfException records the error code and message for upstream logging/handling. If the status is valid, setStatus stores it into the Order’s private status field so subsequent reads via getStatus (and any persistence or execution logic) see a validated, canonical state. This keeps the Order lifecycle transitions consistent for Account, TradingCenter, strategy code and savers that rely on those canonical states.
