The devil is in the details - Part II

Careful consideration of how factors are constructed, adjusted for biases, and combined could affect investment outcomes significantly.


A common perception is that all quantitative managers are the same: we analyze the same data, use the same factors, and employ the same basic statistical techniques. It follows, then, that we invest using the same generic factor models such as Value, Quality, Momentum, or Size. However, while the underlying concepts may be similar, how one constructs a model and transforms a common dataset could make for a substantial difference in the returns and risk profile of the outcomes.

Man Numeric has always felt that the devil is in the details. Careful consideration of how factors are constructed, adjusted for biases, and combined can potentially affect investment outcomes. There is evidence that accounting for these details may produce higher risk-adjusted returns in various models across different regions of the world.

Value as a case study

From Graham and Dodd’s pioneering publication of Security Analysis in 1934, academic researchers have published literature showing that value investing can consistently generate positive returns over the long run. Consequently, value investing has become the bedrock of many quantitative processes with models systematically identifying cheap stocks.

The Fama-French three-factor model1 is the seminal research in this area. Google Scholar shows 3,520 citations referencing the Fama-French Journal of Finance publication “Size and Book-to-Market Factors in Earnings and Returns”.

The Fama-French High minus Low (HML) factor model uses the book-to-price (B/P) measure to partition value and growth stocks. Figure 1 compares the returns of the high B/P firms (value) against the low B/P firms, and confirms higher returns for the value stocks over time. The Fama-French HML portfolio uses historical book value as the only measure of a firm’s value and compares companies in an absolute sense to all other companies in the universe. We can complement book value with other financials measures (like earnings or cash flow) to get a more robust view of a company’s valuation. We can also refine the comparison to look at valuation relative to other companies in the industry, sector, or country, or even strip out some of the biases that consistently arise when identifying value stocks.

Figure 1. Average Value Weighted Compound Returns of Fama French US Small and Big Value Portfolios vs Average Value Weighted Compound Returns of Fama French US Small and Big Growth Portfolios (January 1999 – June 2016)

Source: Fama French 3 factors2.

As value models become more refined in terms of the value metrics used and the frame of comparison used to conduct the evaluation, we generally observe improved total returns as well as better risk-adjusted returns. Figure 2 examines the historical returns of several value models with increasing levels of sophistication. The details of the different value models include:

  1. Fama-French HML Portfolio – as previously described.
  2. Barra Value Model – calculated by MSCI Barra based on forecast earnings-to-price (E/P), reported E/P, reported B/P, and yield.
  3. Industry-Adjusted (IA) Earnings-to-Price Model – selecting the cheapest stocks based on forward E/P compared to industry peers.
  4. Numeric Proprietary Fair Value Model – uses industry relative valuation with adjustments for growth, quality, and a stock’s volatility.
  5. Numeric Proprietary Composite Value Model – combines Fair Value with several other complementary views of cheapness.
Figure 2. Annualized Returns, Volatility, and IR for Selected Value Models for US Universe3 (January 1999 – June 2016)

Source: MSCI Barra Risk Factors, Numeric Investors LLC, and Fama French 6 Portfolios Formed on Size and Book to Market 2X34.

As evidenced by the results in Figure 2, total and risk-adjusted returns increase as: a) the factors used to measure cheapness expand, b) adjustments are made to the peer group evaluation and persistent biases of the factor are reduced, and, c) different value models are combined. Even a simple industry adjustment on a common factor like E/P can increase return and reduce volatility. Also, while people often think that all value signals are highly correlated and there is little advantage to combining them, the benefits of this diversification can be clearly recognized in Numeric’s Proprietary Composite Value Model’s returns and volatility.

We see a similar story when the same concepts are applied to other regions. In both Europe and Japan (Figure 3), we also see that the risk-adjusted returns increase as the models become more sophisticated.

Figure 3. Annualized Returns, Volatility, and IR for Selected Value Models for Europe and Japan Universes (January 1999 – June 2016)

Source: Fama French 6 European Portfolios Formed on Size and Book-to-Market (2 x 3)5; Fama French 6 Japanese Portfolios Formed on Size and Book-to-Market (2 x 3)6; MSCI Barra Risk Factors, Numeric Investors LLC. Big stocks are those in the top 90% of June market cap for the region, and small stocks are those in the bottom 10%. The B/M breakpoints for big and small stocks in a region are the 30th and 70th percentiles of B/M for the big stocks of the region.

What is driving the difference?

Simple factor models have persistent biases that may not be immediately apparent. Rather than the expected exposure to the desired factor, an investor may be unintentionally focusing on certain industries and gaining exposure to other risk factors if the models are not properly adjusted.

Take the industry-adjusted E/P Model for example. On average, Technology and Health Care companies tend to have higher P/E ratios than their Materials or Industrials counterparts (Table 1). When using an unadjusted P/E factor, a portfolio would be concentrated in the historically lower P/E industries and avoid the higher P/E companies. This is why we see a higher risk-adjusted return for the Industry Adjusted E/P Model in Figure 2 and Figure 3 compared to the two unadjusted models.

Table 1. Monthly Average of Forward P/E for Selected Industries for US Universe (2015)
Industry P/E
Health Care Technology 41.6
Internet Software & Services 40.1
Software 37.6
Health Care Equipment & Supplies 25.6
Containers & Packaging 16.8
Metals & Mining 14.6
Construction & Engineering 13.2
Airlines 10.6

Source: Numeric Investors LLC.

Even after adjusting for industry biases, the E/P signal still has significant exposure to other factors (Figure 4). The Industry Adjusted E/P Model shows a persistent pro-leverage and anti-growth bias. So if you employ an industry-relative value approach using forward earnings, you will be inadvertently exposed to lower growth, more highly levered companies.

Numeric goes through great lengths to seek to identify and eliminate any unwanted biases by neutralizing our models to these exposures. This sets our proprietary models apart from the traditional generic factor models and explains the benefits seen in the above graphics.

Figure 4. Historical factor exposures of the Industry Adjusted E/P Model for US Universe (January 1999 – June 2016)

Source: MSCI Barra Risk Factors and Numeric Investors LLC.

Beyond value

The benefits of adjusting for biases are not limited to value models. We see a similar phenomenon in other stock selection models as well, and we can improve a model’s potential risk-adjusted return by altering the frame of comparison or neutralizing persistent biases in the factor.

To demonstrate we evaluate accrual-based and other Quality models. Figure 5 compares Numeric’s proprietary Composite Quality Model against other generic measures of quality like Earnings Risk7, Barra Leverage, and ROE. Similar to the conclusions we saw in the Value section, our proprietary Composite Quality Model, which adjusts for biases and combines several complementary measures of balance sheet quality, both improves returns and decreases risk.

Figure 5. Annualized Returns, Volatility, and IR for Select US Universe Of 3100 stocks (March 1997 – June 2016)

Source: Numeric Investors LLC.

Data matters too

As we have just discussed, thoughtful model construction is very important to a successful quantitative process. However, the most important detail for any quant is data; as the adage goes garbage in, garbage out. At Numeric we spend considerable effort to ensure the integrity of the data that feeds our models. Our data management falls into two broad categories.

The first category is to ensure that the data we receive from vendors is a) correct; b) reflects the reality of the company’s situation; and c) is consistent across sources.

  • Correct in that the reported financial statements are accurate and reflect the most recent financial reports and corporate actions.
  • Reflecting reality means that the data used in our models portrays an accurate picture of the company’s financial condition. While these issues are mostly manifested in smaller companies, we find a regular need to adjust balance sheet data for very large companies like Apple, where their large holdings in Government Bonds are classified as Long-term Investments, but are in essence Cash & Equivalents.
  • Consistent in that the historical financial statement data are properly lined up with forecast data, price/volume data, and risk data, all of which come from different sources. Numeric uses automated algorithms to perform most of this heavy lifting, but some of the data is nuanced and requires human interpretation.

The second category is variable and model selection. While broad factors like value, momentum, and quality may be powerful over the long run, how you measure those factors depends on the type of company or industry. Using cash flow or gross profits (sales minus cost of goods sold) may be appropriate for an industrial or consumer company, but we believe it is misleading for a financial stock. Inventory management is critical for retailers and semiconductor stocks, but is irrelevant for a software company or a temporary staffing agency. In our view, forecasted earnings per share are generally the best measure for assessing the future profitability of most companies, but funds from operations is the preferred metric for REITs. Using a generic factor across all industries may be transparent and work on average, but it will fail in some industries.

Relevance to smart beta and factor investing

Smart beta and factor investing have recently gained popularity among asset owners as a way to gain access to these concepts that have been widely researched in both academic and practitioner circles. These strategies seek to provide cost-effective access to widely accepted factors that are transparent, liquid, and often low turnover. Asset managers have been quick to meet this expanding demand.

However, those same potential benefits that make Smart Beta attractive (transparency, liquidity, low turnover) could also make these strategies less effective if they fall prey to the many biases discussed in the previous sections. As these biases are eliminated, the strategies become more complex and negate some of the benefits of a Smart Beta portfolio. However, if the ultimate goal of investing is to minimize risk and maximize after-fee returns, we believe the naiveté of simple Smart Beta strategies may not be able to overcome the fee differential of a smartly designed, fully active approach.

We created a portfolio from our proprietary Composite Value Model to compare against the MSCI USA Enhanced Value Index (M2USEV) using the same construction methodology8 (Figure 6). The MSCI USA Enhanced Value Index, which aims to capture exposure to large and mid-cap US equities exhibiting value style characteristics based on P/B, Price-to-Forward Earnings, and Enterprise Value-to-Cash Flow from Operations, is tracked by Smart Beta products such as the iShares Edge MSCI USA Value Factor ETF (VLUE).

Figure 6. Compound Returns of Proprietary Composite Value Model Portfolio vs Compound Returns of MSCI USA Enhanced Value Index (April 23 2013 – June 30 2016)

Source: MSCI Barra Risk Factors and Numeric Investors LLC.

Figure 7 shows the total and risk-adjusted returns of the two strategies from Figure 6. It corroborates our previous analysis and empirically demonstrates the power of Numeric’s adjustments in the context of a Smart Beta Index. The Numeric portfolio does indeed outperform by a wide margin by employing our more sophisticated model. Granted, the management fee is lower for Smart Beta ETFs (currently 15bps for VLUE), but sometimes you get what you pay for.

Figure 7. Annualized Returns, Volatility, and IR of Proprietary Composite Value Model Portfolio vs MSCI USA Enhanced Value Index (April 23 2013 – June 30 2016)

Source: Numeric Investors LLC and Bloomberg.


Numeric strongly believes in the merits of popularly researched factors like value, momentum, quality, and size as well as pursuing a systematic, disciplined approach. One might think that the natural result of this is a belief in Generic Factors and Smart Beta. However, our 27 year history in quantitative investing has taught us that data integrity is crucial and that these factors come with some bias and undesirable exposures. Controlling these can increase the signal of a model and reduce the noise, which can improve returns and/or temper volatility. These detailed adjustments are what set a sophisticated model apart from a simple factor model.

At Numeric, we believe that building a successful investment process means using all the tools at our disposal, both quantitative and qualitative. We are strong believers in the benefits of a systematic quantitative approach, but realize it is only as good as the underlying data and that there is no panacea factor that can evaluate all stocks. The process is as much art as science. The devil really is in the details.


1. Fama, Eugene F. and Kenneth R. French, "Common Risk Factors in the Returns on Stocks and Bonds", Journal of Financial Economics, Vol. 33, No. 1, (February 1993), pp. 3-56.
2. ( Fama French HML Portfolio includes all securities traded on NYSE, AMEX, and NASDAQ.
3. Fama French HML Portfolio includes all securities traded on NYSE, AMEX, and NASDAQ, Industry Adjusted E/P, Barra Value, Fair Value and Composite Value models based off top 1500 US stocks.
7. Variability of earnings calculated as the weighted average of the standard deviations of the FY1-FY3 IBES EPS estimates.
8. Market cap weighted score with sector neutrality implementation and assuming a 50 basis point fee. Note we are not actually managing a portfolio based on this methodology, but it uses our aswas signals.