Skip to contents

Filter Sorting Data

Source: R/filter_sorting_data.R
filter_sorting_data.Rd

Applies sample construction filters to a data frame before portfolio sorting. Filters are applied in a fixed order: financials exclusion, utilities exclusion, minimum stock price, minimum size quantile, minimum listing age, positive book equity, and positive earnings. An informational message is emitted for each filter that actually removes at least one observation.

Usage

filter_sorting_data(
  data,
  filter_options = NULL,
  data_options = NULL,
  rebalancing_month = NULL,
  quiet = FALSE
)

Arguments

data

A data frame containing the stock-level panel data to be filtered.

filter_options

A list of class tidyfinance_filter_options created by filter_options(). If NULL (the default), the defaults from filter_options() are used (i.e., no filters are applied). The arguments accepted by filter_options() include

  • exclude_financials A logical indicating whether to exclude financial firms (SIC codes 6000–6799). Defaults to FALSE.

  • exclude_utilities A logical indicating whether to exclude utility firms (SIC codes 4900–4999). Defaults to FALSE.

  • min_stock_price A single positive numeric specifying the minimum stock price required to include an observation. NULL (the default) applies no price filter.

  • min_size_quantile A single numeric strictly between 0 and 1 specifying the minimum cross-sectional size quantile (based on lagged market cap) required to include an observation. NULL (the default) applies no size quantile filter. The cutoff is computed from NYSE stocks only; the exchange column (mapped via data_options()) must be present in the data or an error is raised.

  • min_listing_age A single non-negative integer or numeric specifying the minimum number of months a stock must have been listed in CRSP. NULL (the default) applies no listing age filter.

  • exclude_negative_book_equity A logical indicating whether to exclude observations with non-positive book equity. Defaults to FALSE.

  • exclude_negative_earnings A logical indicating whether to exclude observations with non-positive earnings. Defaults to FALSE.

  • timing Either "formation" (the default) to apply the screens only in the formation month when portfolios are rebalanced once a year, or "every_period" to apply them in every month, which can introduce look-ahead bias.

data_options

A list of class tidyfinance_data_options (created via data_options()) specifying column name mappings. The siccd element is used to specify the SIC code column, price is used to specify the price column of the min_stock_price screen, mktcap_lag is used to specify the market capitalization column, date is used to specify the date column, listing_age is used to specify the listing age column, be is used to specify the book equity column, and earnings is used to specify the earnings column. Uses data_options() default if NULL: "siccd" = "siccd", "price" = "prc_lag", "exchange" = "exchange" "mktcap_lag" = "mktcap_lag", "date" = "date", "listing_age" = "listing_age", "be" = "be", and "earnings" = "ib". With rebalancing_month, the id element (default "permno") is used to keep a stock across its holding period.

rebalancing_month

An optional integer between 1 and 12 specifying the month in which portfolios are formed when they are rebalanced once a year (see compute_portfolio_returns()). NULL (the default) means monthly rebalancing. Together with the timing element of filter_options, it determines when the screens apply; see Details.

quiet

A logical indicating whether informational messages should be suppressed. Defaults to FALSE.

Value

The filtered data frame, preserving the class and structure of the input.

Details

Without rebalancing_month, every month is a formation month and each row is screened on its own values. The screens are then free of look-ahead bias only if the screened columns are known at the start of the month: the default price column prc_lag and mktcap_lag are lagged by one month, and accounting variables such as book equity and earnings should be lagged as well (e.g., with join_lagged_values()).

With rebalancing_month, portfolios are formed once a year and held for twelve months. By default (filter_options(timing = "formation")), each screen is evaluated in the formation month only, and a stock that passes all screens keeps all its rows until the next formation month, even if it fails a screen in a holding month. Rows of stocks that fail a screen at formation, and rows before the first formation month, are removed. With timing = "every_period", the screens are evaluated in every month (ex-post) and a warning is issued, because dropping holding-month returns based on information from those months can bias factor returns.

The threshold and sign screens (min_stock_price, min_size_quantile, min_listing_age, exclude_negative_book_equity, exclude_negative_earnings) drop an observation when the screened value is missing; the industry exclusions (exclude_financials, exclude_utilities) keep an observation whose SIC code is missing.

See also

Other portfolio functions: assign_portfolio(), breakpoint_options(), compute_breakpoints(), compute_long_short_returns(), compute_portfolio_returns(), data_options(), filter_options(), implement_portfolio_sort(), portfolio_sort_options()

Examples

data <- data.frame(
  permno = 1:5,
  date = as.Date("2020-01-01"),
  siccd = c(6100, 2000, 4950, 3000, 6500),
  prc_lag = c(5, 0.5, 15, 20, 10)
)

data |>
  filter_sorting_data(
    filter_options = filter_options(
      exclude_financials = TRUE,
      min_stock_price = 1
    )
  )
#> Filter 'exclude_financials': removed 2 observations.
#> Filter 'min_stock_price': removed 1 observation.
#>   permno       date siccd prc_lag
#> 1      3 2020-01-01  4950      15
#> 2      4 2020-01-01  3000      20

# Annual rebalancing in July: the price screen is evaluated in July only,
# so stock 1 is kept in August although its price falls below 1
data_annual <- data.frame(
  permno = rep(1:2, each = 2),
  date = as.Date(rep(c("2020-07-01", "2020-08-01"), 2)),
  prc_lag = c(2, 0.5, 0.5, 2)
)

data_annual |>
  filter_sorting_data(
    filter_options = filter_options(min_stock_price = 1),
    rebalancing_month = 7
  )
#> Filter 'min_stock_price': removed 1 observation.
#> Screens applied in formation month 7: removed 1 holding-month observation of
#> stocks that failed a screen at formation or precede the first formation month.
#>   permno       date prc_lag
#> 1      1 2020-07-01     2.0
#> 2      1 2020-08-01     0.5