High-dimensional data sets are inherently sparse and hence, can be transformed to lower dimensions without losing too much information about the classes. In this paper we propose a new feature ranking algorithm, termed as, class-dependent density-based feature elimination, for binary data sets. CDFE uses a measure termed as, diff-criterion, to estimate the relevance of features. The diff-criterion is a probabilistic measure and assigns weights to features by determining their density value in each class. The class-dependent density-based feature elimination strategy ranks the features using diff-criterion and sorts them according to decreasing relevance.