The level of detail (LOD) concept for 3D building models, which indicates the degree of closeness between a model and its real-world counterpart, is deeply rooted among the stakeholders in the field of urban research and 3D geoinformation. However, with the increasing use and demand of a wide range of applications, the LOD definition standardized by the City Geography Markup Language (CityGML) standard appears to be generic, potentially resulting in redundancy and inflexibility. To address this issue, we reconsider the LOD concept from an application point of view and suggest a new context-aware heterogeneous LOD modeling paradigm for 3D building models tailored to specific applications. The new proposal challenges the original homogeneous generic modeling logic and instead adopts a bottom-up approach, putting the focus on the building components rather than on the building itself, resulting in models that may lead to a better fitness for use. In this paper, we first specify a number of discrete LODs for building component models, called CLODs, and then assemble them to derive the LODs of building models suited for particular applications, diminishing redundancy and being tailored for a specific application. To obtain the appropriate LOD specification, we introduce five essential evaluation criteria and a series of semantic and geometrically assembled constraints on the CLODs. We implement two experiments, outdoor component selection and indoor furniture simulation, and conclude that the proposed application-driven LOD definition is more suited in the context of particular applications.