The tumor microenvironment (TME) is complex and constantly evolving. This is due, in part, to the crosstalk between tumor cells and the multiple cell types that comprise the TME, which results in a heterogeneous population of tumor cells and TME cells. This review will focus on two stromal cell types, the cancer-associated adipocyte (CAA) and the cancer-associated fibroblast (CAF). In the clinic, the presence of CAAs and CAFs in the TME translates to poor prognosis in multiple tumor types. CAAs and CAFs have an activated phenotype and produce growth factors, inflammatory factors, cytokines, chemokines, extracellular matrix components, and proteases in an accelerated and aberrant fashion. Through this activated state, CAAs and CAFs remodel the TME, thereby driving all aspects of tumor progression, including tumor growth and survival, chemoresistance, tumor vascularization, tumor invasion, and tumor cell metastasis. Similarities in the tumor-promoting functions of CAAs and CAFs suggest that a multipronged therapeutic approach may be necessary to achieve maximal impact on disease. While CAAs and CAFs are thought to arise from tissues adjacent to the tumor, multiple alternative origins for CAAs and CAFs have recently been identified. Recent studies from our lab and others suggest that the hematopoietic stem cell, through the myeloid lineage, may serve as a progenitor for CAAs and CAFs. We hypothesize that the multiple origins of CAAs and CAFs may contribute to the heterogeneity seen in the TME. Thus, a better understanding of the origin of CAAs and CAFs, how this origin impacts their functions in the TME, and the temporal participation of uniquely originating TME cells may lead to novel or improved anti-tumor therapeutics.