Challenges in thermodynamics: Irreversible processes, nonextensive entropies, and systems without equilibrium states