In principle, yes, and I believe a few small hobby projects have attempted to do this and support specific TVs. However, interest in developing a custom Smart TV platform tends to get siphoned away into a project where the output from your actual platform is displayed on the TV rather than running directly on it. Simply, it’s easier to develop and maintain support across different models.
Why would you develop a custom TV OS that runs on one TV when you could develop it for any mini PC and immediately support all TVs? You’d have to develop your OS to run on each specific TV model which will make it quite hard to reach a critical mass sufficient to attract attention from developers and users alike.
The juice isn’t really worth the squeeze. It’s not like TV vendors are publishing detailed hardware specs and drivers. Writing or even porting an OS is hard. Look at the state of the Android ROM scene, and that’s about as good as it gets when some vendors are actually attempting to open source their drivers. The difficulty is much higher and the interest lower due to the existence of a viable alternative.
With that said, motivated minds have done it anyway. You just need to have the right TV for it.
Don’t worry. You’ll get the rest of the little ones with your feet.