Then only join EFTA and you have trade without free movement of goods. I would ask the question in the other direction. Why is free movement such a problem? There were a lot of concerns before Romania, Bulgaria etc. joined. There was the spectre of millions of people marching towards the old member states. It didn’t happen and most of those that came have jobs. Those without a job have no right to receive welfare, which makes staying in a country unattractive. Also, with the economic rise of Eastern Europe, things are changing. There was a lot of economic improvement since they joined and I can imagine a future where people from Spain, Italy and even Germany will be looking for jobs in Poland or Hungary. Romania/Bulgaria will take longer.
The UK seems to be the only country that makes free movement of people a major concern. I have not heard the same from Sweden, Denmark, Netherlands, Italy… wherever you look. Not even the Lega in Italy is arguing against free movement, at least I never read about it.
The UK can quit its Schengen cooperation. I agree that it looks awkward that the EU insists on free movement for future relations while the UK could quit it right now without leaving - at least that is my naive understanding. I think the reason is that the EU doesn’t want to become a lose framework with different sets of optional rules - or a menu to chose from. If one country starts with free movement, the next wants an exemption for something else and so on.
It is a specially British point of view to pick what they like and dismiss the rest. The UK has a history of opportunistically aligning with other “spaces” and quit that cooperation when interests change. Other countries aren’t partners. They are for “economic extraction”. It’s like a ship that sails to somewhere, lays anchor and leaves when other places are more attractive. The UK docked with the EU when its economy was bad and economies on the continent were growing twice as fast. Later the UK caught up and was seeing the benefit dwindling. In the 1990 during the Clinton boom the UK changed its focus on the USA. When the USA fell out of fashion the UK tried to be best friend with the elites from Russia, Asia and the arabic world. It’s a constant search for the best benefit.
Other EU members always understood that we are European and that is the where we live and where we have to build a “common wealth”. The EU is about integration and most if - if not all - see the benefit. I think even Russia envies us for what we have. There is a naturally better understanding on the continent that a country has neighbours and integration and cooperation is natural and borders are losing importance.
That has always happened and the EU has to listen to them and try to adapt, but also has to figure out what is real and what is simply “libel and slander”.