Next time you find him following you, call 911 immediately, then hang around and pretend to be shopping until LE arrives.
You might want to go to the PD that's local to the places you normally encounter him and ask to speak to an officer to explain the situation.
If you live in a small town, I would be astounded if LE was anything less than happy to help you.
Prepare what to say to 911 in advance, based on your conversation with the local PD.
"This is______, I spoke with officer ________ about being followed by a man and he is following me currently. I am located at _________ and plan to remain inside until the police arrive.".... Or something like that.
Identify the man to LE. Make certain you take a clear photograph of him.
In the presence of LE inform the man that his following you is unwelcome and must stop immediately.
At minimum, you will have a police record on file that can be the basis for a restraining order.
As I'm certain you already know, restraining orders do not stop attacks, so continued diligence is required.
The fact that he followed you to another store and hung around watching you is a HUGE red flag.
You have absolutely NO obligation to remain polite.