A common problem when considering safety is confusing averages and "tail" (rare) events in evaluating risks. This helps explain why driver education is largely useless as a way of making cycling safer, and suggests an explanation for why safety is a bigger concern for women and why teenage boys cycle on the pavement.
Both subjective and objective safety when cycling are largely decided by the most dangerous or scary events, not by the average events. Mostly the factors involved in safety are varied, and interact non-linearly, but for a relatively simple example consider passing distance — how much room drivers leave when overtaking people cycling. This clearly follows some kind of long-tailed distribution, with a few very cautious drivers holding back from passing until they can use an entire lane, the vast bulk of drivers passing within half a metre to two metres, and a small number of drivers getting much closer. And it's the latter that put people off cycling completely, or deter them from routes with much motor traffic: the van that hoons past so close its wing-mirror brushes them, leaving them shaken for the rest of the day, if not an actual collision. Cycling may be marginally more pleasant if vehicles pass 1.2m away on average instead of 1m away — a plausible outcome from mass driver-education in this area — but that's going to make a relatively small difference to stress levels and perceived dangers.
Categorising wildly and inventing some numbers, there are perhaps 1% of people (and thus of drivers) who, if not outright psychopaths, just don't care how much they scare or endanger other people. These people only want to avoid hitting people on bicycles because it might slow them down a bit, scratch their paintwork, or involve them in unpleasantness with the police or their insurers. Then there are the 4% (?) of drivers who are, at any one time, distracted and just don't see the bicycle off to the side of the road, or even in front of them. They're falling asleep, they're focused on something else, they're in a real hurry, or they absolutely have to take that phone call. It is these 5% or whatever of drivers who are responsible for almost all the closest passes (and the first 1% for almost all the yelled abuse and threats).
The first group can't usefully be educated but could be constrained by close-passing projects like that run by the West Midlands Police — or by having their licences taken away, or being locked up if they're already driving disqualified. The second group can't really be educated either, and what education/enforcement would help — treating use of a phone while driving in the same way as drink-driving, say — is mostly not at all cycling-specific.
Now what about women cycling? A UK study of passing distances showed that wearing a long wig (in order to look like a woman from behind) resulted in drivers leaving 14cm average extra room when passing. (Wearing a helmet resulted in 8cm closer passing on average, which is one reason the safety benefits of helmets are entirely unclear.) So is cycling safer for women? Unfortunately this doesn't follow at all. On average women may be given more consideration, but what matters (both for perceived safety and many aspects of actual safety) is not the average behaviour but the most unpleasant or dangerous events.
A significant fraction of the most threatening events — the 10cm passes, the yelled abuse — are coming from a tiny fraction of distracted or uncaring/hostile drivers. The people who don't notice the bicycle at all won't notice the rider's gender, while some of the hostile drivers may be more likely to take threatening action (or inaction, such as not moving aside at all when overtaking) against a woman if she is perceived as weak and unlikely to respond by escalation. Also, extreme misogyny may be rare, but if even 1% of the population is actively hostile specifically to women, that could significantly increase the fraction of drivers who would contemplate deliberately hostile or careless action (or inaction) against a woman cycling. A parallel to this is the way women expressing controversial opinions online attract much more aggressive and unpleasant responses than men. (Anecdotally, the women I know seem to get more abuse from drivers than the men, even when cycling with children.)
This is a hypothetical explanation, and I'm not sure how it would be tested — very large experiments would be needed to catch a significant number of one-in-a-thousand ultra-close passing events, while non-experimental evidence runs up against sampling issues, with 90+% of women in most of the UK simply not cycling on roads with motor traffic at all — but it would help explain why women disproportionately rate safety as a concern when cycling, or as a reason they don't cycle at all. Countries with policies and infrastructure that largely separate motor vehicles from bicycles, and where cycling is perceived as a completely normal everyday activity, manage gender-parity in cycling rates — or, indeed, slightly female-skewed rates.
And something similar may hold for minority teenagers cycling on the pavements. Racism may not be widespread in the UK but, as with misogyny, it might only take an extra 1% of drivers being aggressive or careless to double the risk to someone with the wrong coloured skin, style of hair, or clothing. That is then reinforced by perceptions about whose side the police (or bystanders) would take if there is any escalation, making potentially hostile drivers more likely to actually carry out some aggressive action, or not bother taking elementary care. If women cycling with children cop hate on the roads, just imagine what black or swarthy or otherwise marked seventeen year old boys get... It's conceivable that they receive several times as much aggression from motor vehicles as I do, so I cut them some slack when I see them cycling on the pavement.
Note: At least one field experiment confirms that women are close-passed more often.
I think your main point that its only the extreme events not the normal everyday assaults that are a problem is wrong. I know that objectively cycling is much better for me than not cycling, health benefits far outweigh risks, and most 0.5-1.5m passes are not particularly dangerous... yet, I am so sick and tired of them. I have been cycling most of my life. I am a confident and experienced cyclist. I am used to cycle on the road and in traffic. Yet, constantly being passed by vehicles with 0.5-1.5m distance is starting to take a toll on my nerves. I am not sure how much longer I can take it. And I totally believe that a lot of people don't cycle even just because of that '"normal" van passing with around 1 meter distance' events, and not just because of the much rarer crazy person passing at 20cm. If the police did something against close passing, that would at least show that I am not completely forsaken, that alone would mean something to me. Currently, I feel abandoned by council, politicians, and police. In no other area of life would it be acceptable to threaten people like what seems to be accepted regarding people on cycles. Passing me at 30 mph with less than 1 meter distance is in no way acceptable. Doing something like that anywhere else but on the road would get people thrown into prison immediately.
I agree that continuous lower-level threats or aggression are also a problem, and chronic stress is a concern as well as episodes of acute fear. There are also ways in which this is synergistic: background, ever-present dangers such as vehicles passing half a metre away or obstacles in the cyclepath combine to make acute risks worse. If you have a stream of close-passing traffic on one side, a pedestrian suddenly stepping off the pavement onto the cyclepath becomes both harder to avoid and more likely to be lethal. I've written a followup post on safety and stress.
Close-passing actions by police are a good thing, but really the only solution to this is proper separated cycle paths on routes that carry either fast (>20mph) or dense motor traffic.