Well neem oil's main ingredient is azadirachtin and when the plant absorbs this through its roots ( or leaves ) it stays in the plant tissue. When pests then eat the plant tissue they ingest this chemical as well. There's a lot of different information on what the poison does to them; I've heard it simply makes them unable to breed, but I've also heard it basically drives them crazy and makes them eat each other. Pretty sure there's more truth in between because they don't like it at all.
The soap is just to emulsify the neem oil because it's so hydrophobic, and without it would stay in big globs in the water and be difficult for the plant to take in. When used in a foliar spray it also helps it adhere to the leaf surface, but for the roots it's really just a matter of more evenly distributing the neem oil throughout the mixture. Otherwise it can be so concentrated it could cause root damage or just not be absorbed properly.
Neem works a little bit as a contact killer to by clogging insect pores so there's some function of it as a contact killer, but soap also does this, so if you emulsify with insecticidal soap you basically have a doubly strong contact-killing spray with a systemic benefit afterward. It's a good idea to do both since the foliar spray will knock out a large amount of the living pests ( remember to do another spray in 1-3 days to get the generation that hatched ) and then it will also leave azadirachtin in the plant tissue for future generations of pests to eat and die from.
Really the most important aspect of using neem is the emulsification with soap. I hear people knock it all the time and they are always missing the emulsification step. I use root drenches just to not have to waste that much into a gallon since my garden isn't big, but it will also enable you to give neem treatment late in flower.
