If it's not cultivated there, I can't see where it would be any concern. Most of the wild got culled years ago because of its assocation with cannabis. I know some states are starting to allow it, but I'm not certain where it's actually currently in cultivation.
From the sounds of it, you've been growing in it perpetually for a year plus, which means you've had this issue at times of the year when there's no pollen outside, so it has to be something in your place.
It seems odd that it would still be viable though, if this was something from your first grow. Pollen only stays viable for a couple months sealed up and refrigeratored. At room temp and open conditions, it would be far less I would think. I have to admit I'm perplexed.