As a younger man I prayed to Jesus to save me but then things got sort of sinful for me for a good long time. I always wondered about that little voice that was making me feel bad about doing bad things. I rarely felt like I had a good time afterwards though I had enjoyed myself at the time. I always felt like I had done wrong. I had been away from the Lord a long time before I finally went back just a few years ago. Is it possible I had been saved as a younger man and this was Him calling me back through all those years?
There are two possible sources for feeling bad after sinning. One is guilt, that comes from Satan, and the other is the conviction of the Holy Spirit. This conviction can be felt by believers and non-believers alike (John 16:8). The way to tell which is happening is to observe your reaction. Guilt is intended to drive you away from God, as it did with Adam in the garden (Genesis 3:10) while the conviction of the Holy Spirit is intended to bring you to the cross for confession and forgiveness. In your case, although it took many years, it appears that what you were feeling was the conviction of the Holy Spirit.
Our Lord Jesus promised never to lose anyone that the Father has entrusted to Him (John 6:37-40). Your experience demonstrates His patient persistence in keeping that promise.