Karma is one of the many ideas transferred into Western consciousness that has been altered beyond recognition.
"Bastardized" might be the best word for the situation.
As with beginning to understand anything worth its knowledge, the process must start at the beginning; the origin point.
Karma is a Sanskrit word that essentially translates into one's "action" or "deed"; most commonly understood to be one's "doing".
Anything that "happens to you" is your doing.
Now, with this translation, it is easy to see where the mistaking arises. From the earliest mention of karma in the Vedic texts, it is said that one becomes evil from evil deeds and good from good deeds.
Along with this came the Buddhist tradition a few thousand years later, which, interestingly along with declaring that there was no soul (anatman), also developed the ideas of reincarnation and the cycle of life and death.
One's actions get those particular results. The translation is often in terms of what you sow is what you shall reap; of course the West gets this in the Abrahamic tradition.
But in this, there is no idea of "cosmic justice" that is always keeping every action of yours in account and constantly giving you bill in return.
Instead, you are the one buying the ingredients, cooking the meal, and washing up afterwards. All parts of what you do are exactly that, what you do; they are your doing and your doing only.
You yourself are accountable for your own actions, questions of God or the universe and any of these things doing anything to you are completely irrelevant.
No one can hurt you, but you certainly can hurt yourself.
Questions of physicality are also a part of this. One might be able to see how personal attacks on the conscience can only be effective if you are hurt by them; this being hurt is a two-way street that demands both parties come to a mutual and interactive agreement of give and receive. Physical attacks are actually the same way.
Even though physical harm evokes physical pain, the pain is not "hurt". Really. How many times does the body "feel pain" during the day? God forbid you counted. You must count all the times you stubbed your toe or your finger, the legos you step on, the crick in your neck from sleeping awkwardly, the shower water that was a little too cold, the dress shoes a little too tight, the coffee that burned your tongue, the ice cream that was a little too cold for your teeth,
...and so on.
Pain is not the thing to be concerned about. You certainly cannot have "pleasure" without the pain, since they go together like two sides of the same coin. In fact, they should be considered one and the same. Dividing the two is an act of serious indiscretion and result in you being present only 50% of the time; i.e. not fully there.
Your pain is your pain, your pleasure is your pleasure, and it is all your karma.
Take responsibility for your actions; don't give anyone else the credit.