You are grappling with time as if it is something. It's not. Time is an illusion created by memory. Ask yourself if the concept of time would exist without memory.

Time is merely the measure of change. The major equations of physics don't need time to work. With Newton (or Leibniz), we have calculus to give us the instantaneous rate of change which eliminated the time factor as a variable. Time is not real. Time is a conceptual measure of change, extremely useful in daily life, but an illusion nonetheless.

There is no possibility of traveling back in time because there is no place to go. Time does not exist independently of space, so there is no way to travel in time without moving in space. As the movie Interstellar illustrated, you can create a distorting effect where you can move forward in time relative to earth's reference frame, but it is not possible to move backward. If you move in space fast enough and return to your point of origin, you can relativistically distort time forward, but not backward.