Prepositional phrase vs Infinitive phrase

Infinitive Phrase

If “to” is followed by a verb, it is an infinitive phrase.

Example: I am going to swim.

SWIM is an action verb.

TO SWIM is an infinitive phrase, not a prepositional phrase.

Prepositional Phrase

If “to” is followed by a noun or pronoun, it is a prepositional phrase.

Example: I am going to swim with my sister.

WITH is the preposition.  

WITH MY SISTER is the prepositional phrase.