The two words have very different meanings. The word alien can mean foreign, unfamiliar or not a citizen, depending on the context. For example, the word can be used to describe a foreign resident in a country, or a plant or animal that has been imported from a different country or climate. It originally comes from the Latin alius, meaning other. In modern English it is often used to mean 'creature from outer space' or similar.
Monster tends to mean something unnatural; usually something exceptionally ugly or strange, or a person who is exceptionally cruel or evil. This word also comes from a Latin root; in this case monstrum, meaning warning or portent.
In modern horror and science fiction both words are used a lot; they are similar in that they are both used to frighten audiences with the idea of something horrible which is 'out there', ie different from us in some way.