Fighting the Mother Demon on level 28, The Absolution:

Your best chance for defeating the Mother Demon is to first
acquire the three special "Demon Artifacts" that are hidden
on levels 29, 30 and 31.

Each of these items increases the firepower of Doom 64's
unique laser-type weapon, also known as the Unmaker.  Those
items can also be used as special "keys" that allow you to
close the 3 portals that spawn many additional enemies.

With the laser weapon and all 3 Artifacts, you can quickly
destroy the Mother Demon.

Note: If you don't possess the Artifacts, and don't wish to
replay the game to get them, you can acquire them by using
the passwords you'll find here:
http://classicdoom.com/d64maxpw.htm
You can also use the cheat codes you'll find detailed here:
http://classicdoom.com/doomcode.htm#n64

When facing the Mother Demon without the Artifacts, try to
keep solid objects (like the monster portals) between you
and the guided missiles that she fires.

It is sometimes possible to find positions on the map where
her projectiles will strike intervening objects, while you
can fire past the same objects to hit her.  These positions
will shift depending on her location, and will take some
effort and luck to find.  For video demos on YouTube, see:
Map 28 combat trick: Short/Unmaker example
Map 28 combat trick: Long/Chaingun+SSG example

Remember that strafing and dodging skills are important...
With practice you can even circle an opponent while keeping
a weapon trained on them at all times.

Occasionally power-ups will materialize at the base of the
large central tower.  If you can lure the Mother away from
the tower, you may have an opportunity to grab them.

Finally, use a BFG-9000 (at close range if possible) and
rockets to attack her before resorting to lesser weapons.

Good luck! :)