Elite difficulty is difficult for all classes no matter what, especially in the beginning when you have less choice of abilities/strength of abilities.
Necromancers summons wrecks havoc in Elite later on(while leveling, for end-game they are useless) but just starting out, the first 20 or so levels will be quite a struggle, but after that it should become noticeably faster when you unlock more and more summonable minions.
Monsters with shields and/or Invulnerable/Invincible/Impervious/Armored affix may be very hard to kill in the beginning.
When you reach level 16 you will have access to both the Lick, Dragon and Fury, with those 3 minions + Skeletons+ Zombies you should not have much problems. Also, use "Minion Master" and "Faceless Terror" they are great passive abilities for anyone using minions. And ofc buff them through the +to minion passives
Later on you can respecc with a potion of respecc, they cost 1200 i think it is, and they are unlimited so do not be afraid to experiment with your builds.EDIT:
Use the best 2-handed weapon you can find, Skull Barrage works of off Weapon DPS, so it doesn't matter what kind of weapon you use, the damage will always be calculated from the weapon DPS, so a good 2-handed weapon will make it easier to lvl as well.