A lot of fantasy authors include magic as a component of their story, and it makes sense, in a novel, you can have literally anything happen, but therein lies the rub: The great thing about magic is you can do anything; the problem with magic is it can do anything. Omnipotent magic breaks narrative tension the same way an omnipotent god breaks the problem of evil. If your wizard can do anything, why doesn't he? The reader asks that question every single time you don't use the magic and it pulls them out of the story.
There are many wildly successful novels that handle magic and do so in a way that doesn't break the world, and they all have a common thread: Magic must have limitations. For example, when Christopher Paolini wrote the Eragon series, he introduced the concept of magic having a cost and that cost being that of the user's life force, and by using magic rashly, the wielder could die.
Now, while death is a powerful consequence and limitation of a magic system, it is certainly not appropriate for all books. For example, in the YA series Harry Potter, Rowling decided to put limitations around how magic could be used - specifically, Wizards and Witches had to use a wand, and spells could be countered; in fact, that core idea is central to the plot. Harry Potter also introduces the idea of laws around magic, with real consequences for breaking them.
Another great example of limitations in a magic system comes from Brandon Sanderson, who has perhaps the most comprehensive set of rules around using magic:
An author’s ability to solve conflict with magic is DIRECTLY PROPORTIONAL to how well the reader understands said magic. This rule basically says the reader should understand how the magic was used to solve the problem, and that explanation should be reasonable; otherwise, you end up with a case of Deus Ex Machina
Limitations > Power or the limitations of your magic system are more interesting than its power. Remember, your novel is interesting because of conflict, while readers might like to fantasize about themselves being all powerful, it's not very exciting when your characters are. Even Superman has a weakness after all.
Expand on what you have already, before you add something new. This is true of almost every aspect of your novel (such as characters), but it is especially true of your magic system. Adding new elements to your magic system partway through your book will make your readers feel cheated and will break the tension of the book.
Err on the side of AWESOME. (Technically, the Zeroth Law) Your magic system should strike awe in your reader. Remember, when writing a fantasy novel, the sky is the limit and literally anything is possible. If you have magic in your novel, make it interesting! Make it awesome!
When it comes to actually writing your magic system there are two important tips to keep in mind:
Make sure you outline how your magic system works before writing your novel. This doesn't have to be super detailed, but it should include both its capabilities and limitations. That way, when you're writing your novel, you already have a firm grasp of what you can reach for.
Make notes whenever you use magic in a new or novel way. Your first usage of magic, for example, is certain to be novel, but as you work through your novel, note anytime you break the tenets of your outline, and make sure you can justify why.
In addition, make sure when you introduce magic in your novel, you're explaining these core principles, so your readers understand them. This doesn't mean a prologue about the magic system of the book, as that would be boring. But have characters hit these limitations early on and show their consequences. This satisfies the show, don't tell and also makes your characters feel like they have depth.
Adding a magic system to your novel can be both one of the most exciting things to write and the most exciting things to read. But with any powerful component comes great responsibility and thought. If you are intentional and unique with your magic system, your reader will be engrossed. But if your magic system is too convenient or powerful, your reader might not make it past page 1.
Unfortunately, in real life, there is no magic wand to write your novel. The only difference between a dreamer and an author is the stroke of a pen or the tap of the keys.
