A reflection is a diagonalisable linear transformation of finite order whose space of fixed points is a hyperplane. A reflection group is a finite dimensional linear group over a field F, which is generated by a finite number of reflections.
There are no restrictions on the field F and there is no requirement for a reflection to be a transformation of order two. However, if F is a real field, every reflection does have order two and there is a much richer theory. In particular, every Coxeter group is a real reflection group (see Chapter COXETER GROUPS).
The books [LT09], [Bro10] or [Kan01] are useful references for complex reflection groups. Standard references for the theory of real reflection groups include [Bou68, Chapters 4, 5, 6] and [Hum90].