Invariant theory here refers to the situation where there is a finite group G acting on a polynomial ring R = k[x1,…,xn] through a linear action on the variables and we wish to explicitly compute (homogeneous) generators for the finitely-generated subring of invariants RG. From a group-theoretic point of view, the action of G on the space of linear forms in the xi is a representation V and R with its G-action represents the direct sum of all of the symmetric powers of this representation. From a geometric point of view, G represents a finite group of automorphisms on a scheme X and computation of the ring of invariants gives the quotient scheme X/G. In fact, the invariant theory package is used for certain special cases by Magma's curve quotient functionality.
The computation of the ring of invariants is in two phases. Firstly, a basis for the primary invariants is computed, using an algorithm of Kemper. The primary invariants are a set of n algebraically-independent (homogeneous) polynomials in RG, which therefore generate a subring R1 over which RG is a finite ring extension. Next, secondary invariants are computed which give a set of (homogeneous) generators for RG as an algebra over R1. Different algorithms are used for this, depending on whether the order of G is coprime to the characteristic of k. The ring RG is defined as a (graded) k-algebra by the full set of primary and secondary invariants and the ideal of algebraic relations between these invariants. There is a function to compute and return this ideal of relations.
The Hilbert series of RG can be found through the computation of the Molien series of G. The latter is computable from representation theory (modular in the characteristic p case, when G should be given as a permutation group) before performing the algebraic computation of polynomial invariants. It is also used in the actual algorithm for computing invariants to check that a full set of generators have been found by comparison with the Hilbert series.
Using quite different algorithms, invariants may be computed for the algebraic linear action of an algebraic subgroup G of the algebraic group of n-by-n matrices GLn. This algebraic linear representation may be defined by giving explicit matrices for the action with entries as rational functions on the dimension n2-affine space in which GLn naturally lies. The algorithm used is Derksen's algorithm, which will not necessarily terminate if G is not a linearly reductive group (when RG may not even be a finitely-generated k-algebra). This is a computationally-heavy algorithm that may take some time to complete, but there are also functions to compute a k-basis for the finite-dimensional space of invariants of any given degree. This function is generally quite fast when computing the invariants of a specific degree and it may be useful for producing the full ring of invariants in cases where the Hilbert series of RG can be determined theoretically.
An interesting special case is for SL2 acting on the polynomial ring with n = 2g + 3 for g ≥ 2, where the linear action is given by the natural action on the space of homogeneous polynomials of degree 2g + 2 in two variables. The projective variety defined by the homogeneous ring of invariants gives the (affine) moduli space of algebraic hyperelliptic curves of genus g on removing the hyperplane corresponding to D = 0, where D is the discriminant function.