In case of a finite subset of a group, the subgroup test boils down to showing that the subset is closed under the group operation. This holds, in particular, for the subsets of a finite group.
Q. Given a subset $\Sigma\subseteq S_n$ of size $n$, what is the time complexity of the most efficient deterministic algorithm, which implements the test for $\Sigma$? In particular, is it better than $O(n^3)$?
I will briefly describe a deterministic algorithm with time complexity $O(n^2 \log n)$ for testing whether $S$ (with $|S|=n$) is a transitive (and hence regular) subgroup of $S_n$. Note that, as is usual when analysing permutation group algorithms,we assume a RAM model in which images of points under permutations can be computed in constant time.
We start by finding, for each $k \in \{1,\ldots,n\}$ a permutation in $S$ that maps $1$ to $k$. This takes time $O(n)$ and enables us to identify elements in $S$ in constant time, because we only need to look at the image of $1$ under the element to be tested. Of course if this step fails then $S$ is not a transitive subgroup of $S_n$.
The idea of the algorithm is to define a subset $X$ of $S$ that generates $S$ (assuming that $S$ is a subgroup). We will have $|X| \le \log_2 n$, and we only need to test the products $sx$ for $s \in S$ and $x \in X$ for membership of $S$. Computing this product, identifying it, and checking whether it really is in $S$ takes time $O(n)$, so the whole process takes time $O(n^2 \log n)$. If any of the membership tests fails then $S$ is not a subgroup. (Since the group is finite, we don't need to worry about inverses of elements of $X$.)
The algorithm splits into $|X|$ stages. After stage $k$, we have computed a subset $\{x_1,x_2,\ldots,x_k\}$ of $X$ of size $k$, and a subset $S_k$ of $S$ equal to the subgroup of $S_n$ generated by $X_k$.
If $S_k = S$, then $S$ is a subgroup, and we are done. Otherwise we proceed to Stage $k+1$, find an element $x_{k+1} \in S \setminus S_k$ and adjoin it to $X$, and define $S_{k+1}=S_k$. Then we compute all products of elements of $S_k$ with $x_{k+1}$ and adjoin any new elements found to $S_{k+1}$. The new elements found have to be multiplied by all elements of $X$, so we need to keep a record of which products have been calculated, but that is not difficult. Stage $k+1$ finishes when all products $sx$ with $s \in S_{k+1}$ and $x \in X_{k+1}$ have computed and adjoined to $S_{k+1}$ is necessary.
For the more general problem of testing whether $S$ is just a subgroup (not necessarily transitive) of $S_n$, the identification of elements of $S$ is not so straightforward. In practice this would be done using hashing, which would not add much time to the algorithm. But for hashing you can only realistically estimate an expected time, rather than a worst case time. Many of the best and useful algorithms for permutation groups use randomized techniques, and insisting on deterministic algorithms seems to be an unnecessary restriction.