Correction 10.7 · Correction: the in-box Hecke translates of \(\GL_6/\Sp_6\) at \(\sum\mu=4\), and one translate that reaches \((1,1,1,1,0,0)\)

Fix a form \(x\) in the orbit and a basis \(e_1,\dots,e_6\) of \(\cO^6\) with \(x(e_{2p-1},e_{2p})=t^{a_p}\), where \((a_1,a_2,a_3)\) is \((2,0,0)\), \((1,1,0)\), \((2,1,-1)\), \((2,2,-2)\) for \(c_1,\dots,c_4\). As in Theorem 4.9, \((T_\nu\ast\varphi_0)(x)\) counts the lattices \(M\) of relative type \(\nu\) to \(\cO^6\) with \(M=M^*\), where \(M^*=\{v:x(v,M)\subseteq\cO\}\); for such \(M\) the determinant condition gives \(\sum\nu=-\tfrac12\sum\mu=-2\).

1. Reduction to Lagrangians. The three types lie in the box \(t\cO^6\subseteq M\subseteq t^{-1}\cO^6\), and the bound of Theorem 4.9 (integrality on \(t\cO^6\), self-duality inside \(t^{-1}\cO^6\)) gives \(-2\le\mu_i\le2\); with \(\sum\mu=4\) the support is \(\{c_1,\dots,c_4\}\). For \(M=M^*\) in the box, \(A:=t\cO^6+(t^{-1}\cO^6)^*\subseteq M\subseteq A^*\), where, writing \(a(i)\) for the exponent of the pair containing \(e_i\),

$$A=\bigoplus_it^{\min(1,\,1-a(i))}\cO e_i,\qquad A^*=\bigoplus_it^{\max(-1,\,-1-a(i))}\cO e_i .$$

Self-dual lattices in the box correspond to the \(\cO\)-submodules \(L=M/A\) of \(Q=A^*/A\) with \(L=L^\perp\) for the induced perfect pairing.

2. The orbit \(c_4\). Here \(A=A^*=t^{-1}\cO^4\oplus t\cO^2\): a single lattice, of type \((1,1,-1,-1,-1,-1)\), i.e. \(R_3\).

3. The orbit \(c_3\). Here \(A=t^{-1}\cO^2\oplus\cO^2\oplus t\cO^2\) and \(Q\simeq\F_q^4\) with basis \(t^{-1}e_3,t^{-1}e_4,e_5,e_6\); the pairing makes \(H_1=\langle t^{-1}e_3,t^{-1}e_4\rangle\) and \(H_2=\langle e_5,e_6\rangle\) orthogonal hyperbolic planes. Since \(t\) kills \(Q\), the self-dual \(M\) are the Lagrangian planes \(L\), \((q+1)(q^2+1)\) in all. The coordinates \(e_1,e_2\) contribute \((-1,-1)\) to the type; in the coordinates \(e_3,\dots,e_6\) the number of entries \(-1\) is \(r=\dim\operatorname{pr}_{H_1}L=2-\dim(L\cap H_2)\), and since the entries lie in \(\{-1,0,1\}\) and sum to \(0\), the type there is \((1^r,0^{4-2r},(-1)^r)\). Now \(L\cap H_2\) is isotropic in a hyperbolic plane, so \(r\ge1\) and \(R_1\) does not occur. Type \(R_2\) (\(r=1\)): choose the line \(\ell=L\cap H_2\) (\(q+1\) ways) and any line of \(H_1\) (\(q+1\) ways); this gives \((q+1)^2\). Type \(R_3\) (\(r=2\)): \(L\) is the graph of a map \(f:H_1\to H_2\) with \(\langle fu,fv\rangle_2=-\langle u,v\rangle_1\), i.e. \(\det f=-1\) in symplectic bases; this gives \(|\SL_2(\F_q)|=q(q^2-1)\).

4. The orbit \(c_1\). Here \(A=t^{-1}\cO^2\oplus t\cO^4\) and \(A^*=t^{-1}\cO^6\), so \(M=t^{-1}\cO^2\oplus M'\) with \(M'\subset\langle e_3,\dots,e_6\rangle\) self-dual for the unimodular form \(J_{34}+J_{56}\) and \(t\cO^4\subseteq M'\subseteq t^{-1}\cO^4\). Such \(M'\) are the \(g\cO^4\), \(g\in\Sp_4(F)\); the \(\Sp_4\)-relative position \(b=(b_1\ge b_2\ge0)\) has \(b_1\le1\), and the \(\GL_4\)-type of \(M'\) is \((b_1,b_2,-b_2,-b_1)\). Type \(R_1\) corresponds to \(b=(0,0)\): one lattice. Types \(R_2\) and \(R_3\) correspond to \(b=(1,0)\) and \(b=(1,1)\), and Macdonald's formula \(|K't^bK'/K'|=q^{\langle2\rho,b\rangle}W(q^{-1})/W_b(q^{-1})\) for \(K'=\Sp_4(\cO)\) (\(2\rho=(4,2)\), \(W(t)=(1+t)(1+t+t^2+t^3)\), \(W_b(t)=1+t\) in both cases) gives

$$q^4(1+q^{-1}+q^{-2}+q^{-3})=q(q+1)(q^2+1),\qquad q^6(1+q^{-1}+q^{-2}+q^{-3})=q^3(q+1)(q^2+1).$$

(Directly: \(b=(1,1)\) means that \(M'/t\cO^4\) is a free rank-2 Lagrangian submodule of \((\cO/t^2)^4\), a point of the smooth 3-dimensional Lagrangian Grassmannian \(\mathrm{LG}(2,4)\) over \(\cO/t^2\), and there are \(q^3|\mathrm{LG}(2,4)(\F_q)|\) of them.)

5. The orbit \(c_2\). Here \(A=\cO^4\oplus t\cO^2\), \(A^*=t^{-1}\cO^6\), and \(Q=V\oplus W\) with \(V=t^{-1}\cO^4/\cO^4\simeq\F_q^4\) (symplectic) and \(W=t^{-1}\cO^2/t\cO^2\simeq(\cO/t^2)^2\) (with a perfect \(\cO/t^2\)-valued alternating form), \(V\perp W\). Put \(d=\dim(L\cap tW)\). The image of \(M\) in \(t^{-1}\cO^6/\cO^6\) has dimension \(4-d\), so the type has \(4-d\) entries \(-1\) and, by \(\sum\nu=-2\), \(2-d\) entries \(+1\).

• \(d=0\) is impossible: then \(tL\subseteq L\cap tW=0\), so \(L\subseteq V\oplus tW\) is the graph of a linear map \(V\to tW\); since \(\langle tW,tW\rangle=0\), isotropy would force the form on \(V\) to vanish. So \(R_3\) does not occur.
• \(d=2\): \(L\supseteq tW\) gives \(L\subseteq(tW)^\perp=V\oplus tW\), hence \(L=L_V\oplus tW\) with \(L_V\) Lagrangian in \(V\): \((q+1)(q^2+1)\) lattices, of type \(R_1\).
• \(d=1\): choose the line \(\ell=L\cap tW\) (\(q+1\) ways), say \(\ell=\F_q\,tw_0\). Then \(L\subseteq\ell^\perp=V\oplus W_\ell\) with \(W_\ell=\{w\in W:\langle w,\ell\rangle=0\}=\cO w_0+tW\), and \(t\ell^\perp\subseteq\ell\), so every subspace between \(\ell\) and \(\ell^\perp\) is an \(\cO\)-submodule. The quotient \(\ell^\perp/\ell=V\oplus(W_\ell/\ell)\) is a 6-dimensional symplectic \(\F_q\)-space, and \(L\) corresponds to a Lagrangian of it meeting the line \(tW/\ell\) trivially. There are \((q+1)(q^2+1)(q^3+1)-(q+1)(q^2+1)=q^3(q+1)(q^2+1)\) of them, so \(q^3(q+1)^2(q^2+1)\) lattices of type \(R_2\).

6. Part 2. Suppose \(e_2=aR_1+bR_2+cR_3\). The \(c_4\)-entry gives \(c=0\); then the \(c_3\)-entry gives \((q+1)^2b=0\), so \(b=0\); then the \(c_1\)-entry gives \(a=0\), contradicting the \(c_2\)-entry. The order matters: \(R_3\) is nonzero at \(c_3\).

7. Part 3. A lattice of type \(\nu_4\) contains \(\cO^6\), so \(x\) is integral on \(\cO^6\) and \(\mu\ge0\); with \(\sum\mu=4\) only \(c_1\) and \(c_2\) are possible. Unimodularity on \(M=\cO^6+\cO t^{-2}v\) means \(x(v,\cO^6)\subseteq t^2\cO\) (\(x\) is alternating, and the determinant condition holds). At \(c_2\) this forces \(v\in t\cO^6\), impossible for primitive \(v\). At \(c_1\) it forces \(v_3,\dots,v_6\in t^2\cO\) with \(v_1,v_2\) free, so \(M\) corresponds to a point of \(\mathbb P^1(\cO/t^2)\), and there are \(q(q+1)\) of them. Combined with \(R_1=\one_{O_{c_1}}+(q+1)(q^2+1)\one_{O_{c_2}}\) this gives the displayed identity. No count uses the characteristic of the field. \(\square\)

Consequence. For the in-box translates the blocked status of Dead end 4.10 stands, but the label \((1,1,1,1,0,0)\) is reached as soon as one translate from the larger box is added — exactly as \(T_{(2,0,0,0)}\) does for \(\GL_4/\Sp_4\). The run's later theorem Theorem 5.7 gives (SG) for every \(\GL_{2m}/\Sp_{2m}\).

Verified by computation: selfdual_counts.py and selfdual_counts_fast.py (all \(t\)-stable Lagrangians of \(Q\) enumerated for \(q=3,5,7\); all twelve entries agree with the table), gl6_extra_translate.py (\(q=3\): values \(12\) at \(c_1\) and \(0\) at \(c_2\)).