Semi-Baxter and strong-Baxter:

two relatives of the Baxter sequence

Mathilde Bouvel, Veronica Guerrini, Andrew Rechnitzer, and Simone Rinaldi

January 12, 2018

Abstract

In this paper, we enumerate two families of pattern-avoiding permutations: those avoidingthe vincular pattern 2 41 3, which we call semi-Baxter permutations, and those avoiding thevincular patterns 2 41 3, 3 14 2 and 3 41 2, which we call strong-Baxter permutations. Wecall semi-Baxter numbers and strong-Baxter numbers the associated enumeration sequences.We prove that the semi-Baxter numbers enumerate in addition plane permutations (avoiding2 14 3). The problem of counting these permutations was open and has given rise to severalconjectures, which we also prove in this paper.

For each family (that of semi-Baxter – or equivalently, plane – and that of strong-Baxterpermutations), we describe a generating tree, which translates into a functional equation forthe generating function. For semi-Baxter permutations, it is solved using (a variant of) thekernel method: this gives an expression for the generating function while also proving itsD-finiteness. From the obtained generating function, we derive closed formulas for the semi-Baxter numbers, a recurrence that they satisfy, as well as their asymptotic behavior. Forstrong-Baxter permutations, we show that their generating function is (a slight modificationof) that of a family of walks in the quarter plane, which is known to be non D-finite.

1 Introduction

The purpose of this article is the study of two enumeration sequences, which we call the semi-Baxter sequence and the strong-Baxter sequence. They enumerate, among other objects, familiesof pattern-avoiding permutations closely related to the well-known family of Baxter permutations,and to the slightly less popular one of twisted Baxter permutations, which are both counted bythe sequence of Baxter numbers [23, sequence A001181].

Recall that a permutation π = π1π2 . . . πn contains the vincular1 pattern 2 41 3 if there existsa subsequence πiπjπj+1πk of π (with i < j < k − 1), called an occurrence of the pattern, thatsatisfies πj+1 < πi < πk < πj . Containment and occurrences of the patterns 3 14 2, 3 41 2,2 14 3 and 14 23 are defined similarly. A permutation not containing a pattern avoids it. Baxterpermutations [10, among many others] are those that avoid both 2 41 3 and 3 14 2, while twistedBaxter permutations [15, and references therein] are the ones avoiding 2 41 3 and 3 41 2. We denoteby Av(P ) the family of permutations avoiding all patterns in P .

The two sequences that will be our main focus are first the one enumerating permutationsavoiding 2 41 3, called semi-Baxter permutations, and second the one enumerating permutationsavoiding all three patterns 2 41 3, 3 14 2 and 3 41 2, called strong-Baxter permutations. Remark thata permutation avoiding the (classical) pattern 231 necessarily avoids 2 41 3, 3 14 2 and 3 41 2, andrecall that Av(231) is enumerated by the sequence of Catalan numbers. Therefore, the definitionsin terms of pattern-avoidance and the enumeration results given above can be summarized asshown in Figure 1.

1 Throughout the article, we adopt the convention of denoting by the symbol the elements that are requiredto be adjacent in an occurrence of a vincular pattern, rather than using the historical notation with dashes whereverelements are not required to be consecutive. For instance, our pattern 2 41 3 is sometimes written 2− 41− 3 in theliterature.

1

arX

iv:1

702.

0452

9v3

[m

ath.

CO

] 1

1 Ja

n 20

18

Catalan ≤ strong-Baxter ≤ Baxter ≤ semi-Baxter ≤ factorial

Av(2 41 3, ⊆ Av(2 41 3, 3 14 2) ⊆ allAv(231) ⊆ 3 14 2, Av(2 41 3) ⊆ permutations

3 41 2) ⊆ Av(2 41 3, 3 41 2) ⊆

Figure 1: Sequences from Catalan to factorial numbers, with nested families of pattern-avoidingpermutations that they enumerate.

The focus of this paper is the study of the two sequences of semi-Baxter and strong-Baxternumbers.

We deal with the semi-Baxter sequence (enumerating semi-Baxter permutations) in Section 3.It has been proved in [22] (as a special case of a general statement) that this sequence alsoenumerates plane permutations, defined by the avoidance of 2 14 3. This sequence is referencedas A117106 in [23]. We first give a more specific proof that plane permutations and semi-Baxterpermutations are equinumerous, by providing a common generating tree (or succession rule) withtwo labels for these two families. Basics and references about generating trees can be found inSection 2.

We solve completely the problem of enumerating semi-Baxter permutations (or equivalently,plane permutations), pushing further the techniques that were used to enumerate Baxter per-mutations in [10]. Namely, we start from the functional equation associated with our successionrule for semi-Baxter permutations, and we solve it using variants of the kernel method [10, 21].This results in an expression for the generating function for semi-Baxter permutations, showingthat this generating function is D-finite2. From it, we obtain several formulas for the semi-Baxternumbers: first, a complicated closed formula; second, a simple recursive formula; and third, threesimple closed formulas that were conjectured by D. Bevan [7].

The problem of enumerating plane permutations was posed by M. Bousquet-Melou and S. But-ler in [12]. Some conjectures related to this enumeration problem were later proposed, in particularby D. Bevan [6, 7] and M. Martinez and C. Savage [24]. Not only do we solve the problem ofenumerating plane permutations (or equivalently, semi-Baxter permutations) completely, but wealso prove these conjectures. In addition, from one of these (former) conjectures (relating the semi-Baxter sequence to sequence A005258 of [23], whose terms are sometimes called Apery numbers),we easily deduce the asymptotic behavior of semi-Baxter numbers.

We mention that it has been conjectured in [4] by A. Baxter and M. Shattuck that permutationsavoiding 14 23 are also enumerated by the same sequence, but we have not been able to prove it.

In Section 5, we focus on the study of strong-Baxter permutations and of the strong-Baxtersequence. Again, we provide a generating tree for strong-Baxter permutations, and translate thecorresponding succession rule into a functional equation for their generating function. However,we do not solve the equation using the kernel method. Instead, from the functional equation, weprove that the generating function for strong-Baxter permutations is a very close relative of theone for a family of walks in the quarter plane studied in [8]. As a consequence, the generatingfunction for strong-Baxter permutations is not D-finite. Families of permutations with non D-finitegenerating functions are quite rare in the literature on pattern-avoiding permutations (althoughmostly studied for classical patterns, instead of vincular ones – see the analysis in [1, 19]): thismakes the example of strong-Baxter permutations particularly interesting.

The article is next organized as follows. Section 2 recalls easy facts about the Catalan sequence,and includes basics about generating trees and succession rules. Sections 3, 4 and 5 then focus onthe sequences of semi-Baxter numbers, Baxter numbers, and strong Baxter numbers, respectively,and on the associated families of pattern-avoiding permutations.

2Recall that F (x) is D-finite when there exist k ≥ 0 and polynomials Q(x), Q0(x), . . . , Qk(x) of Q[x] withQk(x) 6= 0 such that Q0(x)F (x) + Q1(x)F ′(x) + Q2(x)F ′′(x) + · · ·+ Qk(x)F (k)(x) = Q(x).

2

2 The Catalan family Av(231) and a Catalan succession rule

Generating trees and succession rules will be important for our work. We give a brief generalpresentation below. Details can be found for instance in [2, 3, 10, 29]. We also review the classicalsuccession rule for Catalan numbers. This rule encodes generating trees for many Catalan families(see [3]), but we will present only a generating tree for the family of permutations avoiding 231,since we will build on it later in this work.

Consider any combinatorial class C, that is to say any set of discrete objects equipped with anotion of size, such that there is a finite number of objects of size n for any integer n. Assumealso that C contains exactly one object of size 1. A generating tree for C is an infinite rooted tree,whose vertices are the objects of C, each appearing exactly once in the tree, and such that objectsof size n are at level n in the tree, that is to say at distance n− 1 from the root (thus, the root isat level 1, its children are at level 2, and so on). The children of some object c ∈ C are obtainedby adding an atom (i.e. a piece of object that makes its size increase by 1) to c. Of course, sinceevery object should appear only once in the tree, not all additions are possible. We should ensurethe unique appearance property by considering only additions that follow some restricted rules.We will call the growth of C the process of adding atoms following these prescribed rules.

Our focus in this section is on Av(231), the set of permutations avoiding the pattern 231: apermutation π avoids 231 when it does not contain any subsequence πiπjπk (with i < j < k) suchthat πk < πi < πj . A growth for Av(132) has been originally described in [29], by insertion of amaximal element, which can be translated by symmetry into a growth for Av(231) by insertion ofa maximal or a leftmost element. In our paper, we are however interested in a different (and notsymmetric) growth for Av(231): by insertion of a rightmost element, in the same flavor as whatis done in [5] for subclasses of Av(231).

Indeed, throughout the paper our permutations will grow by performing “local expansions” onthe right of any permutation π. More precisely, when inserting a ∈ 1, . . . , n+ 1 on the right ofany π of size n, we obtain the permutation π′ = π′1 . . . π

′nπ′n+1 where π′n+1 = a, π′i = πi if πi < a

and π′i = πi+ 1 if πi ≥ a. We use the notation π ·a to denote π′. For instance, 1 4 2 3 ·3 = 1 5 2 4 3.This is easily understood on the diagrams representing permutations (which consist of points inthe Cartesian plane at coordinates (i, πi)): a local expansion corresponds to adding a new pointon the right of the diagram, which lies vertically between two existing points (or below the lowest,or above the highest), and finally normalizing the picture obtained – see Figure 2. These placeswhere new elements may be inserted are called sites of the permutation.

(2) (3)(2)(1)

Figure 2: The growth of a permutation avoiding 231: active sites are marked with ♦ and non-active sites by ×.

Clearly, performing this growth without restriction on the values a would produce a generatingtree for the family of all permutations. To ensure that only permutations avoiding 231 appear inthe tree (that all such appear exactly once being then obvious), insertions are not possible in allsites, but only in those such that the insertion does not create an occurrence of 231 – see Figure 2.Such sites are called active sites. In the considered example, the active sites of π ∈ Av(231) areeasily characterized as those above the largest element πi such that there exists j with i < j andπi < πj . The first few levels of the generating tree for Av(231) are shown in Figure 3 (left).

Of importance for enumeration purposes is the general shape of a generating tree, not thespecific objects labeling its nodes. From now on, when we write generating tree, we intend this

3

3 2 1

2 11 2

1

1 2 3 1 3 2 2 1 3 3 1 2

(1)

(3)(2)(1)(2)(1)

(1) (2)

Figure 3: Two ways of looking at the generating tree for Av(231): with objects (left) and withlabels from the succession rule ΩCat (right).

shape of the tree, without the objects labeling the nodes. A succession rule is a compact way ofrepresenting such a generating tree for a combinatorial class C without referring to its objects, butidentifying them with labels. Therefore, a succession rule is made of one starting label correspond-ing to the label of the root, and of productions encoding the way labels spread in the generatingtree. From the beginnings [29], generating trees and succession rules have been used to deriveenumerative results. As we explain in [14], the sequence enumerating the class C can be recoveredfrom the succession rule itself, without reference to the specifics of the objects in C: indeed, thenth term of the sequence is the total number of labels (counted with repetition) that are producedfrom the root by n − 1 applications of the productions, or equivalently, the number of nodes atlevel n in the generating tree.

From the growth for Av(231) described above, examining carefully how the number of activesites evolves when performing insertions (as done in [29] or [5] for example), we obtain the following(and classical) succession rule associated with Catalan numbers (corresponding to the tree shownin Figure 3, right):

ΩCat =

(1)(k) (1), (2), . . . , (k), (k + 1).

The intended meaning of the label (k) is the number of active sites of a permutation, minus 1. InFigure 2 and similar figures later on, the labels of the permutations are indicated below them.

3 Semi-Baxter numbers

3.1 Definition, context, and summary of our results

Definition 1. A semi-Baxter permutation is a permutation that avoids the pattern 2 41 3.

Definition 2. The sequence of semi-Baxter numbers, (SBn), is defined by taking SBn to be thenumber of semi-Baxter permutations of size n.

The name “semi-Baxter” has been chosen because 2 41 3 is one of the two patterns (namely,2 41 3 and 3 14 2) whose avoidance defines the family of so-called Baxter permutations [16, 20],enumerated by the Baxter numbers [23, sequence A001181]. (Remark that up to symmetry,we could have defined semi-Baxter permutations by the avoidance of 3 14 2, obtaining the samesequence.) Note that 2 41 3 is also one of the two patterns (namely, 2 41 3 and 3 41 2) whoseavoidance defines the family of so-called twisted Baxter permutations [28, 30], also enumerated bythe Baxter numbers.

The first few terms of the sequence of semi-Baxter numbers are

1, 2, 6, 23, 104, 530, 2958, 17734, 112657, 750726, 5207910, 37387881, 276467208, . . .

The family of semi-Baxter permutations already appears in the literature, at least on a fewoccasions. Indeed, it is an easy exercise to see that the avoidance of 2 41 3 is equivalent to thatof the barred pattern 25314, which has been studied by L. Pudwell in [27]. (The definition ofbarred patterns, which is not essential to our work, can be found in [27].) In this work, bymeans of enumeration schemes L. Pudwell suggests that the enumerative sequences of semi-Baxterpermutations and plane permutations (see Definition 4 below) coincide. This conjecture has later

4

been proved as a special case of a general statement in [22, Corollary 1.9(b)]. In Section 3.2 we givean alternative and self-contained proof that plane permutations and semi-Baxter permutations areindeed equinumerous. The sequence enumerating plane permutations has already been registeredon the OEIS [23] as sequence A117106, which is then our sequence (SBn).

The enumeration of plane permutations has received a fair amount of attention in the literature.It first arose as an open problem in [12]. This family of permutations, indeed, was identified asa superset of forest-like permutations, which are thoroughly investigated in [12]. A forest-likepermutation is any permutation whose Hasse graph is a forest – the Hasse graph of a permutationπ of size n is the oriented graph on the vertex set 1, . . . , n, which includes an edge from i to j(for i < j) if and only if π(i) < π(j) and there is no k such that i < k < j and π(i) < π(k) < π(j),and with all edges pointing upward. For instance, the Hasse graphs of permutations 2413 and2143 are depicted in Figure 4. In addition, it shows that the Hasse graph of 2413 is plane (i.e.can be drawn on the plane without any crossing of edges), while the one of 2143 is not.

1

2 4

3 1

3 4

2

Figure 4: The Hasse graphs of permutations 2413 (left) and 2143 (right).

The authors of [12] named plane permutations those permutations whose Hasse graph is plane,characterized them as those avoiding 2 14 3, and called for their enumeration. This enumerativeproblem was studied with a quite experimental perspective, as one case of many, through enu-meration schemes by L. Pudwell in [27]. Then, D. Bevan computed the first 37 terms of theirenumerative sequence [6], by iterating a functional equation provided in [6, Theorem 13.1]. Al-though [6] gives a functional equation for the generating function of semi-Baxter numbers, there isno formula (closed or recursive) for SBn. There is however a conjectured explicit formula, which,in addition, gives information about their asymptotic behavior (see Proposition 15 and Corol-lary 17). Another recursive formula for SBn has been conjectured by M. Martinez and C. Savagein [24], in relation with inversion sequences avoiding some patterns (definition and precise state-ment are provided in Subsection 3.3). Finally, closed formulas for SBn have been conjectured byD. Bevan in [7].

Our results about semi-Baxter numbers are the following. Most importantly, we solve theproblem of enumerating semi-Baxter permutations, as well as plane permutations. We providea common succession rule that governs their growth, presented in Subsection 3.2. Next, in Sub-section 3.3, we show that inversion sequences avoiding the patterns 210 and 100 grow along thesame rule, thereby proving a first formula for SBn and settling a conjecture of [24]. Then, bymeans of standard tools we translate the succession rule into a functional equation whose solutionis the generating function of semi-Baxter numbers. Subsection 3.4 gives a closed expression forthe generating function of semi-Baxter numbers, together with closed, recursive and asymptoticformulas for SBn. The results of this subsection are proved in Subsection 3.5 following the samemethod as in [13]: the functional equation is solved using the obstinate kernel method, a firstclosed formula for SBn is obtained by the Lagrange inversion, the recursive formula follows fromit applying the method of creative telescoping [26], which can then be applied again to prove thatthe explicit formulas for SBn conjectured in [7] are correct. Finally, we prove the formula for SBnconjectured in [6], which in turn gives us the asymptotic behavior of SBn.

3.2 Succession rule for semi-Baxter permutations and plane permuta-tions

Similarly to the case of permutations avoiding 231 described in the introduction, we will providebelow generating trees for semi-Baxter permutations and for plane permutations, where permu-

5

tations grow by insertion of an element on the right. Recall that for any permutation π of size nand for any a ∈ 1, . . . , n+ 1, π · a denotes the permutation π′ of size n+ 1 such that π′n+1 = a,π′i = πi if πi < a and π′i = πi + 1 if πi ≥ a.

Proposition 3. A generating tree for semi-Baxter permutations can be obtained by insertions onthe right, and it is isomorphic to the tree generated by the following succession rule:

Ωsemi =

(1, 1)(h, k) (1, k + 1), . . . , (h, k + 1)

(h+ k, 1), . . . , (h+ 1, k).

Proof. First, observe that removing the last element of a permutation avoiding 2 41 3, we obtaina permutation that still avoids 2 41 3. So, a generating tree for semi-Baxter permutations can beobtained with local expansions on the right.

For π a semi-Baxter permutation of size n, the active sites are by definition the points a (orequivalently the values a) such that π · a is also semi-Baxter, i.e., avoids 2 41 3. The other pointsa are called non-active sites. An occurrence of 2 31 in π is a subsequence πjπiπi+1 (with j < i)such that πi+1 < πj < πi. Obviously, the non-active sites a of π are characterized by the fact thata ∈ (πj , πi] for some occurrence πjπiπi+1 of 2 31. We call a non-empty descent of π a pair πiπi+1

such that there exists πj that makes πjπiπi+1 an occurrence of 2 31. Note that in the case whereπn−1πn is a non-empty descent, choosing πj = πn + 1 always gives an occurrence of 2 31, and it isthe smallest possible value of πj for which πjπn−1πn is an occurrence of 2 31.

To each semi-Baxter permutation π of size n, we assign a label (h, k), where h (resp. k) is thenumber of the active sites of π smaller than or equal to (resp. greater than) πn. Remark thath, k ≥ 1, since 1 and n+ 1 are always active sites. Moreover, the label of the permutation π = 1is (1, 1), which is the root in Ωsemi.

Consider a semi-Baxter permutation π of size n and label (h, k). Proving Proposition 3 amountsto showing that permutations π ·a have labels (1, k+1), . . . , (h, k+1), (h+k, 1), . . . , (h+1, k) whena runs over all active sites of π. Figure 5, which shows an example of semi-Baxter permutation πwith label (2, 2) and all the corresponding π · a with their labels, should help understanding thecase analysis that follows. Let a be an active site of π.

Assume first that a > πn (this happens exactly k times), so that π ·a ends with an ascent. Theoccurrences of 2 31 in π · a are the same as in π. Consequently, the active sites are not modified,except that the active site a of π is now split into two actives sites of π ·a: one immediately belowa and one immediately above. It follows that π · a has label (h+ k+ 1− i, i), if a is the i-th activesite from the top. Since i ranges from 1 to k, this gives the second row of the production of Ωsemi.

Assume next that a = πn. Then, π · a ends with a descent, but an empty one. Similarly to theabove case, we therefore get one more active site in π · a than in π, and π · a has label (h, k + 1),the last label in the first row of the production of Ωsemi.

Finally, assume that a < πn (this happens exactly h − 1 times). Now, π · a ends with a non-empty descent, which is (πn + 1)a. It follows from the discussion at the beginning of this proofthat all sites of π ·a in (a+1, πn+1] become non-active, while all others remain active if they wereso in π (again, with a replaced by two active sites surrounding it, one below it and one above). Ifa is the i-th active site from the bottom, it follows that π · a has label (i, k + 1), hence giving allmissing labels in the first row of the production of Ωsemi.

Definition 4. A plane permutation is a permutation that avoids the vincular pattern 2 14 3 (orequivalently, the barred pattern 21354).

Proposition 5. A generating tree for plane permutations can be obtained by insertions on theright, and it is isomorphic to the tree generated by the succession rule Ωsemi.

Proof. The proof of this statement follows applying the same steps as in the proof of Proposi-tion 3. First, observe that removing the last element of a permutation avoiding 2 14 3, we obtain apermutation that still avoids 2 14 3. So, a generating tree for plane permutations can be obtainedwith local expansions on the right.

6

(4,1)(3,2)(2,3)(2,2) (1,3)

Figure 5: The growth of semi-Baxter permutations (with notation as in Figure 2). Non-emptydescents are represented with bold lines.

For π a plane permutation of size n, the active sites are by definition the values a such thatπ · a is also plane, i.e., avoids 2 14 3. The other points a are called non-active sites. An occurrenceof 2 13 in π is a subsequence πjπiπi+1 (with j < i) such that πi < πj < πi+1. Note that the non-active sites a of π are characterized by the fact that a ∈ (πj , πi+1] for some occurrence πjπiπi+1

of 2 13. We call a non-empty ascent of π a pair πiπi+1 such that there exists πj that makesπjπiπi+1 an occurrence of 2 13. As in the proof of Proposition 3, if πn−1πn is a non-empty ascent,πj = πn−1 + 1 is the smallest value of πj such that πjπn−1πn is an occurrence of 2 13.

Now, to each plane permutation π of size n, we assign a label (h, k), where h (resp. k) is thenumber of the active sites of π greater than (resp. smaller than or equal to) πn. Remark thath, k ≥ 1, since 1 and n+ 1 are always active sites. Moreover, the label of the permutation π = 1 is(1, 1), which is the root in Ωsemi. The proof is concluded by showing that the permutations π · ahave labels (1, k+ 1), . . . , (h, k+ 1), (h+ k, 1), . . . , (h+ 1, k), when a runs over all active sites of π.

If a ≤ πn, π · a ends with a descent, and it follows as in the proof of Proposition 3 that theactive sites of π · a are the same as those of π (with a split into two sites). This gives the secondrow of the production of Ωsemi (the label (h+ k + 1− i, i) for 1 ≤ i ≤ k corresponding to a beingthe i-th active site from the bottom).

If a = πn + 1, π · a ends with an empty ascent, and hence has label (h, k + 1) again as in theproof of Proposition 3.

Finally, if a > πn + 1 (which happens h − 1 times), π · a ends with a non-empty ascent. Thediscussion at the beginning of the proof implies that all sites of π · a in (πn + 1, a] are deactivatedwhile all others remain active. If a is the i-th active site from the top, it follows that π · a haslabel (i, k + 1), hence giving all missing labels in the first row of the production of Ωsemi.

Because the two families of semi-Baxter and of plane permutations grow according to the samesuccession rule, we obtain the following.

Corollary 6. Semi-Baxter permutations and plane permutations are equinumerous. In otherwords, SBn is also the number of plane permutations of size n.

Note that the two generating trees for semi-Baxter and for plane permutations which areencoded by Ωsemi are of course isomorphic: this provides a size-preserving bijection between thesetwo families. It is however not defined directly on the objects themselves, but only referring tothe generating tree structure.

3.3 Another occurrence of semi-Baxter numbers

In this section we provide another occurrence of semi-Baxter numbers that is not in terms ofpattern-avoiding permutations, yet of pattern-avoiding inversion sequences. This occurrence ap-pears as a conjecture in the work of M. Martinez and C. Savage on these families of objects [24].

Recall that an inversion sequence of size n is an integer sequence (e1, e2, . . . , en) satisfying0 ≤ ei < i for all i ∈ 1, 2, . . . , n. In [24] the authors introduce the notion of pattern avoidancein inversion sequences in quite general terms.

Of interest to us here is only the set In(210, 100) of inversion sequences avoiding the patterns210 and 100: an inversion sequence (e1, . . . , en) contains the pattern q, with q = q1 . . . qk, if there

7

exist k indices 1 ≤ i1 < . . . < ik ≤ n such that ei1 . . . eik is order-isomorphic to q; otherwise(e1, . . . , en) is said to avoid q. For instance, the sequence (0, 0, 1, 3, 1, 3, 2, 7, 3) avoids both 210and 100, while (0, 0, 1, 2, 0, 1, 1) avoids 210 but contains 100.

The family ∪nIn(210, 100) can moreover be characterized as follows. A weak left-to-right max-imum of an inversion sequence (e1, e2, . . . , en) is an entry ei satisfying ei ≥ ej for all j ≤ i. Everyinversion sequence e can be decomposed in etop, which is the (weakly increasing) sequence of weakleft-to-right maxima of e, and ebottom, which is the (possibly empty) sequence of the remainingentries of e.

Proposition 7 ([24], Observation 10). An inversion sequence e avoids 210 and 100 if and only ifetop is weakly increasing and ebottom is strictly increasing.

The enumeration of inversion sequences avoiding 210 and 100 is solved in [24], with a sum-mation formula as reported in Proposition 8 below. Let top(e) = max (etop) and bottom(e) =max (ebottom). If ebottom is empty, the convention is to take bottom(e) = −1.

Proposition 8 ([24], Theorem 32). Let Qn,a,b be the number of inversion sequences e ∈ In(210, 100)with top(e) = a and bottom(e) = b. Then

Qn,a,b =

b−1∑i=−1

Qn−1,a,i +

a∑j=b+1

Qn−1,j,b,

with initial conditions Qn,a,b = 0, if n ≤ a, and Qn,a,−1 = n−an

(n−1+a

a

). Hence,

|In(210, 100)| =n−1∑a=0

a−1∑b=−1

Qn,a,b =1

n+ 1

(2n

n

)+

n−1∑a=0

a−1∑b=0

Qn,a,b. (1)

We prove the following conjecture of [24, Section 2.27].

Theorem 9. There are as many inversion sequences of size n avoiding 210 and 100 as planepermutations of size n. In other words |In(210, 100)| = SBn.

Proof. We prove the statement by showing a growth for ∪nIn(210, 100) which can be encodedby Ωsemi. Given an inversion sequence e ∈ In(210, 100), we make it grow by adding a rightmostentry.

Let a = top(e) and b = bottom(e). From Proposition 7, it follows that f = (e1, . . . , en, p) is aninversion sequence of size n+ 1 avoiding 210 and 100 if and only if n ≥ p > b. Moreover, if p ≥ a,then f top comprises p in addition to the elements of etop, and f bottom = ebottom; and if b < p < a,then f top = etop and f bottom comprises p in addition to the elements of ebottom.

Now, we assign to any e ∈ In(210, 100) the label (h, k), where h = a − b and k = n − a.The sequence e = (0) has label (1, 1), since a = top(e) = 0 and b = bottom(e) = −1. Let e bean inversion sequence of In(210, 100) with label (h, k). The labels of the inversion sequences ofIn+1(210, 100) produced adding a rightmost entry p to e are

• (h+ k, 1), (h+ k − 1, 2), . . . , (h+ 1, k) when p = n, n− 1, . . . , a+ 1,

• (h, k + 1) when p = a,

• (1, k + 1), . . . , (h− 1, k + 1) when p = a− 1, . . . , b+ 1,

which concludes the proof that the growth of ∪nIn(210, 100) by addition of a rightmost entry isencoded by Ωsemi.

Remark 10. In addition, in [24, Section 2.27] it is proved that the set In(210, 100) has as manyinversion sequences as the sets In(210, 110), In(201, 100), and In(201, 101). Thus, our proof thatthe semi-Baxter numbers enumerate inversion sequences avoiding 210 and 100 also solves theenumeration problem of exactly four cases of [24, Table 2].

8

3.4 Enumerative results

For h, k ≥ 1, let Sh,k(x) ≡ Sh,k denote the size generating function for semi-Baxter permutationshaving label (h, k). The rule Ωsemi translates into a functional equation for the generating functionS(x; y, z) ≡ S(y, z) =

∑h,k≥1 Sh,ky

hzk.

Proposition 11. The generating function S(y, z) satisfies the following functional equation:

S(y, z) = xyz +xyz

1− y(S(1, z)− S(y, z)) +

xyz

z − y(S(y, z)− S(y, y)) . (2)

Proof. Starting from the growth of semi-Baxter permutations according to Ωsemi we write:

S(y, z) = xyz + x∑h,k≥1

Sh,k((y + y2 + · · ·+ yh)zk+1 + (yh+kz + yh+k−1z2 + · · ·+ yh+1zk)

)= xyz + x

∑h,k≥1

Sh,k

(1− yh

1− yy zk+1 +

1−(yz

)k1− y

z

yh+1zk

)= xyz +

xyz

1− y(S(1, z)− S(y, z)) +

xyz

z − y(S(y, z)− S(y, y)) .

From Proposition 11, a lot of information can be derived about the generating function S(1, 1)of semi-Baxter numbers, and about these numbers themselves. The results we obtain are statedbelow, but the proofs are postponed to Subsection 3.5. A Maple worksheet recording the compu-tations in these proofs is available from the authors’ webpage3.

First, using the “obstinate kernel method” (used for instance in [10] to enumerate Baxterpermutations), we can give an expression for S. We let a denote 1/a, and Ω≥[F (x; a)] denote thenon-negative part of F in a, where F is a formal power series in x whose coefficients are Laurentpolynomials in a. More precisely, if F (x; a) =

∑n≥0,i∈Z f(n, i)aixn, then

Ω≥[F (x; a)] =∑n≥0

xn∑i≥0

f(n, i) ai.

Theorem 12. Let W (x; a) ≡W be the unique formal power series in x such that

W = xa(1 + a)(W + 1 + a)(W + a).

The series solution S(y, z) of eq. (2) satisfies

S(1 + a, 1 + a) = Ω≥ [F (a,W )] ,

where the function F (a,W ) is defined by

F (a,W ) = (1 + a)2 x+(a5 + a4 + 2 + 2a

)xW

+(−a5 − a4 + a3 − a2 − a+ 1

)xW 2 +

(a4 − a2

)xW 3.

(3)

Note that in Theorem 12, W and F (a,W ) are algebraic series in x whose coefficients are Laurentpolynomials in a with rational coefficients. It follows, as in [10, page 6], that S(1 + a, 1 + a) =Ω≥[F (a,W )] is D-finite4, and hence also S(1, 1).

Using the Lagrange Inversion, we can derive from Theorem 12 an explicit but complicatedexpression for the coefficients of S(1, 1), which is reported in Corollary 18 in Subsection 3.5.Surprisingly this complicated expression hides a very simple recurrence, which also appears as aconjecture in [6].

3for instance at http://user.math.uzh.ch/bouvel/publications/Semi-Baxter.mw4By definition, a multivariate generating function F (x), where x = (x1, . . . , xk), is D-finite when it satisfies a

system of linear partial differential equations, one for each i = 1 . . . k, of the form Qi,0(x)F (x) +Qi,1(x) ∂∂xi

F (x) +

Qi,2∂2

∂x2i

F (x) + · · · + Qi,ri∂ri

∂xrii

F (x) = 0, where the Qi,j are polynomials. As stated in [18, Theorem B.3], D-

finiteness is preserved by specialization of the variables.

9

Proposition 13. The numbers SBn are recursively characterized by SB0 = 0, SB1 = 1 and forn ≥ 2

SBn =11n2 + 11n− 6

(n+ 4)(n+ 3)SBn−1 +

(n− 3)(n− 2)

(n+ 4)(n+ 3)SBn−2. (4)

From the recurrence of Proposition 13, we can in turn prove closed formulas for semi-Baxternumbers, which have been conjectured in [7]. These are much simpler than the one given inCorollary 18 by the Lagrange inversion, and also very much alike the summation formula forBaxter numbers (which we recall in Subsection 4.1).

Theorem 14. For any n ≥ 2, the number SBn of semi-Baxter permutations of size n satisfies

SBn =24

(n− 1)n2(n+ 1)(n+ 2)

n∑j=0

(n

j + 2

)(n+ 2

j

)(n+ j + 2

j + 1

)

=24

(n− 1)n2(n+ 1)(n+ 2)

n∑j=0

(n

j + 2

)(n+ 1

j

)(n+ j + 2

j + 3

)

=24

(n− 1)n2(n+ 1)(n+ 2)

n∑j=0

(n+ 1

j + 3

)(n+ 2

j + 1

)(n+ j + 3

j

).

There is actually a fourth formula that has been conjectured in [7], namely

SBn =24

(n− 1)n(n+ 1)2(n+ 2)

n∑j=0

(n+ 1

j

)(n+ 1

j + 3

)(n+ j + 2

j + 2

).

Taking the multiplicative factors inside the sums, it is easy to see (for instance going back to thedefinition of binomial coefficients as quotients of factorials) that it is term by term equal to thesecond formula of Theorem 14.

As indicated in Subsection 3.1, in addition to the formulas reported in Theorem 14 above, twoconjectural formulas for SBn have been proposed in the literature, in different contexts.

The first one has been shown in Subsection 3.3 (eq. (1)). This formula was proved in [24] andits validity for semi-Baxter numbers follows by Theorem 9. The second formula is attributed toM. Van Hoeij and reported by D. Bevan in [6]. This second conjecture is an explicit formula for

semi-Baxter numbers that involves the numbers an =∑nj=0

(nj

)2(n+jj

)(sequence A005258 on [23]).

We will prove in Subsection 3.5 the validity of this conjecture by using the recursive formula forsemi-Baxter numbers (Proposition 13).

Proposition 15 ([6], Conjecture 13.2). For n ≥ 2,

SBn =24

5

(5n3 − 5n+ 6)an+1 − (5n2 + 15n+ 18)an(n− 1)n2(n+ 2)2(n+ 3)2(n+ 4)

Remark 16. With Corollary 18, Theorem 14 and Proposition 15, we get five expressions for thenth semi-Baxter number as a sum over j. Note that although the sums are equal, the correspondingsummands in each sum are not. Therefore, Corollary 18, Theorem 14 and Proposition 15 give fiveessentially different ways of expressing the semi-Baxter numbers. Note however that we are notaware of any combinatorial interpretation of the summation index j, for any of them.

From the formula of Proposition 15, we can derive the dominant asymptotics of SBn.

Corollary 17. Let λ = 12 (√

5− 1). It holds that

SBn ∼ Aµn

n6,

where A = 12π 5−1/4λ−15/2 ≈ 94.34 and µ = λ−5 = (11 + 5

√5)/2.

10

3.5 Enumerative results: proofs

Recall that S(y, z) denotes the multivariate generating function of semi-Baxter permutations. InTheorem 12, we have given an expression for S(1 + a, 1 + a), which we now prove.

Proof of Theorem 12. The linear functional equation of eq. (2) has two catalytic variables, y andz. To solve eq. (2) it is convenient to set y = 1 + a and collect all the terms having S(1 + a, z) inthem, obtaining the kernel form of the equation:

K(a, z)S(1 + a, z) = xz(1 + a)− xz(1 + a)

aS(1, z)− xz(1 + a)

z − 1− aS(1 + a, 1 + a), (5)

where the kernel is

K(a, z) = 1− xz(1 + a)

a− xz(1 + a)

z − 1− a.

For brevity, we refer to the right-hand side of eq. (5) as R(x, a, z, S(1, z), S(1 + a, 1 + a)).The kernel is quadratic in z. Denoting Z+(a) and Z (a) the solutions of K(a, z) = 0 with

respect to z, and Q =√a2 − 2ax− 6a2x+ x2 + 2ax2 + a2x2 − 4a3x, we have

Z+(a) =1

2

a+ x+ ax−Qx(1 + a)

= (1 + a) + (1 + a)2x+(1 + a)3(1 + 2a)

ax2 +O(x3),

Z (a) =1

2

a+ x+ ax+Q

x(1 + a)=

a

(1 + a)x− a− (1 + a)2x− (1 + a)3(1 + 2a)

ax2 +O(x3).

Both Z+ and Z are Laurent series in x whose coefficients are Laurent polynomials in a.However, only the kernel root Z+ is a formal power series in x whose coefficients are Laurentpolynomials in a. So, setting z = Z+, the function S(1 + a, z) is a formal power series in x whosecoefficients are Laurent polynomials in a, and the right-hand side of eq. (5) is equal to zero, i.e.R(x, a, Z+, S(1, Z+), S(1+a, 1+a)) = 0. Note in addition that the coefficients of Z+ are multiplesof (1 + a).

At this point we follow the usual kernel method (see for instance [10]) and attempt to elimi-nate the term S(1, Z+) by exploiting transformations that leave the kernel, K(a, z), unchanged.Examining the kernel shows that the transformations

Φ : (a, z)→(z − 1− a

1 + a, z

)and Ψ : (a, z)→

(a,z + za− 1− az − 1− a

)leave the kernel unchanged and generate a group of order 10.

Among all the elements of this group we consider the following pairs (f1(a, z), f2(a, z)):

[a, z]←→Φ

[z − 1− a

1 + a, z

]←→

Ψ

[z − 1− a

1 + a,z − 1

a

]←→

Φ

[z − 1− a

az,z − 1

a

]←→

Ψ

[z − 1− a

az,

1 + a

a

].

These have been chosen since, for each of them, f1(a, Z+) and f2(a, Z+) are formal power seriesin x with Laurent polynomial coefficients in a. Consequently, they share the property that S(1 +f1(a, Z+), f2(a, Z+)) are formal power series in x. It follows that, substituting each of these pairsfor (a, z) in eq. (5), we obtain a system of five equations, whose left-hand sides are all 0, and withsix unknowns:

11

0 = R(x, a, Z+, S(1, Z+), S(1 + a, 1 + a))

0 = R(x, Z+−1−a

1+a , Z+, S(1, Z+), S(1 + Z+−1−a1+a , 1 + Z+−1−a

1+a ))

0 = R(x, Z+−1

a , Z+−1−a1+a , S(1, Z+−1

a ), S(1 + Z+−1−a1+a , 1 + Z+−1−a

1+a ))

0 = R(x, Z+−1−a

aZ+, Z+−1

a , S(1, Z+−1a ), S(1 + Z+−1−a

aZ+, 1 + Z+−1−a

aZ+))

0 = R(x, Z+−1−a

aZ+, 1+a

a , S(1, 1+aa ), S(1 + Z+−1−a

aZ+, 1 + Z+−1−a

aZ+)).

Eliminating all unknowns except S(1 + a, 1 + a) and S(1, 1 + a), this system reduces (aftersome work) to the following equation:

S(1 + a, 1 + a) +(1 + a)2x

a4S (1, 1 + a) + P (a, Z+) = 0, (6)

where P (a, z) = (−z + 1 + a)(−za4 + z2a4 − za3 + z2a3 − z3a2 − 2a2 + z2a2 + za2 − 4a+ 5az −3az2 + z3a + 3z − z2 − 2)/(za4(z − 1)). Note that the coefficient of S(1, 1 + a) in eq. (6) resultsto be equal to (1 + a)2xa4 only after setting z = Z+ and simplifying the expression obtained.

Now, the form of eq. (6) allows us to separate its terms according to the power of a:

• S(1 + a, 1 + a) is a power series in x with polynomial coefficients in a whose lowest power ofa is 0,

• S(1, 1 + a) is a power series in x with polynomial coefficients in a whose highest power of a

is 0; consequently, we obtain that (1+a)2xa4 S(1, 1 + a) is a power series in x with polynomial

coefficients in a whose highest power of a is −2.

Hence when we expand the series −P (a, Z+) as a power series in x, the non-negative powers of ain the coefficients must be equal to those of S(1 + a, 1 + a), while the negative powers of a come

from (1+a)2xa4 S(1, 1 + a).

Then, in order to have a better expression for P (a, z), we perform a further substitution settingz = w+1+a. More precisely, let W ≡W (x; a) be the power series in x defined by W = Z+−(1+a).Since K(a,W + 1 + a) = 0, the function W is recursively defined by

W = xa(1 + a)(W + 1 + a)(W + a), (7)

as claimed. Moreover, we have the following expression for F (a,W ) := −P (a, Z+):

F (a,W ) = −P (a,W + 1 + a) = (1 + a)2 x+

(1

a5+

1

a4+ 2 + 2a

)xW

+

(− 1

a5− 1

a4+

1

a3− 1

a2− 1

a+ 1

)xW 2

+

(1

a4− 1

a2

)xW 3,

in which the denominator of −P (a,W + 1 + a) is eliminated by substituting in it a factor W forthe right-hand side of eq. (7).

From the expression of S(1 + a, 1 + a) obtained above, the Lagrange inversion allows us toderive an explicit expression for the semi-Baxter numbers, as shown below in Corollary 18. Fromit, we next obtain the simple recurrence of Proposition 13, the conjectured simpler formulas forSBn given in Theorem 14, and the asymptotic estimate of SBn stated in Corollary 17.

12

Corollary 18. The number SBn of semi-Baxter permutations of size n satisfies, for all n ≥ 2:

SBn= 1n−1

∑nj=0

(n−1j

)[(n−1j+1

) [(n+j+1j+5

)+ 2(n+j+1

j

)]+ 2(n−1j+2

) [−(n+j+2j+5

)+(n+j+1j+3

)−(n+j+2j+2

)+(n+j+1

j

)]+ 3(n−1j+3

) [(n+j+2j+4

)−(n+j+2j+2

)]].

Proof. The nth semi-Baxter number, SBn, is the coefficient of xn in S(1, 1), which we denote asusual [xn]S(1, 1). Notice that this number is also the coefficient [a0xn]S(1 + a, 1 + a), and so byTheorem 12 it is the coefficient of a0xn in F (a,W ) = −P (a,W + 1 + a), namely

SBn = [a0xn−1]

((1 + a)2 +

(1

a5+

1

a4+ 2 + 2a

)W +

(− 1

a5− 1

a4+

1

a3− 1

a2− 1

a+ 1

)W 2

+

(1

a4− 1

a2

)W 3

).

This expression can be evaluated from [asxk]W i, for i = 1, 2, 3. Precisely,

SBn = [a5xn−1]W + [a4xn−1]W + 2[a0xn−1]W + 2[a−1xn−1]W − [a5xn−1]W 2 − [a4xn−1]W 2

+[a3xn−1]W 2 − [a2xn−1]W 2 − [a1xn−1]W 2 +[a0xn−1

]W 2 + [a4xn−1]W 3 − [a2xn−1]W 3.

The Lagrange inversion and eq. (7) then prove that

[asxk]W i =i

k

k−i∑j=0

(k

j

)(k

j + i

)(k + j + i

j + s

), for i = 1, 2, 3.

We can then substitute this into the above expression for SBn and, for n ≥ 2, obtain the announcedexplicit formula for the semi-Baxter coefficients SBn setting SBn =

∑n−1j=0 FSB(n, j), where

FSB(n, j) =1

n− 1

(n− 1

j

)[(n− 1

j + 1

)[(n+ j + 1

j + 5

)+ 2

(n+ j + 1

j

)]

+2

(n− 1

j + 2

)[−(n+ j + 2

j + 5

)+

(n+ j + 1

j + 3

)−(n+ j + 2

j + 2

)+

(n+ j + 1

j

)](8)

+3

(n− 1

j + 3

)[(n+ j + 2

j + 4

)−(n+ j + 2

j + 2

)]].

Proof of Proposition 13. From Corollary 18, we can write SBn =∑n−1j=0 FSB(n, j), where the

summand FSB(n, j) given by eq. (8) is hypergeometric, and we prove the announced recurrenceusing creative telescoping [26]. The Maple package SumTools[Hypergeometric][Zeilberger]

implements this approach: using FSB(n, j) as input, it yields

(n+ 5)(n+ 6) · FSB(n+ 2, j)− (11n2 + 55n+ 60) · FSB(n+ 1, j)− n(n− 1) · FSB(n, j)

= GSB(n, j + 1)−GSB(n, j), (9)

whereGSB(n, j) is known as the certificate. It has the additional property thatGSB(n, j)/FSB(n, j)is a rational function of n and j. The expression GSB(n, j) is quite cumbersome and we do notreport it here — it can be readily reconstructed using Zeilberger as done in the Maple worksheetassociated with our paper.

To complete the proof of the recurrence it is sufficient to sum both sides of eq. (9) over j, jranging from 0 to n+ 1. Since the coefficients on the left-hand side of eq. (9) are independent of

13

j, summing it over j gives

(n+ 5)(n+ 6) · SBn+2 − (11n2 + 55n+ 60) · SBn+1 − n(n− 1) · SBn− (11n2 + 55n+ 60) · FSB(n+ 1, n+ 1)− n(n− 1) · (FSB(n, n) + FSB(n, n+ 1)). (10)

Summing the right-hand side over j gives a telescoping series, and simplifies as GSB(n, n + 2) −GSB(n, 0). From the explicit expression of FSB(n, j) and GSB(n, j), it is elementary to check that

FSB(n+ 1, n+ 1) = FSB(n, n) = FSB(n, n+ 1) = GSB(n, n+ 2) = GSB(n, 0) = 0.

Summing eq. (9) therefore gives

(n+ 5)(n+ 6) · SBn+2 − (11n2 + 55n+ 60) · SBn+1 − n(n− 1) · SBn = 0.

Shifting n 7→ n− 2 and rearranging finally gives the recurrence of Proposition 13.

Proof of Theorem 14. For each of the summation formulas given in Theorem 14, we apply themethod of creative telescoping, as in the proof of Proposition 13. In all three cases, this producesa recurrence satisfied by these numbers, and every time we find exactly the recurrence given inProposition 13. Checking that the initial terms of the sequences coincide completes the proof.

Proof of Proposition 15. For the sake of brevity we write A(n) = 5n3 − 5n + 6 and B(n) =5n2 + 15n+ 18 so that the statement becomes

SBn =24(A(n) an+1 −B(n) an)

5(n− 1)n2(n+ 2)2(n+ 3)2(n+ 4). (11)

The validity of eq. (11) is proved by induction on n using Proposition 13 and the following recur-rence satisfied by the numbers an, for n ≥ 1:

an+1 =11n2 + 11n+ 3

(n+ 1)2an +

n2

(n+ 1)2an−1, with a0 = 1, and a1 = 3. (12)

For n = 2, 3, it holds that SB2 = (A(2)a3 −B(2)a2)/2000 = (36 · 147− 68 · 19)/2000 = 2 andSB3 = (A(3)a4 −B(3)a3)/23625 = (126 · 1251− 108 · 147)/23625 = 6.

Then, suppose that eq. (11) is valid for n − 1 and n − 2. In order to prove it for n, considerthe recursive formula of eq. (4) and substitute in it SBn−1 and SBn−2 by using eq. (11). Now,after some work of manipulation and by using eq. (12) we can write SBn as in eq. (11).

Proof of Corollary 17. Applying the main theorem of [25], it follows immediately that

an ∼µn+1/2

2πλn√ν

for λ =

√5− 1

2, µ =

11 + 5√

5

2and ν =

2√

5

3−√

5.

Plugging this expression in the relation

SBn =24

5(n− 1)n2(n+ 2)2(n+ 3)2(n+ 4)

((5n3 − 5n+ 6)an+1 − (5n2 + 15n+ 18)an

),

we see that only the first of these two terms contributes to the asymptotic behavior of SBn, andmore precisely that

SBn ∼ Aµnn−6 for µ as above and A =24µ3/2

2πλ√ν.

The claimed statement then follows noticing that µ = λ−5 and ν =√

5λ2 .

14

4 Baxter numbers

This section starts with an overview of some known results about Baxter numbers. We believe ithelps understanding the relations, similarities and differences between this well-known sequenceand the two main sequences studied in our work (semi-Baxter numbers in Section 3 and strong-Baxter numbers in Section 5). Next, studying two families of restricted semi-Baxter permutationsenumerated by Baxter numbers, we show that Ωsemi generalizes two known succession rules forBaxter numbers.

4.1 Baxter numbers and restricted permutations

Baxter permutations (see [20] among others) are usually defined as permutations avoiding the twovincular patterns 2 41 3 and 3 14 2. Denoting Bn the number of Baxter permutations of size n, thesequence (Bn) is known as the sequence of Baxter numbers. It is identified as sequence A001181in [23] and its first terms are 1, 2, 6, 22, 92, 422, 2074, 10754, 58202, 326240, 1882960, 11140560, . . . .Since [16], an explicit formula for Bn has been known:

for all n ≥ 1, Bn =2

n(n+ 1)2

n∑j=1

(n+ 1

j − 1

)(n+ 1

j

)(n+ 1

j + 1

).

In [10], M. Bousquet-Melou investigates further properties of Baxter numbers. The above formulacan also be found in [10, Theorem 1]. Moreover, using the succession rule reviewed in Proposi-tion 19 below, [10] characterizes the generating function of Baxter numbers as the solution of abivariate functional equation. It is then solved with the obstinate kernel method, implying thatthe generating function for Baxter numbers is D-finite [10, Theorem 4]. Although technical detailsdiffer, it is the same approach than the one we used in Section 3. In the light of our recurrencefor semi-Baxter numbers (see Proposition 13), it is also interesting to note that Baxter numberssatisfy a similar recurrence, reported by R. L. Ollerton in [23], namely

B0 = 0, B1 = 1, and for n ≥ 2, Bn =7n2 + 7n− 2

(n+ 3)(n+ 2)Bn−1 +

8(n− 2)(n− 1)

(n+ 3)(n+ 2)Bn−2.

In addition to Baxter permutations, several combinatorial families are enumerated by Baxternumbers. See for instance [17] which collects some of them and provides links between them. Wewill be specifically interested in a second family of restricted permutations which is also enumeratedby Baxter numbers, namely the twisted Baxter permutations, defined by the avoidance of 2 41 3and 3 41 2 [28, 30].

4.2 Succession rules for Baxter and twisted Baxter permutations

It is clear from their definition in terms of pattern-avoidance that the families of Baxter andtwisted Baxter permutations are subsets of the family of semi-Baxter permutations. Therefore,the growth of semi-Baxter permutations provided in Subsection 3.2 can be restricted to each ofthese families, producing a succession rule for Baxter numbers. In the following, we present thesetwo restrictions, which happen to be (variants of) well-known succession rules for Baxter numbers.This reinforces our conviction that the generalization of Baxter numbers to semi-Baxter numbersis natural.

Let us first consider Baxter permutations. To that effect, recall that a LTR (left-to-right)maximum of a permutation π is an element πi such that πi > πj for all j < i. Similarly, a RTLmaximum (resp. RTL minimum) of π is an element πi such that πi > πj (resp. πi < πj) forall j > i. Following [10, Section 2.1] we can make Baxter permutations grow by adding newmaximal elements to them, which may be inserted either immediately before a LTR maximum orimmediately after a RTL maximum. Giving to any Baxter permutation the label (h, k) where h(resp. k) is the number of its RTL (resp. LTR) maxima, this gives the most classical successionrule for Baxter numbers.

15

Proposition 19 ([10], Lemma 2). The growth of Baxter permutations by insertion of a maximalelement is encoded by the rule

ΩBax =

(1, 1)(h, k) (1, k + 1), . . . , (h, k + 1)

(h+ 1, 1), . . . , (h+ 1, k),

where h (resp. k) is the number of RTL (resp. LTR) maxima.

But note that Baxter permutations are invariant under the 8 symmetries of the square. Con-sequently, up to a 90 rotation, inserting a new maximum element in a Baxter permutation canbe easily regarded as inserting a new element on the right of a Baxter permutation (as we did forsemi-Baxter permutations). Those are then inserted immediately below a RTL minimum or imme-diately above a RTL maximum. Note that, in a semi-Baxter permutation, these are always activesites, so the generating tree associated with ΩBax is a subtree of the generating tree associatedwith Ωsemi. Through the rotation, the interpretation of the label (h, k) of a Baxter permutationis modified as follows: h (resp. k) is the number of its RTL minima (resp. RTL maxima), that isto say of active sites below (resp. above) the last element of the permutation. As expected, thiscoincides with the interpretation of labels in the growth of semi-Baxter permutations accordingto Ωsemi.

Turning to twisted Baxter permutations, specializing the growth of semi-Baxter permutations,we obtain the following.

Proposition 20. A generating tree for twisted Baxter permutations can be obtained by insertionson the right, and it is isomorphic to the tree generated by the following succession rule:

ΩTBax =

(1, 1)(h, k) (1, k), . . . , (h− 1, k), (h, k + 1)

(h+ k, 1), . . . , (h+ 1, k).

Proof. As in the proof of Proposition 3, we let twisted Baxter permutations grow by performinglocal expansions on the right, as illustrated in Figure 6. (This is possible since removing the lastelement in a twisted Baxter permutation produces a twisted Baxter permutation.)

Let π be a twisted Baxter permutation of size n. By definition an active site of π is an elementa such that π · a avoids the two forbidden patterns. Then, we assign to π a label (h, k), where h(resp. k) is the number of active sites smaller than or equal to (resp. greater than) πn. As in theproof of Proposition 3, the permutation 1 has label (1, 1) and now we describe the labels of thepermutations π · a when a runs over all the active sites of π.

If a < πn, then π · a ends with a non-empty descent and, as in the proof of Proposition 3, allsites of π in the range (a+ 1, πn + 1] become non-active in π · a (due to the avoidance of 2 41 3).Moreover, due to the avoidance of 3 41 2, the site immediately above a in π · a also becomes non-active. All other active sites of π remain active in π ·a, hence giving the labels (i, k), for 1 ≤ i < h,in the productions of ΩTBax ((i, k) corresponds to the case where a is the ith active site from thebottom).

If a = πn, no sites of π become non-active, giving the label (h, k + 1).If a > πn, then π · a ends with an ascent and no site of π become non-active. Hence, we

obtain the missing labels in the production of ΩTBax: (h + k + 1 − i, i), for 1 ≤ i ≤ k (the label(h+ k + 1− i, i) corresponds to a being the ith active site from the top).

We remark that although ΩTBax is not precisely the succession rule presented in [15] fortwisted Baxter permutations, it is an obvious variant of it: indeed, starting from the rule of [15],it is enough to replace every label (q, r) by (r + 1, q − 1) to recover ΩTBax.

It follows immediately from the proof of Proposition 20 that ΩTBax is a specialization of Ωsemi.With ΩBax, we therefore obtain two such specializations. In addition, we can observe that theproductions of ΩTBax on second line are the same as in Ωsemi, whereas the productions on the

16

(2,1) (1,1) (2,2) (3,1)

Figure 6: The growth of twisted Baxter permutations (with notation as in Figure 5).

first line of ΩBax are the same as Ωsemi. This means that the restrictions imposed by these twospecializations are “independent”. We will combine them in Section 5, obtaining a succession rulewhich consists of the first line of ΩTBax and the second line of ΩBax.

5 Strong-Baxter numbers

While Section 3 was studying a sequence larger than the Baxter numbers (with a family of permu-tations containing both the Baxter and twisted Baxter permutations), we now turn to a sequencesmaller than the Baxter numbers (associated with a family of permutations included in both fam-ilies of Baxter and twisted Baxter permutations). We present a succession rule for this sequence,and properties of its generating function.

5.1 Strong-Baxter numbers, strong-Baxter permutations, and their suc-cession rule

Definition 21. A strong-Baxter permutation is a permutation that avoids all three vincular pat-terns 2 41 3, 3 14 2 and 3 41 2.

Definition 22. The sequence of strong-Baxter numbers is the sequence that enumerates strong-Baxter permutations.

We have added the sequence enumerating strong-Baxter permutations to the OEIS, where itis now registered as [23, A281784]. It starts with:

1, 2, 6, 21, 82, 346, 1547, , 7236, 35090, 175268, 897273, 4690392, 24961300, . . .

The pattern-avoidance definition makes it clear that the family of strong-Baxter permutations isthe intersection of the two families of Baxter and twisted Baxter permutations. In that sense,these permutations “satisfy two Baxter conditions”, hence the name strong-Baxter.

A succession rule for strong-Baxter numbers is given by the following proposition.

Proposition 23. A generating tree for strong-Baxter permutations can be obtained by insertionson the right, and it is isomorphic to the tree generated by the following succession rule:

Ωstrong =

(1, 1)(h, k) (1, k), . . . , (h− 1, k), (h, k + 1)

(h+ 1, 1), . . . , (h+ 1, k).

Proof. As in the proof of Propositions 3 and 20, we build a generating tree for strong-Baxterpermutations performing local expansions on the right, as illustrated in Figure 7. Note that this ispossible since removing the last point from any strong-Baxter permutation gives a strong-Baxterpermutation.

Let π be a strong-Baxter permutation of size n. By definition, the active sites of π are the a’ssuch that π · a is a strong-Baxter permutations. Any non-empty descent (resp. ascent) of π is a

17

(3,2)(2,2) (1,2) (2,3) (3,1)

Figure 7: The growth of strong-Baxter permutations (with notation as in Figure 5).

pair πiπi+1 such that there exists πj that makes πjπiπi+1 an occurrence of 2 31 (resp. 2 13). Then,the non-active sites a of π are characterized by the fact that a ∈ (πi+1, πi] (resp. a ∈ (πi, πj ] ), forsome occurrence πjπiπi+1 of 2 31 (resp. 2 13). Note that in the case where πn−1πn is a non-emptydescent (resp. ascent), choosing πj = πn + 1 (resp. πj = πn − 1) always gives an occurrence of2 31 (resp. 2 13), and it is the smallest (resp. largest) possible value of πj for which πjπn−1πn isan occurrence of 2 31 (resp. 2 13).

To the strong-Baxter permutation π we assign the label (h, k), where h (resp. k) is the numberof active sites that are smaller than or equal to (resp. greater than) πn. As in the proof ofProposition 3, the permutation 1 has label (1, 1), and we now need to describe, for π of label(h, k), the labels of the permutations π · a when a runs over all active sites of π. So, let a be suchan active site.

If a < πn, then π · a ends with a non-empty descent. As in the proof of Proposition 3, all sitesof π · a in (a+ 1, πn + 1] become non-active (due to the avoidance of 2 41 3). Moreover, due to theavoidance of 3 41 2, the site immediately above a in π · a also becomes non-active. All other activesites of π remain active in π · a, hence giving the labels (i, k) for 1 ≤ i < h in the production ofΩstrong (again, i is such that a is the i-th active site from the bottom).

If a = πn, no site of π becomes non-active, giving the label (h, k + 1) in the production ofΩstrong.

Finally, if a > πn, then π · a ends with an ascent. Because of the avoidance of 3 14 2, we needto consider the occurrences of 2 13 in π to identify which active sites of π become non-active inπ · a. It follows from a discussion similar to that in the proof of Proposition 3 that all sites of π · ain [πn+1, a) become non-active. Hence, we obtain the missing labels in the production of Ωstrong:(h+ 1, i) for 1 ≤ i ≤ k (where i indicates that a is the i-th active site from the top).

In the same sense that both ΩBax and ΩTBax specialize Ωsemi, it is easy to see that thesuccession rule Ωstrong is a specialization of the rule ΩBax (for Baxter permutations) as well asof the rule ΩTBax (for twisted Baxter permutations). In this case, the rule Ωstrong associatedwith the intersection of these two families is simply obtained by taking, for each object produced,the minimum label among the two labels given by ΩBax and ΩTBax. This appears clearly in thefollowing representation:

Ωsemi : (h, k) → (1, k + 1) . . . (h− 1, k + 1) (h, k + 1) (h+ k, 1) . . . (h+ 1, k)ΩBax : (h, k) → (1, k + 1) . . . (h− 1, k + 1) (h, k + 1) (h+ 1, 1) . . . (h+ 1, k)ΩTBax : (h, k) → (1, k) . . . (h− 1, k) (h, k + 1) (h+ k, 1) . . . (h+ 1, k)Ωstrong : (h, k) → (1, k) . . . (h− 1, k) (h, k + 1) (h+ 1, 1) . . . (h+ 1, k).

This is easily explained. Note first that in all four cases h (resp. k) records the number of activesites below (resp. above) the rightmost element of a permutation. Then, it is enough to remarkthat among the active sites of a semi-Baxter permutation (avoiding 2 41 3), the avoidance of 3 41 2deactivates only sites above the rightmost element of the permutation, while the avoidance of3 14 2 deactivates only sites below it.

18

5.2 Generating function of strong-Baxter numbers

Let Ih,k(x) ≡ Ih,k denote the generating function for strong-Baxter permutations having label(h, k), with h, k ≥ 1, and let I(x; y, z) ≡ I(y, z) =

∑h,k≥1 Ih,ky

hzk. (The notation I stands forIntersection, of the families of Baxter and twisted Baxter permutations.)

Proposition 24. The generating function I(y, z) satisfies the following functional equation:

I(y, z) = xyz +x

1− y(y I(1, z)− I(y, z)) + xz I(y, z) +

xyz

1− z(I(y, 1)− I(y, z)). (13)

Proof. From the growth of strong-Baxter permutations according to Ωstrong we write:

I(y, z) = xyz + x∑h,k≥1

Ih,k((y + y2 + · · ·+ yh−1)zk + yhzk+1 + yh+1(z + z2 + · · ·+ zk)

)= xyz + x

∑h,k≥1

Ih,k

(1− yh−1

1− yy zk + yhzk+1 +

1− zk

1− zyh+1 z

)= xyz +

x

1− y(y I(1, z)− I(y, z)) + xz I(y, z) +

xyz

1− z(I(y, 1)− I(y, z)) .

In order to study the nature of the generating function I(1, 1) for strong-Baxter numbers, welook at the kernel of eq. (13), which is

K(y, z) = 1 + x

(1

1− y− z +

yz

1− z

). (14)

We perform the substitutions y = 1 + a and z = 1 + b so that eq. (14) is rewritten as

K(1 + a, 1 + b) = 1− xQ(a, b) where Q(a, b) =1

a+

1

b+a

b+ a+ 2 + b. (15)

As in the proof of Theorem 12 (see Subsection 3.5), we look for the birational transformationsΦ and Ψ in a and b that leave the kernel unchanged, which are:

Φ : (a, b)→(a,

1 + a

b

), and Ψ : (a, b)→

(− b

a(1 + b), b

).

One observes, using Maple for example, that the group generated by these two transformations isnot of small order. We actually suspect that it is of infinite order, preventing us from using theobstinate kernel method to solve eq. (13).

Nevertheless, after the substitution y = 1 + a and z = 1 + b, the kernel we obtain in eq. (15)resembles kernels of functional equations associated with the enumeration of families of walks inthe (positive) quarter plane [11].

Proposition 25. Let W (t; a, b) be the generating function for walks confined in the quarter planeand using (−1, 0), (0,−1), (1,−1), (1, 0), (0, 1) as step set, where t counts the number of stepsand a (resp. b) records the x-coordinate (resp. y-coordinate) of the ending point. The functionW (t; a, b) satisfies the following functional equation:

W (t; a, b) = 1 + t

(1

a+

1

b+a

b+ a+ b

)W (t; a, b)− t

aW (t; 0, b)− t (1 + a)

bW (t; a, 0). (16)

Not only can we take inspiration from the literature on walks in the quarter plane for ourproblem of solving eq. (13), but modifying the step set, we can even arrange that K(1 + a, 1 + b)is exactly the kernel arising in the functional equation for enumerating a family of walks.

19

Lemma 26. Let W2(t; a, b) be the generating function for walks confined in the quarter planeand using (−1, 0), (0,−1), (1,−1), (1, 0), (0, 1), (0, 0), (0, 0) as step (multi-)set, where t countsthe number of steps and a (resp. b) records the x-coordinate (resp. y-coordinate) of the endingpoint. The difference with the step set of Proposition 25 is that we have added two copies of thetrivial step (0, 0), which are distinguished (they can be considered as counterclockwise and clockwiseloops for instance).

The generating functions W (t; a, b) and W2(t; a, b) are related by

W2(x; a, b) = W

(x

1− 2x; a, b

)1

1− 2x(17)

Moreover, denoting by J(x; a, b) := I(x; 1 + a, 1 + b) the generating function for strong-Baxterpermutations, it holds that

J(x; a, b) = (1 + a)(1 + b)xW2(x; a, b). (18)

Proof. First, walks counted by W2 can be described from walks counted by W as follows: a W2-walk is a (possibly empty) sequence of trivial steps, followed by a W -walk where, after each step,we insert a (possibly empty) sequence of trivial steps. This simple combinatorial argument showsthat W2(x; a, b) = W ( x

1−2x ; a, b) 11−2x .

Next, consider the kernel form of eq. (13) after substituting y = 1 + a and z = 1 + b, which is

(1− xQ(a, b))J(x; a, b) = x(1 + a)(1 + b)− x 1 + a

aJ(x; 0, b)− x (1 + a)(1 + b)

bJ(x; a, 0). (19)

Compare it to the kernel form of eq. (16):

(1− t(Q(a, b)− 2))W (t; a, b) = 1− t

aW (t; 0, b)− t (1 + a)

bW (t; a, 0). (20)

Substituting t with x1−2x in eq. (20), and multiplying this equation by (1 + a)(1 + b)x, we see

that (1 + a)(1 + b)xW2(x; a, b) satisfies eq. (19), proving our claim.

With results of [8], this easily gives the following theorem.

Theorem 27. The generating function I(1, 1) of strong-Baxter numbers is not D-finite. The sameholds for the refined generating function I(a+ 1, b+ 1).

Proof. Because D-finiteness is preserved by specialization, it is enough to prove that I(1, 1) is notD-finite. So, with the notation of Lemma 26, our goal is to prove that J(x; 0, 0) is not D-finite.Recall from eq. (18) that J(x; a, b) = (1 + a)(1 + b)xW2(x; a, b), so J(x; 0, 0) and W2(x; 0, 0)coincide up to a factor x. Therefore, proving that W2(x; 0, 0) is non D-finite is enough.

It is proved in [8] that W (t; 0, 0) is not D-finite. Consequently, since 11−2x and x

1−2x are rationalseries, it follows from eq. (17) W2(x; 0, 0) is not D-finite, as desired.

Moreover, some information on the asymptotic behavior of the number of strong-Baxter per-mutations can be derived starting from the connection to walks confined in the quarter plane.In [8] the following proposition is presented.

Proposition 28 (Denisov and Wachtel, [8](Theorem 4)). Let S ⊆ 0,±12 be a step set whichis not confined to a half-plane. Let en denote the number of S-excursions of length n confined tothe quarter plane N2 and using only steps in S. Then, there exist constants K, ρ, and α whichdepend only on S, such that:

• if the walk is aperiodic, en ∼ K ρn nα,

• if the walk is periodic (then of period 2), e2n ∼ K ρ2n (2n)α, e2n+1 = 0.

20

From [8, Section 2.5], the growth constant ρW associated with W (t; 0, 0) is an algebraic numberwhose minimal polynomial is µρ = t3 +t2−18t−43. The approximate value for ρW is 4.729031538.We show below that the growth constant of strong-Baxter numbers is closely related to ρW .

Corollary 29. The growth constant for the strong-Baxter numbers is ρW + 2 ≈ 6.729031538.

Proof. From Lemma 26, I(x; 1, 1) = xW2(x; 0, 0) = xW ( x1−2x ; 0, 0) 1

1−2x . And from the discussion

above, 1ρW

is the radius of convergence of W (t; 0, 0). The radius of convergence of g(x) = x1−2x

is 12 , and lim x→1/2

x<1/2

g(x) = +∞ > 1ρW

. So, the composition W (g(x); 0, 0) is supercritical (see

[18, p. 411]), and the radius of convergence of W ( x1−2x ; 0, 0) is g−1

(1ρW

)= 1

ρW +2 . Since 1ρW +2

is smaller than the radius of convergence 12 of 1

1−2x , 1ρW +2 is also the radius of convergence of

xW ( x1−2x ; 0, 0) 1

1−2x = I(x; 1, 1), proving our claim.

Acknowledgements

The comments of several colleagues on an earlier draft of our paper have helped us improve itsignificantly.

First, we would like to thank David Bevan, for sharing his conjectural formulas for SBn in [7],for bringing to our attention the conjecture about the enumeration of permutations avoiding 14 23,and for suggesting the method used in an earlier version of this paper to derive the asymptoticbehavior of SBn.

We also thank Christian Krattenthaler for independently suggesting this method, and pointingto the reference [13].

We are very grateful to the referee for their numerous and helpful suggestions. In particular, thecurrent proof of the asymptotic behavior of SBn (together with the reference [25]) were suggestedto us by the referee.

Finally, we thank Andrew Baxter for clarifying the status of the conjecture about 14 23-avoidingpermutations, which brought reference [22] to our attention.

References

[1] M. H. Albert, C. Homberger, J. Pantone, N. Shar, V. Vatter, Generating permutations withrestricted containers, Arxiv preprint 1510.00269, to appear in J. Comb. Theory A.

[2] C. Banderier, M. Bousquet-Melou, A. Denise, P. Flajolet, D. Gardy, D. Gouyou-Beauchamps,Generating functions for generating trees, Disc. Math., 246:29–55, 2002.

[3] E. Barcucci, A. Del Lungo, E. Pergola, R. Pinzani, ECO: a methodology for the Enumerationof Combinatorial Objects, J. Diff. Eq. and App., 5:435–490, 1999.

[4] A. M. Baxter, M. Shattuck, Some Wilf-equivalences for vincular patterns, J. Comb., 6(1-2):19–45, 2015.

[5] A. Bernini, L. Ferrari, R. Pinzani, Enumerating Permutations Avoiding Three Babson-Steingrımsson Patterns, Ann. Comb., 9:137–162, 2005.

[6] D. I. Bevan, On The Growth Of Permutation Classes, PhD thesis, The Open University,2015, available as Arxiv preprint 1506.06688.

[7] D. I. Bevan, Private communication, 2017.

[8] A. Bostan, K. Raschel, B. Salvy, Non D-finite excursions in the quarter plane, J. Comb.Theory A, 121:45–63, 2014.

[9] N. Bonichon, M. Bousquet-Melou, E. Fusy, Baxter permutations and plane bipolar orienta-tions, Sem. Lothar. Combin., 61A, article B61Ah, 2008.

21

[10] M. Bousquet-Melou, Four classes of pattern-avoiding permutations under one roof: generatingtrees with two labels, Electron. J. Combin., 9(2), article R19, 2003.

[11] M. Bousquet-Melou, M. Mishna, Walks with small steps in the quarter plane, Contemp.Math., 520:1–40, 2010.

[12] M. Bousquet-Melou, S. Butler, Forest-like permutations, Ann. Combin., 11:335–354, 2007.

[13] M. Bousquet-Melou, G. Xin, On partitions avoiding 3-crossings, Sem. Lothar. Combin., 54,article B54e, 2005.

[14] M. Bouvel, V. Guerrini, S. Rinaldi, Slicings of parallelogram polyominoes, or how Baxter andSchroder can be reconciled, Proceedings of FPSAC 2016, DMTCS proc. BC, 287–298, 2016.

[15] M. Bouvel, O. Guibert, Refined enumeration of permutations sorted with two stacks and aD8-symmetry, Ann. Comb., 18(2):199–232, 2014.

[16] F.R.K. Chung, R. Graham, V. Hoggatt, M. Kleiman, The number of Baxter permutations,J. Comb. Theory A, 24(3):382–394, 1978.

[17] S. Felsner, E. Fusy, M. Noy, D. Orden, Bijections for Baxter families and related objects, J.Comb. Theory A, 118(3):993–1020, 2011.

[18] P. Flajolet, R. Sedgewick, Analytic Combinatorics, Cambridge University Press, Cambridge,2009.

[19] S. Garrabrant, I. Pak, Pattern avoidance is not P-recursive, Arxiv preprint 1505.06508.

[20] S. Gire, Arbres, permutations a motifs exclus et cartes planaires: quelques problemes algo-rithmiques et combinatoires, PhD thesis, Universite Bordeaux I, 1993.

[21] E.J. Janse van Rensburg, T. Prellberg, A. Rechnitzer, Partially directed paths in a wedge, J.Comb. Theory A, 115:623–650, 2008.

[22] A. Kasraoui, New Wilf-equivalence results for vincular patterns, Europ. J. Combin., 34(2):322–337, 2013.

[23] OEIS Foundation Inc., The On-line Encyclopedia of Integer Sequences, http://oeis.org,2011.

[24] M. A. Martinez, C. D. Savage, Patterns in Inversion Sequences II: Inversion SequencesAvoiding Triples of Relations, Arxiv preprint 1609.08106.

[25] R. J. McIntosh, An asymptotic formula for binomial sums, J. Number Theory, 58:158–172,1996.

[26] M. Petkovsek, H.S. Wilf, D. Zeilberger, A=B, AK Peters, Wellesley, 1996.

[27] L. Pudwell, Enumeration schemes for permutations avoiding barred patterns, Electron. J.Combin., 17(1), article R29, 2010.

[28] N. Reading, Lattice congruence, fans and Hopf algebras, J. Comb. Theory A, 110(2):237–273,2005.

[29] J. West, Generating trees and the Catalan and Schroder numbers, Discrete Math., 146:247–262, 1995.

[30] J. West, Enumeration of Reading’s twisted Baxter permutations, The Fourth Annual Inter-national Conference on Permutation Patterns, Reykjavik University, 2006.

22