Visibly Tree Automata with Memory and Constraints

Tree automata with one memory have been introduced in 2001. They generalize both pushdown (word) automata and the tree automata with constraints of equality between brothers of Bogaert and Tison. Though it has a decidable emptiness problem, the main weakness of this model is its lack of good closure properties. We propose a generalization of the visibly pushdown automata of Alur and Madhusudan to a family of tree recognizers which carry along their (bottom-up) computation an auxiliary unbounded memory with a tree structure (instead of a symbol stack). In other words, these recognizers, called Visibly Tree Automata with Memory (VTAM) define a subclass of tree automata with one memory enjoying Boolean closure properties. We show in particular that they can be determinized and the problems like emptiness, membership, inclusion and universality are decidable for VTAM. Moreover, we propose several extensions of VTAM whose transitions may be constrained by different kinds of tests between memories and also constraints a la Bogaert and Tison comparing brother subtrees in the tree in input. We show that some of these classes of constrained VTAM keep the good closure and decidability properties, and we demonstrate their expressiveness with relevant examples of tree languages.


Introduction
The control flow of programs with calls to functions can be abstracted as pushdown systems.This allows to reduce some program verification problems to problems (e.g.modelchecking) on pushdown automata.When it comes to functional languages with continuation passing style, the stack must contain information on continuations and has the structure of a dag (for jumps).Similarly, in the context of asynchronous concurrent programming languages, for two concurrent threads the ordering of return is not determined (synchronized) and these threads can not be stacked.In these cases, the control flow is better modeled as a tree structure rather than a stack.That is why we are interested in tree automata with one memory, which generalize the pushdown (tree) automata, replacing the a stack with a tree.Here, a "memory" has to be understood as a storage device, whose structure is a tree.For instance, two memories would correspond to two storage devices whose access would be independent.
The tree automata with one memory introduced in [7] compute bottom-up on a tree, with an auxiliary memory carrying a tree, as in former works such as [14].Along a computation, at any node of the tree, the memory is updated incrementally from the memory reached at the sons of the node.This update may consist in building a new tree from the memories at the sons (this generalizes a push) or retrieving a subtree of one of the memories at the sons (this generalizes a pop).In addition, such automata may perform equality tests: a transition may be constrained to be performed, only when the memories reached at some of the sons are identical.In this way, tree automata with one memory also generalize certain cases of tree automata with equality and disequality tests between brothers [4].
Automata with one memory have been introduced in the context of the verification of security protocols, where the messages exchanged are represented as trees.In the context of (functional or concurrent) programs, the creation of a thread, or a callcc, corresponds to a push, the termination of a thread or a callcc corresponds to a pop.The emptiness problem for such automata is in EXPTIME (note that for the extension with a second memory the emptiness problem becomes undecidable).However, the class of tree languages defined by such automata is neither closed by intersection nor by complement.This is not surprising as they are strictly more general than context free languages.
On the other hand, Alur and Madhusudan have introduced the notion of visibility for pushdown automata [2], which is a relevant restriction in the context of control flow analysis.With this restriction, determinization is possible and actually the class of languages is closed under Boolean operations.
In this paper, we propose the new formalism of Visibly Tree Automata with Memory (VTAM).On one hand, it extends visibly pushdown languages to the recognition of trees, and with a tree structure instead of a stack, following former approaches [14,21,10].On the other hand, VTAM restrict tree automata with one memory, imposing a visibility condition on the transitions: each symbol is assigned a given type of action.When reading a symbol, the automaton can only perform the assigned type of action: push or pop.
We first show in Section 2 that VTAM can be determinized, using a proof similar to the proof of [2], and do have the good closure properties.The main difficulty here is to understand what is a good notion of visibility for trees, with memories instead of stacks.We also show that the problems of membership and emptiness are decidable in deterministic polynomial time for VTAM.
In a second part of the paper (Section 3), we extended VTAM with constraints.Our constraints here are recognizable relations; a transition can be fired only if the memory contents of the sons of the current node satisfy such a relation.We give then a general theorem, expressing conditions on the relations, which ensure the decidability of emptiness.Such conditions are shown to be necessary on one hand, and, on the other hand, we prove that they are satisfied by some examples, including syntactic equality and disequality tests and structural equality and disequality tests.The case of VTAM with structural equality and disequality tests (this class is denoted VTAM ≡ ≡ ) is particularly interesting, since the determinization and closure properties of Section 2 carry over this generalization, which we show in Section 3.4.2.The automata of VTAM ≡ ≡ also enjoy a good expressive power, as we show in Section 3.7 by presenting some non-trivial examples of languages in this class: well-balanced binary trees, red-black trees, powerlists...As an intermediate result, we show that, in case of equality tests or structural equality tests, the language of memories that can be reached in a given state is always a regular language.This is a generalization of the well-known result that the set of stack contents in a pushdown automaton is always regular.To prove this, we observe that the memories contents are recognized by a two-way alternating tree automaton with constraints.Then we show, using a saturation strategy, that two-way alternating tree automata with (structural) equality constraints are not more expressive than standard tree automata.
Finally, in Section 4 we propose a class of visibly tree automata, which combines the structural constraints of VTAM ≡ ≡ , testing memory contents, with Bogaert-Tison constraints of [4] (equality and disequality tests between brothers subterms) which operate on the term in input.We show that the tree automata of this class can be determinized, are closed under Boolean operations and have a decidable emptiness problem.
Related Work.Generalizations of pushdown automata to trees (both for input and stack) are proposed in [14,21,10].Our contributions are the generalization of the visibility condition of [2] to such tree automata -our VTAM (without constraints) strictly generalize the VP Languages of [2], and the addition of constraints on the stack contents.The visibly tree automata of [1] use a word stack which is less general than a tree structured memory but the comparison with VTAM is not easy as they are alternating and compute top-down on infinite trees.
Independently, Chabin and Rety have proposed [5] a formalism combining pushdown tree automata of [14] with the concept of visibly pushdown languages.Their automata recognize finite trees using a word stack.They have a decidable emptiness problem and the corresponding tree languages (Visibly Pushdown Tree Languages, VPTL) are closed under Boolean operations.Following remarks of one of these two authors, it appeared that VTAM and VPTL are incomparable, see Section 2.2.
1. Preliminaries 1.1.Term algebra.A signature Σ is a finite set of function symbols with arity, denoted by f , g. . .We write Σ n the subset of function symbols of Σ of arity n.Given an infinite set X of variables, the set of terms built over Σ and X is denoted T (Σ, X ), and the subset of ground terms is denoted T (Σ).The set of variables occurring in a term t ∈ T (Σ, X ) is denoted vars(t).A substitution σ is a mapping from X to T (Σ, X ) such that {x|σ(x) = x}, the support of σ, is finite.The application of a substitution σ to a term t is written tσ.It is the homomorphic extension of σ to T (Σ, X ).The positions Pos(t) in a term t are sequences of positive integers (Λ, the empty sequence, is the root position).A subterm of t at position p is written t| p , and the replacement in t of the subterm at position p by u denoted t[u] p .1.2.Rewriting.We assume standard definitions and notations for term rewriting [11].A term rewriting system (TRS) over a signature Σ is a finite set of rewrite rules ℓ → r, where ℓ ∈ T (Σ, X ) and r ∈ T (Σ, vars(ℓ)).A term t ∈ T (Σ, X ) rewrites to s by a TRS R (denoted t → R s) if there is a rewrite rule ℓ → r ∈ R, a position p of t and a substitution σ such that t| p = ℓσ and s = t[rσ] p .The transitive and reflexive closure of → R is denoted − − → * R .
1.3.Tree Automata.Following definitions and notation of [8], we consider tree automata which compute bottom-up (from leaves to root) on (finite) ground terms in T (Σ).At each stage of computation on a tree t, a tree automaton reads the function symbol f at the current position p in t and updates its current state, according to f and to the respective states reached at the positions immediately under p in t.Formally, a bottom-up tree automaton (TA) A on a signature Σ is a tuple (Q, Q f , ∆) where Σ is the computation signature, Q is a finite set of nullary state symbols, disjoint from Σ, Q f ⊆ Q is the subset of final states and ∆ is a set of rewrite rules of the form: f (q 1 , . . ., q n ) → q, where f ∈ Σ and q 1 , . . ., q n ∈ Q.A term t is accepted (we may also write recognized) by A in state q iff t − − → * ∆ q, and the language L(A, q) of A in state q is the set of ground terms accepted in q.The language L(A) of A is q∈Q f L(A, q) and a set of ground terms is called regular if it is the language of a TA.

Visibly Tree Automata with Memory
We propose in this section a subclass of the tree automata with one memory [7] which is stable under Boolean operations and has decidable emptiness and membership problems.
2.1.Definition of VTAM.Tree automata have been extended [14,21,10,7] to carry an unbounded information along the states in computations.In [7], this information is stored in a tree structure and is called memory.We keep this terminology here, and call our recognizers tree automata with memory (TAM).For consistency with the above formalisms, the memory contents will be ground terms over a memory signature Γ.
Like for TA we consider bottom-up computations of TAM in trees; at each stage of computation on a tree t, a TAM, like a TA, reads the function symbol at the current position p in t and updates its current state, according to the states reached immediately under p.Moreover, a configuration of TAM contains not only a state but also a memory, which is a tree.The current memory is updated according to the respective contents of memories reached in the nodes immediately under p in t.
As above, we use term rewrite systems in order to define the transitions allowed in a TAM.For this purpose, we add an argument to state symbols, which will contain the memory.Hence, a configuration of TAM in state q and whose memory content is the ground term m ∈ T (Γ), is represented by the term q(m).We propose below a very general definition of TAM.It is similar to the one of [7], except that we have here general patterns m 1 , . . ., m n , m, while these patterns are restricted in [7], for instance avoiding memory duplications.Since we aim at providing closure and decision properties, we will also impose (other) restrictions later on.Definition 2.1.A bottom-up tree automaton with memory (TAM) on a signature Σ is a tuple (Γ, Q, Q f , ∆) where Γ is a memory signature, Q is a finite set of unary state symbols, disjoint from Σ ∪ Γ, Q f ⊆ Q is the subset of final states and ∆ is a set of rewrite rules of the form f q 1 (m 1 ), . . ., q n (m n ) → q(m) where f ∈ Σ n , q 1 , . . ., q n , q ∈ Q and m 1 , . . ., m n , m ∈ T (Γ, X ).
The rules of ∆ are also called transition rules.A term t is accepted by A in state q ∈ Q and with memory m ∈ T (Γ) iff t − − → * ∆ q(m), and the language L(A, q) and memory language M (A, q) of A in state q are respectively defined by: The language of A is the union of languages of A in its final states, denoted: Visibility Condition.The above formalism is of course far too expressive.As there are no restrictions on the operation performed on memory by the rewrite rules, one can easily encode a Turing machine as a TAM.We shall now define a decidable restriction called visibly tree automata with memory (VTAM).
First, we consider only three main families (later divided into the subcategories defined in Figure 1) of operations on memory.We assume below a computation step at some position p of a term, where memories m 1 , . . ., m n have been reached at the positions immediately below p: PUSH: the new current memory m is built with a symbol h ∈ Γ n pushed on the top of memories m 1 , . . ., m n : f q 1 (m 1 ), . . ., q n (m n ) → q h(m 1 , . . ., m n ) .According to the terminology of [2], this corresponds to a call move in a program represented by an automaton.POP: the new current memory is a subterm of one of the memories reached so far: f . . ., q i (h(m ′ 1 , . . ., m ′ k )), . . .→ q(m ′ j ).The top symbol h of m i is also read.This corresponds to a function's return in a program.
We have here to split POP operations into four categories, depending on whether we pop on the memory at the left son or on the memory at the right son and on whether we get the left son of that memory or its right son.INT (internal): the new current memory is one of the memories reached: This corresponds to an internal operation (neither call nor return) in a function of a program.
Again, we need to split INT operations into three categories: one for constant symbols and two rules for binary symbols, depending on which of the two sons memories we keep.
Next, we adhere to the visibility condition of [2].The idea behind this restriction, which was already in [16], is that the symbol read by an automaton (in a term in our case and [1], in a word in the case of [2]) corresponds to an instruction of a program, and hence belongs to one of the three above families (call, return or internal).Indeed, the effect of the execution of a given instruction on the current program state (a stack for [2] or a tree in our case) will always be in the same family.In other words, in this context, the family of the memory operations performed by a transition is completely determined by the function symbol read.
Let us assume from now on for the sake of simplicity the following restriction on the arity of symbols: Figure 1: VTAM transition categories.
All the symbols of Σ and Γ have either arity 0 or 2. This is not a real restriction, and the results of this paper can be extended straightforwardly to the case of function symbols with other arities.The signature Σ is partitioned in eight subsets: The eight corresponding categories of transitions (transitions of the same category perform the same kind of operation on the memory) are defined formally in Figure 1.In this figure, one constant symbol has a particular role: ⊥ is a special constant symbol in Γ, used to represent an empty memory.Note that there are three categories for Now, let us try to explain more precisely the relation with the visibly pushdown languages of [2], when considering finite word languages.
If the stack is empty in any accepting configuration of some finite word pushdown automaton A, then it is easy to compute a pushdown automaton A, which accepts the reverses (mirror images) of the words accepted by A. Moreover, if A is a visibly pushdown automaton, then A is also a visibly pushdown automaton: it suffices to exchange the push and pop symbols.
For pushdown word languages, there is a well-known lemma showing that the recognition by final state is equivalent to the recognition by empty stack.This equivalence however requires ǫ-transitions to empty the stack when a final state is reached.There are however no ǫ-transitions in visibly pushdown automata.So, if we consider for instance the language of words w ∈ {a, b} * such that any prefix of w contains more a than b's, it is recognized by a visibly pushdown automaton.While, if we consider the mirror image (all suffixes contain more a's than b's), it is not recognized by a visibly pushdown automaton.
In conclusion, as long as visibility is relevant, the way the automaton is moving is also relevant.This applies of course to trees as well: there is a difference between top-down and bottom-up recognition.Now, if we encode a word as a tree on a unary alphabet, starting from right to left, VTAM generalize visibly pushdown automata: moving bottom-up in the tree corresponds to moving left-right in the word.
VPTA transitions and VPTL are defined in [5] in the same formalism (rewrite rules) as in Figure 1, except that the rules are oriented in the other direction (top-down computations) and the memory contains a word, i.e. terms built with unary function symbols and one constant (empty stack).
As sketched above, since the automata of [5] work top-down, a language can be recognized by a VTAM (which works bottom-up) and not by a VPTL.As a typical example, consider the trees containing only unary symbols a, b and a constant 0 and such that all subterms contain more a's than b's.
But the converse is also true: there are similarly languages that are recognized by VPTA and not by VTAM (and there, constraints cannot help!) Now, if we consider a slight modification of VPTA, in which the automata work bottomup (simply change the direction of transition rules), it is not clear that good properties (closure and decision) are preserved since, now, we get equality tests between memory contents, increasing the original expressive power; when going top-down we always duplicate the memory content and send one copy to each son, while going bottom-up we may have different memory contents at two brother positions.

Determinism.
A VTAM A is said complete if every term of T (Σ) belongs to L(A, q) for at least one state q ∈ Q.Every VTAM can be completed (with a polynomial overhead) by the addition of a trash state.Hence, we shall consider from now on only complete VTAM.
Theorem 2.3.For every VTAM A = (Γ, Q, Q f , ∆) there exists a deterministic VTAM Proof.We follow the technique of [2] for the determinization of visibly pushdown automata: we do a subset construction and postpone the application (to the memory) of PUSH rules, until a matching POP is met.The construction of [2] is extended in order to handle the branching structure of the term read and of the memory.
With the visibility condition, for each symbol read, only one kind of memory operation is possible.This permits a uniform construction of the rules of A det for each symbol of Σ.As we shall see below, A det does not need to keep track of the contents of memory (of A) during its computation, it only needs to memorize information on the reachability of states of A, following the path (in the term read) from the position of the PUSH symbol which has pushed the top symbol of the current memory (let us call it the last-memory-push-position) to the current position in the term.We let : is the subset of states whose second component contains a final state of Q f .The first component is a flag indicating whether the memory is currently empty (value 0) or not (value 1).The second component is the subset of states of Q that A can reach at current position, and the third component is a binary relation on Q which contains (q, q ′ ) iff starting from a state q and memory m at the last-memory-push-position, A can reach the current position in state q ′ , and with the same memory m.We consider memory symbols made of pairs of states and PUSH symbols: The components of a symbol p ∈ Γ det refer to the transition who pushed p: the first and second components of p are respectively the left and right initial states of the transition and the third component is the symbol read.
The transition rules of ∆ det are given below, according to the symbol read.
INT.For every i and for every f ∈ Σ INT i , we have the following rules in ∆ det : where R := q ∃q 1 ∈ R 1 , q 2 ∈ R 2 , f q 1 (y 1 ), q 2 (y 2 ) → q(y 1 ) ∈ ∆ , and S is the update of S 1 according to the INT 1 -transitions of ∆, when b 1 = 1 (the case b 1 = 0 is similar): PUSH.For every f ∈ Σ PUSH , we have the following rules in ∆ det : where is used to initialize the memorization of state reachability from the position of the symbol f , and f .Note that the two states reached just below the position of application of this rule are pushed on the top of the memory.They will be used later in order to update R and S when a matching POP symbol is read.
POP.For every f ∈ Σ POP 11 , we have the following rules in ∆ det : where When a POP symbol is read, the top symbol of the memory, which is popped, contains the states reached just before the application of the matching PUSH.We use this information in order to update b The above constructions ensure the three invariants stated above, after the definition of Q det and corresponding to the three components of these states.It follows that L(A) = L(A det ).
2.4.Closure Properties.The tree automata with one memory of [7] are closed under union but not closed under intersection and complement (even their version without constraints).The visibility condition makes possible these closures for VTAM.
Theorem 2.4.The class of tree languages of VTAM is closed under Boolean operations.One can construct VTAM for union, intersection and complement of given VTAM languages whose sizes are respectively linear, quadratic and exponential in the size of the initial VTAM.
For the union of the languages of A 1 and A 2 , we construct a VTAM A ∪ whose memory signature, state set, final state set and rules set are the union of the respective memory signatures, state sets, final state sets and rules sets of the two given VTAM.We have For the intersection of the languages of A 1 and A 2 , we construct a VTAM A ∩ whose memory signature, state set and final state set are the Cartesian product of the respective memory signatures, state sets and final state sets of the two given VTAM.
The rule set ∆ ∩ of the intersection VTAM A ∩ is obtained by "product" of rules of the two given VTAM with same function symbols.The product of rules means Cartesian products of the respective states and memory symbols pushed or popped.More precisely, ∆ ∩ is the smallest set of rules such that: • if ∆ 1 contains f q 11 (y 1 ), q 12 (y 2 ) → q 1 h 1 (y 1 , y 2 ) and ∆ 2 contains f q 21 (y 1 ), q 22 (y 2 ) → q 2 h 2 (y 1 , y 2 ) , for some f ∈ Σ PUSH , then ∆ ∩ contains f q 11 , q 21 (y 1 ), q 12 , q 22 (y 2 ) → q 1 , q 2 h 1 , h 2 (y 1 , y 2 ) .
For the complement, we use the construction of Theorem 2.3 and a completion (this operation preserves determinism), and take the complement of the final state set of the VTAM obtained.

Decision Problems.
Every VTAM is a particular case of tree automaton with one memory of [7].Since the emptiness problem (whether the language accepted is empty or not) is decidable for this latter class, it is also decidable for VTAM.However, whereas this problem is EXPTIME-complete for the automata of [7], it is only PTIME for VTAM.
Proof.Assume given a VTAM A = (Γ, Q, Q f , ∆).By definition, for each state q ∈ Q, the language L(A, q) is empty iff the memory language M (A, q) is empty.For each state q, we introduce a predicate symbol P q and we construct Horn clauses in such a way that P q (m) belongs to the least Herbrand model of this set of clauses, iff the configuration with state q and memory m is reachable by the automaton (i.e.m ∈ M (A, q)).
For such a construction (already given in [7]), we simply forget the function symbol, associating to a transition rule f (q 1 (m 1 ), q 2 (m 2 )) → q(m) the Horn clause P q 1 (m 1 ), P q 2 (m 2 ) ⇒ P q (m).Then, according to the restrictions in Definition 2.2, we get only Horn clauses of one of the following forms: ⇒ P q (c) P q 1 (y 1 ), P q 2 (y 2 ) ⇒ P q h(y 1 , y 2 ) P q 1 h(y 11 , y 12 ) , P q 2 (y 2 ) ⇒ P q (y 11 ) P q 1 h(y 11 , y 12 ) , P q 2 (y 2 ) ⇒ P q (y 12 ) P q 1 (⊥), P q 2 (y 2 ) ⇒ P q (⊥) where all the variables are distinct.Such clauses belong to the class H 3 of [19], for which it is proved in [19] that emptiness is decidable in cubic time.It follows that emptiness of VTAM is decidable in cubic time.
Hardness for PTIME follows from the PTIME-hardness of emptiness of finite tree automata [8].
Another proof relying on similar techniques, but for a more general result, will be stated in Lemma 3.7 and can be found in Appendix 5.
The universality is the problem of deciding whether a given automaton recognizes all ground terms.Inclusion refers to the problem of deciding the inclusion between the respective languages of two given automata.
Corollary 2.6.The universality and inclusion problem are EXPTIME-complete for VTAM.
Proof.A VTAM A is universal iff the language of its complement automaton A is empty, and With the bounds given in Theorem 2.4 these problems can be decided in EXPTIME for VTAM (these operations require a determinization of a given VTAM first).
The EXPTIME-hardness follows from the corresponding property of finite tree automata (see [8] for instance).
The membership problem is, given a term t and an automaton A, to know whether t is accepted by A.
Corollary 2.7.The membership problem is decidable in PTIME for VTAM.
Proof.Given a term t we can build a VTAM A t which recognizes exactly the language {t}.The intersection of A t with the given VTAM A recognizes a non empty language iff t belongs to the language of A.

Visibly Tree Automata with Memory and Constraints
In the late eighties, some models of tree recognizers were obtained by adding equality and disequality constraints in transitions of tree automata.They have been proposed in order to solve problems with term rewrite systems or constraints systems with non-linear patterns (terms with multiple occurrences of the same variable).The tree automata of [4] for instance can perform equality and disequality tests between subterms located at brother positions of the input term.
In the case of tree automata with memory, constraints are applied to the memory contents.Indeed, each bottom-up computation step starts with two states and two memories (and ends with one state and one memory), and therefore, it is possible to compare the contents of these two memories, with respect to some binary relation.
We state first the general definition of visibly tree automata with constraints on memories (Section 3.1), then give sufficient conditions on the binary relation for the emptiness decidability (Section 3.2) and show that, if in general regular binary relations do not satisfy these conditions (and indeed, the corresponding class of constrained VTAM has an undecidable emptiness problem, Section 3.3) some relevant examples do satisfy them.In particular, we study in Section 3.4.2 the case of VTAM with structural equality constraints.They enjoy not only decision properties but also good closure properties.Some relevant examples of tree languages recognized by constrained VTAM of this class are presented at the end of the section.We shall not extend the rules PUSH and POP with constraints for some reasons explained in section 3.5.A ground term t rewrites to s by a constrained rule r (where c is either R or ¬R) if there exists a position p of t and a substitution σ such that t| p = ℓσ, y 1 σ c y 2 σ and s = t[rσ] p .
For example, if R is term equality, the transition is performed only when the memory contents are identical.Definition 3.1.A visibly tree automaton with memory and constraints (VTAM R ¬R ) on a signature Σ is a tuple (Γ, R, Q, Q f , ∆) where Γ, Q, Q f are defined as for TAM, R is an equivalence relation on T (Γ) and ∆ is a set of rewrite rules in one of the above categories: We let VTAM R be the subclass of VTAM R ¬R with positive constraints only.The acceptance of terms of T (Σ) and languages of term and memories are defined and denoted as in Section 2.1.

The definition of complete VTAM R
¬R is the same as for VTAM.As for VTAM, every VTAM R ¬R can be completed (with a polynomial overhead) by the addition of a trash state q ⊥ .The only subtle difference concerns the constrained rules: for every f 9 ∈ INT R 1 and every states q 1 , q 2 , • if there is a rule f 9 q 1 (y 1 ), q 2 (y 2 ) q(y 1 ) and no rule of the form • if there is a rule q(y 1 ) and no rule of the form f 9 q 1 (y 1 ), q 2 (y 2 ) − −−−− → y 1 R y 2 q ′ (y 1 ), then we add f 9 q 1 (y 1 ), q 2 (y 2 ) − −−−− → y 1 R y 2 q ⊥ (y 1 ), • if there is no rule of the form f 9 q 1 (y 1 ), q ′ (y 1 ), then we add f 9 q 1 (y 1 ), for all q 1 , q 2 ∈ Q, there are at most two rules in ∆ with left-member f q 1 (y 1 ), q 2 (y 2 ) , and if there are two, one has a positive constraint and the other has a negative constraint.
We will see in Section 3.4 a subclass of VTAM R ¬R that can be determinized (when R is structural equality) and another one that cannot (when R is syntactic equality).

Sufficient Conditions for Emptiness Decision.
We propose here a generic theorem ensuring emptiness decision for VTAM R ¬R .The idea of this theorem is that under some condition on R, the transition rules with negative constraints can be eliminated.Theorem 3.2.Let R be an equivalence relation satisfying these two properties: i. for every automaton A of VTAM R and for every state q of A, the memory language M (A, q) is effectively a regular tree language, ii. for every term m ∈ T (Γ), the cardinality of the equivalence class of m for R is finite and and its elements can be enumerated.Then the emptiness problem is decidable for VTAM R ¬R .Proof.The proof relies on the following Lemma 3.3 which states that the negative constraints in VTAM R ¬R can be eliminated, while preserving the memory languages.The elimination can be done thanks to the condition ii , by replacement of the rules of Next, we can use i in order to decide emptiness for the VTAM R obtained by elimination of negative constraints.Indeed, for all states q of A, by definition, L(A, q) is empty iff M (A, q) is empty.
Proof.The construction of A + is by induction on the number n of rules with negative constraints in ∆ and uses the bound on the size of equivalence classes, condition ii of the theorem.The result is immediate if n = 0. We assume that the result is true for n − 1 rules, and show that we can get rid of a rule of ∆ with negative constraints (and replace it with rules unconstrained or with positive constraints).Let us consider one such rule: We show that, under the induction hypothesis, we have the following lemma which will be used below in order to get rid of the rule (3.1).Lemma 3.4.Given m 1 , . . ., m k ∈ M (A, q 2 ), it is effectively decidable whether M (A, q 2 ) \ {m 1 , . . ., m k } is empty or not and, in case it is not empty, we can effectively build a m k+1 in this set.Proof.Let [m i ] R denote the equivalence class of m i .By condition ii, every [m i ] R is finite, hence for each i ≤ k, we can build a VTAM A i with a state p i such that M (A i , p i ) is the complement of [m i ] R .We add all the rules of A i to A, obtaining A ′ (we assume that the state sets of A 1 , . . ., A k , A are disjoint, and that the states of A 1 , . . ., A k are not final in A ′ ).
Since R is an equivalence relation, we have: Hence, if y 2 = m i is a witness for the rule (3.1), then we can apply instead a rule: Then we add to A ′ the rules (3.2) as above and obtain A ′′ .It can be shown that M (A ′′ , q 2 ) = M (A, q 2 ).Let m k+1 be a term of M (A ′′ , q 2 ) \ {m 1 , . . ., m k } of minimal size (if one exists).This term m k+1 can be created in a run of A ′′ which does not use the rule (3.1).Otherwise, the witness for y 2 in the application of this rule would be a term of M (A ′′ , q 2 ) \ {m 1 , . . ., m k } smaller than m k+1 (it cannot be one of {m 1 , . . ., m k } because for these particular values of y 2 , we assume the application of (3.2)).It follows that m k+1 ∈ M (A ′′ \ (3.1), q 2 ).This automaton A 1 = A ′′ \ (3.1) has n − 1 rules with negative constraints.Hence, by induction hypothesis, there is a VTAM R A + 1 with m k+1 in its memory language M (A + 1 , q 2 ).By condition i, this language is regular and we can build m k+1 from a TA for this language.Now, let us come back to the proof that we can replace rule (3.1), while preserving the memory languages.If M (A, q 2 ) = ∅ (which can be effectively decided according to lemma 3.4) then the rule (3.1) is useless and can be removed from A without changing its memory language.Note that the condition M (A, q 2 ) = ∅ is decidable because by hypothesis i, M (A, q 2 ) is regular.Otherwise, let m 1 ∈ M (A, q 2 ) be built with Lemma 3.4 and let N 1 be the cardinal of the equivalence class [m 1 ] R .We apply N 1 times the construction of Lemma 3.4.There are three cases: (1) if we find more than N 1 terms in M (A, q 2 ), then one of them, say m k is not in [m 1 ] R .
Then (3.1) is useless for the point of view of memory languages: whatever value for y 1 , we know a y 2 ∈ M (A, q 2 ) which permits to fire the rule.Indeed, if y 1 ∈ [m 1 ] R , then we can choose y 2 = m k , and otherwise we choose y 2 = m 1 .Hence (3.1) can be replaced without changing the memory language by: f q 1 (y 1 ), q 0 (y 2 ) − → q(y 1 ) ( where q 0 is any state of A such that M (A, q 0 ) = ∅.We can then apply the induction hypothesis to the VTAM R ¬R obtained.(2) if we find less than N 1 terms in M (A, q 2 ), but one is not in [m 1 ] R .The case is the same as above.
(3) if we find less than N 1 terms in M (A, q 2 ), all in [m 1 ] R , it means that one of the applications of Lemma 3.4 was not successful, and hence that we have found all the terms of M (A, q 2 ).It follows that the rule (3.1) can be fired iff Hence, we can replace (3.1) by Then we can apply the induction hypothesis.
We present in Section 3.4 two examples of relations satisfying i. and ii.

3.3.
Regular Tree Relations.We first consider the general case of VTAM R ¬R where the equivalence R is based on an arbitrary regular binary relation on T (Γ).By regular binary relation, we mean a set of pairs of ground terms accepted by a tree automaton computing simultaneously in both terms of the pair.More formally, we use a coding of a pair of terms of T (Σ) into a term of T (Σ ∪ {⊥}) 2 , where ⊥ is a new constant symbol (not in Σ).This coding is defined recursively by: Then, a binary relation R ⊆ T (Σ) × T (Σ) is called regular iff the set {s ⊗ t (s, t) ∈ R} is regular.The above coding of pairs is unrelated to the product used in Theorem 2.4.
Theorem 3.5.The membership problem for VTAM R ¬R is NP-complete when R is a regular binary relation.

Proof. Assume given a ground term t ∈ T (Σ) and a VTAM
Because of the visibly condition, for every subterm s of t, we can compute in polynomial time in the size of s the shape denoted struct(s), which is an abstraction of the memory reached when A runs on s.More precisely, struct(s) is an unlabeled tree, and every possible content of memory m reachable by A in a computation s − − → * ∆ q(m) is obtained by a labeling of the nodes of struct(s) with symbols of Γ.Note that for all subterm s, the size of struct(s) is smaller than the size of t.
Let us guess a decoration of every node of t with a state of Q and a labeling of struct(s) (where s is the subterm of t at the given node), such that the root of t is decorated with a final state of Q f .We can check in polynomial time whether this decoration represents a run of A on t or not.
The NP-hardness is a consequence of Theorem 3.9, which applies to the particular case where R is the syntactic equality between terms.
Note that the NP algorithm works with every equivalence R based on a regular relation, but the the NP-hardness concerns only some cases of such relations.For instance, in Section 3.4, we will see one example of relation for which membership is NP-hard and another example for which it is in PTIME.

The class of VTAM R
¬R when R is a binary regular tree relation constitutes a nice and uniform framework.Note however the condition ii of Theorem 3.2 is not always true in this case.Actually, this class is too expressive.Theorem 3.6.Given a regular binary relation R and an automaton A in VTAM R , the emptiness of L(A) is undecidable.
Proof.We reduce the blank accepting problem for a deterministic Turing machine M. We encode configurations of M as "right-combs" (binary trees) built with the tape and state symbols of M, in Σ PUSH (hence binary) and a constant symbol ε in Σ INT 0 .Let R be the regular relation which accepts all the pairs of configurations c ⊗ c ′ such that c ′ is a successor of c by M. A sequence of configurations c 0 c 1 . . .c n (with n ≥ 1) is encoded as a tree We construct a VTAM R A which accepts exactly the term-representations t of computation sequences of M starting with the initial configuration c 0 of M and ending with a final configuration c n with blank tape.Following the type of the function symbols, the rules of A will • push all the symbols read in subterms of t corresponding to configurations, • compare, with R, c i and c i+1 (the memory contents in respectively the left and right branches) and store c i in the memory, with a transition applied at the top of a subterm f (c i , f (c i+1 , . ..)).This way, A checks that successive configurations in t correspond to transitions of M, hence that the language of A is not empty iff M accepts the initial configuration c 0 .

Syntactic and Structural Equality and Disequality Constraints.
We present now two examples of relations satisfying the conditions of Theorem 3.2: syntactic and structural term equality.The satisfaction of condition i will be proved with the help of the following crux Lemma.Lemma 3.7.Let R be a regular binary relation defined by a TA whose state set is R i i = {1..n} and such that ∀i, j ∃k, l, ∀x, y, z.
) be a tree automaton with memory and constraints (not necessarily visibly).Then it is possible to compute in exponential time a finite tree automaton A ′ , such that, for every state q ∈ Q, the language M (A, q) is the language accepted in some state of A ′ .
Proof.(Sketch) To prove this lemma, we first observe that the M (A, q) (for q ∈ Q) are actually the least sets that satisfies the following conditions (we assume here for simplicity that the non-constant symbols are binary and display only some of the implications; the others can be easily guessed): if there is a rule f (q 1 (x 1 ), q 2 (x 2 )) → q(g( if there is a rule f (q 1 (g(x, y), q 2 (z) In terms of automata, this means that M (A, q) is a language recognized by a two-way alternating tree automaton with regular binary constraints.In other words, such languages are the least Herbrand model of a set of clauses of the form POP 12 , POP 22 The lemma then shows that languages that are recognized by two-way alternating tree automata with some particular regular constraints, are also recognized by a finite tree automaton.This corresponds to classical reductions of two-way automata to one-way automata (see e.g [8], chapter 7, [13], or [12,6] for the first relevant references).The idea of the reduction is to find shortcuts: moving up and down yields a move at the same level.Add such shortcuts as new rules, until getting a "complete set".Then only keep the non-redundant rules: this yields a finite tree automaton.Such a procedure relies on the definitions of ordered strategies, redundancy and saturation (aka complete sets), which are classical notions in automated first-order theorem proving [13,3,20].Indeed, formally, a "shortcut" must be a formula, which allows for smaller proofs than the proof using the two original rules.A saturated set corresponds to a set of formulas whose all shortcuts are already in the set.
The advantage of the clausal formalism is to enable an easy representation of the above shortcuts, as intermediary steps.Such shortcuts are clauses, but are not automata rules.Second, we may rely on completeness results for Horn clauses.
That is why, only for the proof of this lemma, which follows and extend the classical proofs adding some regular constraints, we switch to a first-order logic formalization.The complete proof can be found in Appendix 5.As in the classical proofs, we saturate the set of clauses by resolution with selection and eager splitting.This saturation terminates, and the set of clauses corresponding to finite tree automata transitions in the saturated set recognizes the language M (A, q), which is therefore regular.
The condition on R in the lemma allows to break chains such as ∃x 1 , . . ., x n .xRx 1 ∧ x 1 Rx 2 ∧ • • • ∧ x n Ry ∧ P (x, y), which would be a source of non-termination in the saturation procedure.We may indeed replace such chains by ∃x 1 , . . ., x n .xR 1 x 1 ∧xR 2 x 2 ∧. ..∧xR n x n ∧ xR 0 y ∧ P (x, y), which can again be simplified into ∃x 1 .xSx 1 ∧ xR 0 y ∧ P (x, y) where S is the intersection of R 1 , . . ., R n .Possible such intersections range in a finite set as the relation R is regular and the R i s are states of the automaton accepting R.
Finally note that finding k, l in the lemma's assumption can always be performed in an effective way since R is regular.
3.4.1.Syntactic Constraints.We first apply Lemma 3.7 to the class VTAM = = where = denotes the equality between ground terms made of memory symbols.Note that it is a particular case of constrained VTAM R ¬R of the above section 3.3, since the term equality is a regular relation.The automata of the subclass with positive constraints only, VTAM = , are particular cases of tree automata with one memory of [7], and have therefore a decidable emptiness problem.We show below that VTAM = = fulfills the hypotheses of Theorem 3.2, and hence that the emptiness is also decidable for the whole class.
We can first verify that the relation = checks the hypothesis of Lemma 3.7, hence the condition i of Theorem 3.2.Moreover, the relation = obviously also checks the condition ii of Theorem 3.2.
Corollary 3.8.The emptiness problem is decidable for VTAM = = .A careful analysis of the proof of Theorem 3.2 permits to conclude to an EXPTIME complexity for this problem with VTAM = = .Theorem 3.9.The membership problem is NP-complete for VTAM = = .Proof.An NP algorithm is given in the proof of Theorem 3.5.For the NP-hardness, we use a logspace reduction of 3-SAT.Let us consider an instance of 3-SAT with n propositional variables X 1 , . . ., X n and a conjunction of m clauses: where every α i,j is either a variable X k (k ≤ n) or a negation of variable ¬X k .We assume wlog that every variable occurs at most once in a clause.
We consider an encoding t of the given instance as a term over the signature Σ containing the symbols: X 1 , . . ., X n (constants), id , false, ¬ (unary) and ∧ and ∨ (binary).The encoding is: where C ∧ (resp.C ∨ ) is a context built solely with ∧ (resp.∨) and where every δ i,j is either: • δ i,j = false (interpreted as the constant function returning false) if X j does not occur in α i,1 , α i,2 , α i,3 .Now, let us partition the signature Σ with: X 1 , . . ., X n , ∨ ∈ PUSH, id , false, ¬ ∈ INT 1 and ∧ ∈ INT = 1 ; and let consider the memory signature Γ = {0, 1, ∨}.We construct now a VTAM = A = (Γ, =, {q 0 , q 1 }, {q 1 }, ∆) whose transition will, intuitively: • guess an assignment for each constant symbol X k of t, by mean of a non-deterministic choice of one state q 0 or q 1 , • compute the value of t with these assignments, • push each tuple of assignment for each clause, in the contexts C ∨ , • check the coherence of assignments by means of equality tests between the tuples pushed, in the context C ∧ .More formally, we have the following transitions in ∆: We can verify that the above VTAM = A recognizes t iff the instance of 3-SAT has a solution.VTAM = = is closed under union (using the same construction as before) but not under complementation.This is a consequence of the following Theorem.
Theorem 3.10.The universality problem is undecidable for VTAM = = .Proof.We reduce the blank accepting problem for a deterministic Turing machine M. Like in the proof of Theorem 3.6, we encode configurations of M as right-combs on a signature Σ containing the tape and state symbols of M, considered as binary symbols of Σ PUSH and a constant symbol ε in Σ PUSH .A sequence of configurations c 0 , c 1 , . . ., c n (with n ≥ 1) is encoded as Figure 3: The VTAM = = A 3 in the proof of Theorem 3.10.
Figure 4: The VTAM = = A 4 in the proof of Theorem 3.10.and for all 0 ≤ i < n, c i+1 is the successor of c i with M.Moreover, we assume that all the c i have the same length (for this purpose we complete the representations of configurations with blank symbols).
We want to construct a VTAM = = A which recognizes exactly the terms which are not computations of M. Hence, A recognizes all the terms of T (Σ) iff M does not accept the initial blank configuration.
For the construction of A, let us first observe that we can associate to M a VTAM A ✷ which, while reading a configuration c i , will push on the memory its successor c i+1 .The existence of such an automaton is guaranteed by the first fact that for each regular binary relation R, as defined in Section 3.3, there exists a VTAM which, for each (s, t) ∈ R, will push t while reading s, and by the second fact that the language of c i ⊗ c i+1 , hence the relation of successor configuration, are regular.Moreover, since only push operations are performed, we can ensure that A ✷ satisfies the visibly condition.Let us note q ✷ the final state (which is assumed unique wlog) of the VTAM A ✷ .We also use the following VTAMs: A ∀ : a VTAM with (unique) final state q ∀ which, while reading a configuration c i will push on the memory any configuration with same length as c i , A = : a VTAM with final state q = which, while reading a configuration c i will push c i on the memory, A B : a VTAM with final state q B which, while reading a configuration c i will push on the memory a configuration with same length as c i and containing only blank symbols.The VTAM = = A is the union of the following automata: A 1 : a VTAM = = recognizing the terms of T (Σ) which are not representations of sequences of configurations (malformed terms).Its language is actually a regular tree language.A 2 : a VTAM = = recognizing the sequences of configurations f (c n (f (c n−1 , . . .f (c 0 , ε)))) such that c 0 is not initial or c n is not final.Again, this is a regular tree language.A 3 : a VTAM = = recognizing the sequences of configurations with two configurations of different lengths.It contains the transitions rules of A B and the additional transitions described in Figure 3, which perform this test.A 4 : a VTAM = = recognizing the sequences of configurations f (c n (f (c n−1 , . . .f (c 0 , ε)))) such that all the c i have the same length but there exists 0 ≤ i < n such that c i+1 is not the successor of c i by M. This last VTAM = = contains the transitions of A ✷ , A ∀ , A = , and the additional transitions described in Figure 4.
With the transition rules in Figure 4, the automaton A 4 guesses a i < n and, while reading each of the configurations c j with j ≤ i, it pushes the successor configuration of c j , say c ′ j (second column of figure 4).Then, while reading c i+1 A 4 pushes c i+1 , and it checks that c ′ i and c i+1 differ.After that, when reading each of the remaining configurations, A 4 pushes c i+1 (third column of figure 4).
The VTAM = = A 1 to A 4 cover all the cases of term T (Σ) not being an accepting computation of M starting with the initial blank configuration.Hence the language of their union A is T (Σ) iff M does not accept the initial blank configuration.≡ , where ≡ denotes structural equality of terms, defined recursively as the smallest equivalence relation on ground terms such that: Note that it is a regular relation, and that it satisfies the hypothesis of Lemma 3.7 and the condition ii of Theorem 3.2.
Corollary 3.12.The emptiness problem is decidable for VTAM ≡ ≡ .Following the procedure in the proof of Theorem 3.2, we obtain a 2-EXPTIME complexity for this problem and this class.
The crucial property of the relations ≡ and ≡ is that, unlike the above class VTAM = = or the general VTAM R ¬R , they ignore the labels of the contents of the memory.They just care of the structure of these memory terms.A benefit of this property of VTAM ≡ ≡ is that the decision of the membership problem drops to PTIME for this class.
Theorem 3.13.The membership problem is decidable in PTIME for VTAM ≡ ≡ .Proof.Let A = (Γ, ≡, Q, Q f , ∆) be a VTAM ≡ ≡ on Σ and let t be a term in T (Σ).Let sub(t) be the set of subterms of t and let us construct a VTAM A ′ = (Γ, sub(t)×Q, {t}×Q f , ∆ ′ ) on Σ ′ where the symbols of Σ ′ and Σ are the same, and we assume that the symbols in category in the partition of Σ ′ .The transitions of ∆ ′ are obtained by the following transformation of the transitions of ∆.We only describe the construction for the cases INT 1 and INT ≡ 1 with positive constraints.The other cases are similar.
• for every f 7 (q 1 (y 1 ), q 2 (y 2 )) → q(y 1 ) ∈ ∆, we add to ∆ ′ all the transitions: f 7 q 1 , t 1 (y 1 ), q 2 , t 2 (y 2 ) → q, f (t 1 , t 2 ) (y 1 ) such that f (t 1 , t 2 ) ∈ sub(t), • for every f 9 (q 1 (y 1 ), q 2 (y 2 )) −−−−→ y 1 ≡y 2 q(y 1 ) ∈ ∆, we add to ∆ ′ all the transitions as above (in this case, f 9 is assumed a symbol of category INT 1 in Σ ′ ) such that moreover struct(t 1 ) = struct(t 2 ), where struct(s) is defined, like in the proof of Theorem 3.5, as the shape (unlabeled tree) that will have the memory of A after A processed s.The VTAM A ′ can be computed in time O( t 2 × A ).It recognizes at most one term, t, and it recognizes t iff A recognizes t.Therefore, t is recognized by A iff the language of A ′ is not empty.This can be decided in PTIME according to Theorem 2.5.
Even more interesting, the construction for determinization of Section 2.3 still works for VTAM ≡ ≡ .Theorem 3.14.For every Proof.We use the same construction as in the proof of Theorem 2.3, with a direct extension of the construction for INT to INT ≡ .The key property for handling constraints is that the structure of memory (hence the result of the structural tests) is independent from the nondeterministic choices of the automaton.With the visibility condition it only depends on the term read.
Theorem 3.15.The class of tree languages of VTAM ≡ ≡ is closed under Boolean operations.One can construct VTAM ≡ ≡ for union, intersection and complement of given VTAM ≡ ≡ languages whose sizes are respectively linear, quadratic and exponential in the size of the initial VTAM ≡ ≡ .Proof.We use the same constructions as in Theorem 2.4 (VTAM) for union and intersection.For the intersection, in the case of constrained rules we can safely keep the constraints in product rules, thanks to the visibility condition (as the structure of memory only depends on the term read, see the proof of Theorem 3.14).For instance, the product of the INT ≡ 1 rules f 9 q 11 (y 1 ), q 12 (y 2 ) −−−−→ y 1 ≡y 2 q 1 (y 1 ) and f 9 q 21 (y 1 ), q 22 (y 2 ) −−−−→ y 1 ≡y 2 q 1 (y 1 ) is f 9 q 11 , q 21 (y 1 ), q 12 , q 22 (y 2 ) −−−−→ y 1 ≡y 2 q 1 , q 2 (y 1 ).The product of two INT ≡ 1 is constructed similarly.We do not need to consider the product of a rule INT ≡ 1 with a rule INT ≡ 1 , and vice-versa, because in this case the product is empty (no rule is added to the VTAM ≡ ≡ for intersection).For the complementation, we use Theorem 3.14 and completion.
Corollary 3.16.The universality and inclusion problems are decidable for VTAM ≡ ≡ .Proof.This is a consequence of Corollary 3.12 and Theorem 3.15.
3.5.Constrained PUSH Transitions.Above, we always considered constraints in transitions with INT symbols only.We did not consider a constrained extension of the rules PUSH.The main reason is that symbols of a new category PUSH ≡ , which test two memories for structural equality and then push a symbol on the top of them, permit us to construct a constrained VTAM A whose memory language M (A, q) is the set of well-balanced binary trees.This language is not regular, whereas the base of our emptiness decision procedure is the result (Theorem 3.2, Lemma 3.7) of regularity of these languages for the classes considered.
3.6.Contexts as Symbols and Signature Translations.Before looking for some examples of VTAM ≡ ≡ languages, we show a "trick" that (seemingly) adds expressiveness to VTAM ≡ ≡ .One symbol can perform either a PUSH or a POP operation, or make an INT transition (constrained or not), but it cannot combine several of these operations.Here, we propose a way to combine several operations in one symbol, and thus increase the expressiveness of VTAM ≡ ≡ , without losing the good properties of this class.
The trick is to replace symbols by contexts.For instance a context g 2 (g 1 (•, •), g 0 ) can replace a symbol of arity 2. Assume that g 2 is a PUSH symbol, g 1 is an INT 1 symbol with test, and g 0 is an INT 0 symbol.This context first performs a test on the memories of the sons, and then a PUSH operation on the memory kept by g 1 (and on the ⊥ leaf created by g 0 ).Such a combination is normally not possible, and replacing symbols by contexts brings a lot of additional expressiveness.
Here is how we precisely proceed: we want to recognize a language (on a signature Σ) with a VTAM, and we have then to choose the categories for each symbol of the signature (PUSH, POP ij , INT ≡ 1 , ...).As we will see in the examples below, it might be useful in practice to have some extra categories combining the powers of two or more categories of VTAM ≡ ≡ .We can do that still with VTAM ≡ ≡ , by mean of an encoding of the terms of T (Σ).More precisely, we replace some symbols of the initial signature Σ by contexts built with new symbols.For instance, we replace a g ∈ Σ, which will perform the complex operation described above, by the context g 2 (g 1 (•, •), g 0 ).Then, we will have to ensure that the new symbols (in our example g 0 , g 1 and g 2 ) are only used to form the contexts encoding the symbols of Σ.This can easily be done with local information maintained in the state of the automaton.The set of well formed terms, built with new symbols organized in allowed contexts, is a regular tree language.We will call the VTAM ≡ ≡ signature obtained a translation of the initial signature.If L is a tree language on Σ, then c(L) is the translation of L.
In summary, we have shown here a general method for adding new categories of symbols corresponding to (relevant) combinations of operations of VTAM ≡ ≡ , and hence defining extensions of VTAM ≡ ≡ with the same good properties as VTAM ≡ ≡ .By relevant, we mean that some combinations are excluded, like for instance, PUSH + constraint ≡ at the same time (see paragraph above).Such forbidden combination cannot be handled by our method.With similar encodings, we can deal with symbols of arity bigger than 2, e.g.g(•, •, •) can be replaced by g 2 (•, g 1 (•, •)).
Note however first that this encoding concerns the recognized tree, not the memories.For instance, it is not possible to systematically encode the syntactic equality as structural equality (on memories) in this way.And indeed, the decision results are drastically different in the two cases.
Also note that, even if c(L) is accepted by a VTAM, which implies that ¬c(L) is also accepted by a VTAM, it may well be the case that c(¬L) is not recognized by a VTAM.So, the above trick does not show that we can extend our results to a wider class of tree languages.
3.7.Some VTAM ≡ ≡ Languages.The regular tree languages and VPL are particular cases of VTAM languages.We present in this section some other examples of relevant tree languages translatable, using the method of Section 3.6, into VTAM ≡ ≡ languages.
Well balanced binary trees.The VTAM ≡ ≡ with memory signature {f, ⊥}, state set {q, q f }, unique final state q f , and whose rules follow accepts the (non-regular) language of well balanced binary trees build with g and a.
Here a is a constant in Σ INT 0 , and g is in a new category, and is translated into the context g 2 (g 1 (•, •), g 0 ), where Powerlists.A powerlist [18] is roughly a list of length 2 n (for n ≥ 0) whose elements are stored in the leaves of a balanced binary tree.For instance, the elements may be integers represented in unary notation with the unary successor symbol s and the constant 0, and the balanced binary tree on the top of them can be built with a binary symbol g.This data structure has been used in [18] to specify data-parallel algorithms based on divideand-conquer strategy and recursion (e.g.Batcher's merge sort and fast Fourier transform).
It is easy following the above construction to characterize translations of powerlists with a VTAM ≡ ≡ .We do not push on the "leaves", i.e. on the elements of the powerlist, and compute in the higher part (the complete binary tree) as above.
Some equational properties of algebraic specifications of powerlists have been studied in the context of automatic induction theorem proving and sufficient completeness [17].Tree automata with constraints have been acknowledged as a very powerful formalism in this context (see e.g.[9]).We therefore believe that a characterization of powerlists (and their complement language) with VTAM ≡ ≡ is useful for the automated verification of algorithms on this data structure.
Red-black trees.A red-black tree is a binary search tree following these properties: (1) every node is either red or black, (2) the root node is black, (3) all the leaves are black, (4) if a node is red, then both its sons are black, (5) every path from the root to a leaf contains the same number of black nodes.
The four first properties are local and can be checked with standard TA rules.The fifth property make the language red-black trees not regular and we need VTAM ≡ ≡ rules to recognize it.It can be checked by pushing all the black nodes read.We use for this purpose a symbol black ∈ Σ PUSH .
When a red node is read, the number of black nodes in both its sons are checked to be equal (by a test ≡ on the corresponding memories) and only one corresponding memory is kept.This is done with a symbol red ∈ Σ INT ≡ 1 .When a black node is read, the equality of number of black nodes in its sons must also be tested, and a black must moreover be pushed on the top of the memory kept.It means that two operations must be combined.We can do that by defining an appropriate context with the method of Section 3.6.
In [15] a special class of tree automata is introduced and used in a procedure for the verification of C programs which handle balanced tree data structures, like red-black tree.Based on the above example, we think that, following the same approach, VTAM ≡ ≡ can also be used for similar purposes.
q(y 1 ) In Section 3, we have only considered VTAM with constraints testing the memories contents.In this section, we go a bit further and add to VTAM R ¬R some Bogaert-Tison constraints [4], i.e. equality and disequality tests between brother subterms in the term read by the automaton.
We consider two new categories for the symbols which we call BTINT 1 and BTINT 2 , for "Bogaert-Tison Internal".A transition with a symbol in one of these categories will make no test on the memory contents, but rather an equality or disequality test between the brother subterms directly under the current position of computation.In Figure 5, we describe the new transitions categories.We use the same notation as in [4] for the constraints.Note that again, we only allow Bogaert-Tison constraints in internal rules.
The acceptance of terms of T (Σ) and languages of term and memories are defined and denoted as in Section 2.1.

The definition of complete BTVTAM R
¬R is the same as before.Every BTVTAM R ¬R can be completed (with a polynomial overhead) by the addition of a trash state q ⊥ (the construction is similar to the one for VTAM : for all f ∈ Σ BTINT 1 ∪ Σ BTINT 2 for all q 1 , q 2 ∈ Q, there are at most two rules in ∆ with left-member f q 1 (y 1 ), q 2 (y 2 ) , and if there are two, then their constraints have different signs.

Theorem 4.2. For every BTVTAM
Proof.We use, again, the same construction as in the proof of Theorem 2.3, with a direct extension of the construction for INT to INT ≡ and BTINT.As mentioned in Theorem 3.14, the extension works for INT ≡ because the results of the tests are independent from the non-deterministic choices of the automaton.For BTINT it is exactly the same (the brother terms are not changed by the automaton!).Proof.We use the same constructions as in Theorem 2.4 for union and intersection.For the intersection, as in Theorem 3.15, the constraints (even Bogaert-Tison tests) can be safely kept in product rules, thanks to the visibility condition.For the complementation, we use Theorem 4.2 and complementation.
The proof of the following theorem follows the same idea as the proof for Bogaert-Tison automata [4], but we need here to take care of the structural constraints on the memory contents.A consequence is that the complexity of emptiness decision is much higher.
Theorem 4.4.The emptiness problem is decidable for BTVTAM ≡ ≡ .Proof.Let A be a BTVTAM ≡ ≡ .First we determinize it into A det and assume that A det is also complete.Then, we delete the rules BTINT 1 of the form: f q 1 (y 1 ), q 2 (y 2 ) − −− → 1=2 q(y 1 ).with q 1 distinct from q 2 (idem for BTINT 2 rules) because they can't be used (the automaton is deterministic so one term cannot lead to two different states).
We call the newly obtained automaton A new .It is still deterministic and recognizes the same language as A det .Actually, the careful reader may notice that A new is not a true BTVTAM ≡ ≡ , because some unconstrained rules may involve symbols in BTINT in this automaton.However, it is just an intermediate step in the construction of another automaton A ′ below.Now, we consider the remaining BTINT 1 or BTINT 2 rules with negative Bogaert-Tison constraints, which are of the form: f q 1 (y 1 ), q 1 (y 2 ) − −− → 1 =2 q(y 1 ) (or q(y 2 )).We denote them by R 1 , ..., R i , ..., R N , and denote by q i the state in the left member of R i , for each i ≤ N .We also denote the corresponding BTINT 1 or BTINT 2 rules by S 1 ,...., S i ,..., S N .Note that, since A det is deterministic and complete, we can associate to each rule of BTINT i , whose constraint is negative, a unique rule of BTINT i with a positive constraint and the same states in its left member.So, the state in the left member of S i is the same q i as for R i .
It is important to notice that if a rule R i can effectively be used, then there must exist two distinct terms leading to the state q i (we will call them witnesses).If not, the rule can be removed.
So, our purpose is now to find, for each rule R i , whether two witnesses exist or not.We let R be initially {R 1 , . . ., R N }.Suppose that at least one R i rule can be used, and consider a run on a term t that uses such a rule.We consider an innermost application of a rule R i in this run on a subterm f (t 1 , t 2 ).The run on t 1 and the run on t 2 both lead to the state q i , without any use of an R j rule.
Let us remove all the R i rules from A new , and we remove all the equality tests in the S i rules.Let A ′ be the resulting automaton.It is a deterministic VTAM ≡ ≡ (considering the symbols in BTINT as INT symbols in this new automaton), and each term in L(A ′ , q i ) can be transformed (we will call it BT-transformation) into a term in L(A new , q i ): each time we use a modified S i rule, for instance of type BTINT 1 , on a subtree f (t 1 , t 2 ), we replace t 2 with t 1 so that the equality test is satisfied (and the resulting memory is unchanged).Important: all the replacements must be performed bottom-up.The proof of the emptiness decidability of VTAM ≡ ≡ (Corollary 3.12) is constructive, hence if we choose a reachable state q j , we can find a term in L(A ′ , q j ) to this state, and then convert it into a witness.So, we can find a first witness t A ∈ L(A new , q j ).
If no witness can be found, then all the R i rules are useless and we can definitely remove them all.Otherwise, we still need to find another witness, and if there is at least one such other witness, then one of them can be recognized without using a R i rule.We can construct a VTAM ≡ ≡ recognizing all the terms whose BT-transformation leads to t A .To design it, we read t A top-down (knowing the state of A ′ at each node), and each time we see a subterm f (t 1 , t 2 ) to which a modified S i rule has to be applied, for instance a modified BTINT 1 (resp.BTINT 2 ) rule, the right (resp.left) son of f only needs to be a term in L(A ′ , q i ), and the left (resp.right) son of f only needs to be BT-transformed into t 1 (resp.t 2 ).Once this VTAM ≡ ≡ is constructed, we can combine it with A ′ in order to obtain a VTAM ≡ ≡ recognizing all the terms leading A ′ to q j (the state reached by A ′ on t A ) except the terms whose BT-transformation is t A .Then we find another term in L(A ′ , q j ) (if it exists) and its BT-transformation is not t A : it is actually another witness t B .
When we have two witnesses for a rule R j , we remove it from R, and we add this rule R j to A ′ , but without the disequality test.The automaton A ′ keeps its good property: a term t leading A ′ to some state q can be BT-transformed into a term leading A new to state q: when we "meet" the use of a rule formerly in the set R on f (t 1 , t 1 ) during the bottom-up exploration of t, we replace the right (for a rule that was of type BTINT 1 and with negative constraints) or the left son (otherwise) by a witness different from t 1 , so that the disequality test is satisfied.Note that even if t 1 is a witness, we can do so because we have found two witnesses.
With the new rule in A ′ we look for 2 witnesses for some remaining R i rule.Again, we can show that if a couple of witnesses exists, then at least one couple can be found without any use of the remaining R i rules.When we find a first witness t A for a remaining rule R j , we can find another one (if it exists) using approximately the same technique as previously: we read t A top-down, and when we see a rule formerly in R, used on f (t 1 , t 2 ) (e.g. a rule formerly of type BTINT 1 with a negative constraint), we just go on recursively, saying that the left son must be a term whose BT-transformation is t 1 , and the right son must be either: • a term whose BT-transformation is t 2 , • or, if our BT-transformation would change f (t 1 , t 1 ) into f (t 1 , t 2 ), a term whose BTtransformation is t 1 .As previously, we construct a VTAM ≡ ≡ , fully using the Boolean closure of this class, that recognizes the terms in L(A ′ , q j ) (the state reached by A ′ on t A ), except those whose BTtransformation is t A , and therefore we can find another witness (if it exists) t B .
We continue to use this method, finding couples of witnesses, until there is no rule in the set R anymore, or until we are not able to find a new couple of witnesses anymore: in that latter case, we remove the remaining R i rules because they are useless.
So, now we use the final version of A ′ obtained in order to find a term leading to a final state, and since we have a couple of witnesses for each rule formerly in the set R, we can BT-transform it into a term accepted by A new (hence by A).If such a term does not exist, the language recognized by A new (i.e. the language recognized by A) is empty.

Conclusion
Having a tree memory structure instead of a stack is sometimes more relevant (even when the input functions symbols are only of arities 1 and 0).We have shown how to extend the visibly pushdown languages to such memory structures, keeping determinization and closure properties of VPL.Our second contribution is then to extend this automaton model, constraining the transition rules with some regular conditions on memory contents.The structural equality and disequality tests appear to a be a good class of constraints since we have then both decidability of emptiness and Boolean closure properties.Moreover, they can be combined (while keeping decidability and closure results) with equality and disequality tests a la [4], operating on brothers subterms of the term read.
Several further studies can be done on the automata of this paper.For instance, the problem of the closure of the corresponding tree languages under certain classes of term rewriting systems is particularly interesting, as it can be applied to the verification of infinite state systems with regular model checking techniques.It could be interesting as well to study how the definition of VTAM can be extended to deal with unranked trees, with the perspective of applications to problems related to semi-structured documents processing.
(1) Q 1 (y 1 ), Q 2 (y 2 ), R(y 1 , y 2 ) ⇒ Q 3 (y 1 ) (2) These clauses have a least Herbrand model.We write [[Q]] the interpretation of Q in this model.This is the language recognized by the automaton in state Q.
The goal is to prove that, for every Q, [[Q]] is recognized by a finite tree automaton We use a selection strategy, with splitting and complete the rules (1)-( 4) above.We show that the completion terminates and that we get out of it a tree automaton which accepts exactly the memory contents.Splitting will introduce nullary predicate symbols (propositional variables).
We consider the following selection strategy.Let E 1 be the set of literals which contain at least one function symbol and E 2 be the set of negative literals We introduce the procedure by starting to run the completion with the selection strategy, before showing the general form of the clauses we get.
We let C be the set of clauses obtained by repeated applications of resolution with splitting, with the above selection strategy (a priori C could be infinite).We claim that all It follows that all clauses of C are of the above form.Since there are only finitely many such clauses, C is finite and computed in finite (exponential) time.Now, we let A be the alternating tree automaton defined by clauses (P 1 ) and (P 2 ) (and automata clauses defining the S states).Let, for any state Q, [[Q]] A be the language accepted in state Q by A. We claim that [ To prove this, we first show (the proof is omitted here) that NE Assume, by contradiction, that there is a term t and a predicate symbol Q such that all proofs of Q(t) using the clauses in C involve at least a clause, which is not an automaton clause.Then, considering an appropriate sub-proof, there is a term u and a predicate symbol P such that all proofs of P (u) involve at least one non-automaton clause and there is a proof of P (u) which uses exactly one non-automaton clause, at the last step of the proof.
We investigate all possible cases for the last clause used in the proof of P (u) and derive a contradiction in each case.Clause I 1 : The last step of the proof is P 1 (u 1 ), . . ., P n (u 1 ), P ′ 1 (u 2 ), . . ., P ′ m (u 2 ), P ′′ 1 (f (u 1 , u 2 )), . . ., P ′′ k (f (u 1 , u 2 )) and we assume u = f (u 1 , u 2 ).Assume also that, among the proofs we consider, k is minimal.(If k = 0 then we have a push clause, which is supposed not to be the case).By hypothesis, for all i, u

Figure 2 : 3 . 1 .
Figure 2: New transition categories for VTAM R ¬R .3.1.Definitions.Assume given a fixed equivalence relation R on T (Γ).We consider now two new categories for the symbols of Σ: INT R 1 and INT R 2 , in addition to the eight previous categories of page 6.The new categories correspond to the constrained versions of the transition rules INT 1 and INT 2 presented in Figure 2. The constraint y 1 R y 2 in the two first rules of Figure 2 is called positive and the constraint y 1 ¬R y 2 in the two last rules is called negative.We shall not extend the rules PUSH and POP with constraints for some reasons explained in section 3.5.A ground term t rewrites to s by a constrained rule f q 1 (y 1 ), q 2 (y 2 ) −−−− →

Lemma 3 . 3 .
Let R satisfy the hypotheses i and ii of Theorem 3.2, and let A

Figure 5 : 4 .
Figure 5: New transition categories for BTVTAM R ¬R .4. Visibly Tree Automata with Memory and Structural Constraints and Bogaert-Tison Constraints

Definition 4 . 1 .
A visibly tree automaton with memory and constraints and Bogaert-Tison tests (BTVTAM R ¬R ) on a signature Σ is a tuple (Γ, R, Q, Q f , ∆) where Γ, Q, Q f are defined as for TAM, R is an equivalence relation on T (Γ) and ∆ is a set of rewrite rules in one of the above categories: PUSH, POP 11 , POP 12 , POP 21 , POP 22 , INT 0

Theorem 4 . 3 .
The class of tree languages of BTVTAM ≡ ≡ is closed under Boolean operations.

( 1 )
If the clause contains a negative literal ¬R(u, v) or a negative literal ¬S(u) where either u, v is not a variable, then select such literals only.This case is ruled out in what follows (2) If the clause contains at least one negated propositional variable, select the negated propositional variables only.This case is ruled out in what follows(3) If E 1 ∩ E 2 = ∅, then select E 1 ∩ E 2 (4) If E 1 = ∅ and E 1 ∩ E 2 = ∅, then select E 1 (5) If E 1 = ∅ and E 2 = ∅,then select the negative literals ¬R(x, y) and ¬S(x) if any, otherwise select E 2 (6) Otherwise, select the only literal of the clause In what follows (and precedes), selected literals are underlined.

6 .
Propositional clauses.(E 1 ) NE S 1 , . . ., NE Sn ⇒ NE S (E 2 ) ⇒ NE S (E 3 ) P 1 (a), . . .P n (a) ⇒ NE S Every resolution step using the selection strategy of two of the above clauses yield a clause in the above set POP+PUSH: yields an alternating clause (A 1 ) and a split clause (S 4 ).INT + PUSH: yields a Push clause or an intermediate clause alternating + PUSH: yields an intermediate clause (I 1 ) or (I 2 ).split + R: yields a split clause (S) 2 or (S 3 ) or an intermediate clause (I 3 ) or (I 4 ).(S 2 ) + PUSH: yields clauses (S 1 ) and push clauses.Note that here, we use the property of the relation R to split clauses, which may involve predicates Q R j i .(S 3 )+ PUSH: yields push clause and split clauses (S 4 ).(S 4 )+ PUSH: yields split clauses (S 5 ) or propositional clause (E 3 ).(S 5 )+ PUSH: yields split clauses (S 5 ) or propositional clause (E 1 ).
{P 1 ,...,Pn} is in C iff [[P 1 ]] A ∩ . . .∩ [[P n ]] A = ∅.Then observe that [[Q]] is also the interpretation of Q in the least Herbrand model of C: indeed, all computations yielding C are correct.Since[[Q]] A ⊆ [[Q]] is trivial, we only have to prove the converse inclusion.For every t ∈ [[Q]] there is a proof of Q(t) using the clauses in C.

INT, INT 0 is for constant symbols and INT 1 , INT 2 are for binary symbols and differ according to the memory which is kept. Similarly, there are four variants of POP transitions, POP 11
, . . ., POP 22 .Moreover, each POP rule has a variant, which reads an empty memory (i.e. the symbol ⊥).Q, Q f , ∆) such that every rule of ∆ belongs to one of the above categories PUSH, POP 11 , POP 12 , POP 21 , POP 22 , INT 0 , INT 1 , INT 2 .2.2.Expressiveness, Comparison.Standard bottom-up tree automata are particular cases of VTAM (simply assume all the symbols of the signature in INT 0 or INT 1 ).
).The definition of deterministic VTAM R ¬R is based on the same conditions as for VTAM for the function symbols in categories of PUSH 0 , PUSH, POP 11 , . . ., POP 22 , INT 1 , INT 2 .For the function symbols of INT R 1 , INT R 2 , we have the following condition: for all R ¬R in Section 3.1).The definition of deterministic BTVTAM R ¬R is based on the same conditions as for VTAM R ¬R for the function symbols in categories PUSH 0 , PUSH, POP 11 , . . ., POP 22 , INT 1 , INT 2 , INT R 1 , INT R 2 , and for the function symbols of BTINT 1 , BTINT 2 , we use the same kind of conditions as for INT R 1 , INT R 2 ′′i ]] A .In particular, if we consider the last clause used in the proof of P ′′ 1 ∈ [[P i ]] A , u 2 ∈ [[P ′ i ]] A and f (u 1 , u 2 ) ∈ [[P k (u): Q 1 (x), . . ., Q r (x), Q ′ 1 (y), . . ., Q ′ s (y) ⇒ P ′′ k (f (x, y))