APPROXIMATING A BEHAVIOURAL PSEUDOMETRIC WITHOUT DISCOUNT FOR PROBABILISTIC SYSTEMS

. Desharnais, Gupta, Jagadeesan and Panangaden introduced a family of behavioural pseudometrics for probabilistic transition systems. These pseudometrics are a quantitative analogue of probabilistic bisimilarity. Distance zero captures probabilistic bisimilarity. Each pseudometric has a discount factor, a real number in the interval (0 , 1]. The smaller the discount factor, the more the future is discounted. If the discount factor is one, then the future is not discounted at all. Desharnais et al. showed that the behavioural distances can be calculated up to any desired degree of accuracy if the discount factor is smaller than one. In this paper, we show that the distances can also be approximated if the future is not discounted. A key ingredient of our algorithm is Tarski’s decision procedure for the ﬁrst order theory over real closed ﬁelds. By exploiting the Kantorovich-Rubinstein duality theorem we can restrict to the existential fragment for which more eﬃcient decision procedures exist.


Introduction
For systems that contain quantitative information, like, for example, probabilities, time and costs, several behavioural pseudometrics (and closely related notions) have been introduced (see, for example, [6,8,10,14,15,18,19,20,21,28,33]).In this paper, we focus on probabilistic transition systems, which are a variant of Markov chains.Desharnais, Gupta, Jagadeesan and Panangaden [18] introduced a family of behavioural pseudometrics for these systems.These pseudometrics assign a distance, a real number in the interval [0, 1], to each pair of states of the probabilistic transition system.The distance captures the behavioural similarity of the states.The smaller the distance, the more alike the states behave.The distance is zero if and only if the states are probabilistic bisimilar, a behavioural equivalence introduced by Larsen and Skou [26].
The pseudometrics of Desharnais et al. are defined via real-valued interpretations of Larsen and Skou's probabilistic modal logic.Formulae assume truth values in the interval [0, 1].Conjunction and disjunction are interpreted using the lattice structure of the unit interval.The modality a is interpreted arithmetically by integration.The behavioural distance between states s 1 and s 2 is then defined as the supremum over all formulae ϕ of the difference in the truth value of ϕ in s 1 and in s 2 . 1  The definition of the behavioural pseudometrics of Desharnais et al. is parametrized by a discount factor δ, a real number in the interval (0, 1].The smaller the discount factor, the more (behavioural differences in) the future are discounted.In the case that δ equals one, the future is not discounted.All differences in behaviour, whether in the near or far future, contribute alike to the distance.For systems that (in principle) run forever, we may be interested in all these differences and, hence, in the pseudometric that does not discount the future.
In [16], Desharnais et al. presented an algorithm to approximate the behavioural distances for δ smaller than one.The first and third author [7] presented also an approximation algorithm for δ smaller than one.
There is a fundamental difference between pseudometrics that discount the future and the one that does not.This is, for example, reflected by the fact that all pseudometrics that discount the future give rise to the same topology, whereas the pseudometric that does not discount the future gives rise to a different topology (see, for example, [18, page 350]).As a consequence, it may not be surprising that neither approximation algorithm mentioned in the previous paragraph can be modified in an obvious way to handle the case that δ equals one.
The main contribution of this paper is an algorithm that approximates behavioural distances in case the discount factor δ equals one.Starting from the logical definition of the pseudometric by Desharnais et al., we first give a characterisation of the pseudometric as the greatest (post-)fixed point of a functional on a complete lattice [0, 1] S , where S is the set of states of the probabilistic transition system in question.This functional is closely related to the Kantorovich metric [24] on probability measures.Next, we dualize this characterization exploiting the Kantorovich-Rubinstein duality theorem [25].Subsequently, we show, exploiting the dual characterization, that a pseudometric being a post-fixed point can be expressed in the existential fragment of the first order theory over real closed fields.Based on the fact that this first order theory is decidable, a result due to Tarski [31], we show how to approximate the behavioural distances.Finally, we discuss an implementation of our algorithm in Mathematica.
Exploiting the techniques put forward in this paper, we have also developed an algorithm to approximate the behavioural pseudometric that is presented in [3].The other algorithm can be found in [30].
1 More generally, de Alfaro [13] and McIver and Morgan [27] have given real-valued interpretations to the modal mu-calculus following this pattern.Moreover, de Alfaro has shown that the behavioural pseudometrics induced by mu-calculus formulae agree with those of [18].

Systems and pseudometrics
Some basic notions that will play a role in the rest of this paper are presented below.First we introduce the systems of interest: probabilistic transition systems.Definition 2.1.A probabilistic transition system is a tuple S, π consisting of • a finite set S of states and For states s and s ′ , π(s, s ′ ) is the probability of making a transition to state s ′ given that the system is in state s.Each state s either has no outgoing transitions (s →) or a transition is taken with probability 1 (s →).To simplify the presentation, we do not consider the case that a state s may refuse to make a transition with some probability, that is, s ′ ∈S π(s, s ′ ) ∈ (0, 1).However, all our results can easily be generalized to handle that case as well (see [30]).We also do not consider transitions that are labelled with actions.All our results can also easily be modified to handle labelled transitions (see [30]).In the labelled case, the definition of probabilistic transition system is a mild generalisation of the notion of Markov chain.
We restrict to rational transition probabilities in order that probabilistic transitions systems be finitely representable.Here we assume that rational numbers are represented as pairs of integers in binary.We believe that the algorithm presented in this paper could be adapted to accommodate transition probabilities that are algebraic numbers, but we do not pursue this question here.
In the rest of this paper, we will use the following probabilistic transition system as our running example.
) ) We consider states of a probabilistic transition system behaviourally equivalent if they are probabilistic bisimilar [26].
Definition 2.3.Let S, π be a probabilistic transition system.An equivalence relation R on the set of states S is a probabilistic bisimulation if s 1 R s 2 implies s∈E π(s 1 , s) = s∈E π(s 2 , s) for all R-equivalence classes E. States s 1 and s 2 are probabilistic bisimilar, denoted s 1 ∼ s 2 , if s 1 R s 2 for some probabilistic bisimulation R.
Note that probabilistic bisimilar states s 1 and s 2 have the same probability of transitioning to an equivalence class E of probabilistic bisimilar states.
Example 2.4.Consider the probabilistic transition system of Example 2.2.The smallest equivalence relation containing (s 3 , s 5 ) is a probabilistic bisimulation.Hence, the states s 3 and s 5 are probabilistic bisimilar.
The behavioural pseudometrics that we study in this paper yield pseudometric spaces on the state space of probabilistic transition systems.Definition 2.5.A 1-bounded pseudometric space is a pair (X, d X ) consisting of a set X and a distance function d X : X × X → [0, 1] satisfying (1) for all x ∈ X, d X (x, x) = 0, (2) for all x, y ∈ X, d X (x, y) = d X (y, x), and (3) for all x, y, z ∈ X, d X (x, z) ≤ d X (x, y) + d X (y, z).Instead of (X, d X ) we often write X and we denote the distance function of a metric space X by d X .
Example 2.6.Let X be a set.The discrete metric d X : X × X → [0, 1] is defined by A (1-bounded) pseudometric space differs from a (1-bounded) metric space in that different points may have distance zero in the former and not in the latter.Since different states of a system may behave the same, such states will have distance zero in our behavioural pseudometrics.
In the characterization of a behavioural pseudometric in Section 4 nonexpansive functions play a key role.Definition 2.7.Let X be a 1-bounded pseudometric space.A function f : The set of nonexpansive functions from X to [0, 1] is denoted by X ------[0, 1].
Example 2.8.If the set X is endowed with the discrete metric, then every function from X to [0, 1] is nonexpansive.

Behavioural pseudometrics
Desharnais, Gupta, Jagadeesan and Panangaden [18] introduced a family of behavioural pseudometrics for probabilistic transitions systems.Below, we will briefly review the key ingredients of their definition.
To define their behavioural pseudometrics, Desharnais et al. defined a real-valued semantics of a variant of Larsen and Skou's probabilistic modal logic [26].We describe this variant, adapted to the case of unlabelled transition systems, in Definition 3.1.
Definition 3.1.The logic L is defined by The main difference between the above logic and the one of Larsen and Skou is that we have ♦ϕ and ϕ ⊖ q whereas they combine the operators ♦ and ⊖q into one.Since they consider labelled transitions, they use the notation a q for this combined operator.
Desharnais et al. provided a family of real-valued interpretations of the logic.That is, given a probabilistic transition system and a discount factor δ, the interpretation gives a quantitative measure of the validity of a formula ϕ of the logic in a state s of the system.The interpretation ϕ δ (s) is a real number in the interval [0, 1].It measures the validity of the formula ϕ in the state s.This real number can roughly be thought of as the probability that ϕ is true in s.Definition 3.2.Given a probabilistic transition system S, π and a discount factor δ ∈ (0, 1], for each ϕ ∈ L, the function ϕ δ : S → [0, 1] is defined by Example 3.3.Consider the probabilistic transition system of Example 2.2.For this system, ♦true δ (s 3 ) = δ and ♦true δ (s 4 ) = 0.
Given a discount factor δ ∈ (0, 1], the behavioural pseudometric d δ assigns a distance, a real number in the interval [0, 1], to every pair of states of a probabilistic transition system.The distance is defined in terms of the logical formulae and their interpretation.Roughly speaking, the distance is captured by the logical formula that distinguishes the states the most.
Definition 3.4.Given a probabilistic transition system S, π and a discount factor δ ∈ (0, 1], the distance function d δ : S × S → [0, 1] is defined by Example 3.5.Consider the probabilistic transition system of Example 2.2.For example, the states s 3 and s 4 are δ apart.This distance is witnessed by the formula ♦true.The distances 2 are collected in the following table.Since a distance function is symmetric and the distance from a state to itself is zero, we do not give all the entries.
Proof.First, observe that 2 These distances were obtained by ad-hoc methods including Proposition B.5 and checked for numerous different discount factors using the algorithm described in [7].
States having distance zero defines an equivalence relation.That is, for a pseudometric d on states, the relation ≡ d on states defined by is an equivalence relation.We denote the equivalence class that contains the state s by is a quantitative analogue of probabilistic bisimilarity.This behavioural equivalence is exactly captured by those states that have distance zero.
Assume that e i is an element of E i .By induction, the function ϕ δ restricted to E i is constant.Hence, • We show that the relation Without loss of any generality, we may assume that ϕ s ′ δ (s) > ϕ s ′ δ (s ′ ).
Hence, there exists a rational In [16], Desharnais et al. present a decision procedure for the behavioural pseudometric d δ when δ is smaller than one.Let us briefly sketch their algorithm.They define the depth of a logical formula as follows.
One can easily verify that ϕ δ (s 1 )− ϕ δ (s 2 ) ≤ δ depth(ϕ) for each ϕ ∈ L. This suggests that one can compute d δ to any desired degree of accuracy by restricting attention to formulae ϕ of a fixed modal depth.Clearly, there exist infinitely many formulae of each fixed modal depth.Nevertheless, Desharnais et al. show how to construct a finite subset F n of the logical formulae of at most depth n such that In this way, d δ (s 1 , s 2 ) can be approximated up to arbitrary accuracy provided δ is smaller than one.

A fixed point characterization and its dual
For the rest of this paper, we focus on the behavioural pseudometric that does not discount the future.That is, we concentrate on the pseudometric d 1 .Below, we present an alternative characterization of this pseudometric.In particular, we characterize d 1 as the greatest (post-)fixed point of a function ∆ from a complete lattice to itself.This characterization can be viewed as a quantitative analogue of the greatest fixed point characterization of bisimilarity [29].
We also dualize the definition of ∆ exploiting the Kantorovich-Rubinstein duality theorem [25].As we will see in Section 5, this dual characterization will allow us to define ∆ as the solution to a minimization problem rather than a maximization problem, as above.
In turn this will allow us to capture the fact that a pseudometric is a post-fixed point of ∆ in the existential fragment of the first order theory over real closed fields.
For the rest of this paper, we fix a probabilistic transition system S, π .We endow the set of pseudometrics on S with the following order.Definition 4.1.The relation ⊑ on 1-bounded pseudometrics on S is defined by Note the reverse direction of ⊑ and ≥ in the above definition.We decided to make this reversal so that d 1 is a greatest fixed point, in analogy with the characterization of bisimilarity, rather than a least fixed point.This choice has no impact on any results in this paper.Proof.Obviously, ⊑ is a partial order.The top element is the 1-bounded pseudometric ⊤ defined by ⊤(s 1 , s 2 ) = 0.The bottom element is the 1-bounded pseudometric ⊥ defined by Let D be a nonempty set of 1-bounded pseudometrics on S. The meet of D is the 1-bounded pseudometric D defined by The join of D can be expressed in terms of the meet of D (see, for example, [12,Lemma 2.15]).
Whereas meets of pseudometrics are computed pointwise using the supremum on [0,1], joins of pseudometrics are not.
Next, we introduce a function from this complete lattice to itself of which the behavioural pseudometric d 1 is the greatest fixed point.
Note that we can write max above rather than sup since (S, d) ------[0, 1], being a closed subset of the product space [0, 1] S , is compact.
The functional ∆ is closely related to the Kantorovich metric [24] on probability measures.In the definition of that metric, nonexpansive functions play a key role. 3roposition 4.4.∆(d) is a 1-bounded pseudometric on S.
Now that we have this alternative representation of ∆(d), checking that it satisfies the three conditions of Definition 2.5 is straightforward.Since ∆(d) is a 1-bounded pseudometric on S and ∆ is order-preserving, we can conclude from Tarski's fixed point theorem [32, Theorem 1] that ∆ has a greatest fixed point.We denote the greatest fixed point of ∆ by gfp(∆).This greatest fixed point of ∆ is also the greatest post-fixed point of ∆ (see, for example, [12,Theorem 4.11] 4 ).Proof.We first prove that d 1 is a post-fixed point of ∆.That is, we show that ∆(d 1 )(s 1 , s 2 ) ≤ d 1 (s 1 , s 2 ).To prove this, we distinguish the following three cases.
• If s 1 → and s 2 → then the property is vacuously true.
• If s 1 → and s 2 →, or s 1 → and s 2 →, then the formula ♦true witnesses that the states s 1 and s 2 have distance one.• Assume that s 1 → and s 2 →.According to [6, Proposition 39], the set In [12, page 94], such a d is called a pre-fixpoint.arbitrary accuracy by some ϕ 1 .As a consequence, Next we prove that d 1 is the greatest post-fixed point of ∆.Assume that d is a postfixed point of ∆.We have to show that d ⊑ d 1 .That is, d 1 (s 1 , s 2 ) ≤ d(s 1 , s 2 ).We restrict our attention to the case that s 1 → and s 2 →.It suffices to show that for all ϕ ∈ L. This can be proved by structural induction on ϕ.We consider only the nontrivial case: ♦ϕ.
A similar result can be obtained by combining Theorem 40 and 44 of [4].Let us recall (a minor variation of) the Kantorovich-Rubinstein duality theorem.Let X be a 1-bounded compact pseudometric space.Let µ 1 and µ 2 be Borel probability measures on X.We denote the set of Borel probability measures on the product space with marginals µ 1 and µ 2 , that is, the Borel probability measures µ on X 2 such that for all Borel subsets B of X, µ(B × X) = µ 1 (B) and µ(X × B) = µ 2 (B), by µ 1 ⊗ µ 2 .The Kantorovich-Rubinstein duality theorem tells us max The following proposition, which is a consequence of the Kantorovich-Rubinstein duality theorem, defines ∆(d) as a minimum as opposed to the maximum in Definition 4.3.Proposition 4.7 ([7, Corollary 19]).Let d be a 1-bounded pseudometric on S. Let s 1 , s 2 ∈ S such that s 1 → and s 2 →.Then Proof.Since the set S is finite, the space (S, d) is compact.The probability distributions π(s 1 , •) and π(s 2 , •) define Borel probability measures on (S, d).Applying the Kantorovich-Rubinstein gives us the desired result.

The algorithm
Before we present our algorithm, we first show that the fact that a pseudometric is a post-fixed point of ∆ can be expressed in (the existential fragment of) the first order theory over real closed fields.This will allow us to exploit Tarski's decision procedure to approximate the behavioural pseudometric.
For the rest of this paper, we assume that the probabilistic transition system S, π has N states s 1 , s 2 , . . ., s N .Instead of π(s i , s j ) we will write π ij .We represent a 1-bounded pseudometric on the set S of states of the probabilistic transition system, as (the values of) a collection of real valued variables d ij .
The fact that d is a 1-bounded pseudometric can now be captured as follows.
Definition 5.1.The predicate pseudo(d) is defined by Furthermore, the fact that d is a post-fixed point of ∆ can be captured as follows.
Definition 5.2.The predicate post-fixed(d) is defined by post-fixed(d) Now we are ready to present our algorithm.Consider the states s i 0 and s j 0 .We restrict our attention to the case that s i 0 → and s j 0 →.In the other cases the computation of the distance is trivial.
In our algorithm, we use the algorithm tarski that takes as input a sentence of the first order theory of real closed fields and decides the truth or falsity of the given sentence.The fact that there exists such an algorithm was first proved by Tarski [31].
Note that the argument of tarski is a sentence that is part of the existential fragment of the first order theory over real closed fields.For this fragment there are more efficient decision procedures than for the general theory (see, for example, [2]).
Let us sketch a correctness proof of our algorithm.Assume that d 1 (s i 0 , s j 0 ) ∈ [ℓ, u].We distinguish the following three cases.
• If u − ℓ ≤ ǫ, then the algorithm obviously returns the desired result.
• Assume that u−ℓ>ǫ and suppose that tarski returns true.Then there exists a 1-bounded pseudometric d that is a post-fixed point of ∆ and d(s i 0 , s j 0 ) ≤ m.Since d 1 is the greatest post-fixed point of ∆, we have that d • Assume that u − ℓ > ǫ and suppose that tarski returns false.Then d(s i 0 , s j 0 ) > m for every 1-bounded pseudometric d that is a post-fixed point of ∆.Since d 1 is a post-fixed point of ∆, we have that d 1 (s i 0 , s j 0 ) > m.By assumption Obviously, the algorithm terminates.

Conclusion
This paper combines a number of ingredients, known already for a long time, including the Kantorovich-Rubinstein duality theorem of the fifties, Tarski's fixed point theorem of the forties and Tarski's decision procedure for the first order theory of real closed fields of the thirties.We show that the behavioural pseudometric d 1 , which does not discount the future, can be approximated up to an arbitrary accuracy.While the combination of the above results into a decision procedure for the pseudometric is not technically difficult, we do solve a problem that has been open since 1999.Most of the results in Section 3 and 4 are (variations on) known results.As far as we know, the results in Section 5 and Appendix B are new.The techniques exploited in this paper have also been used to approximate other behavioural pseudometrics that do not discount the future such as, for example, the one presented in [3].Furthermore, our algorithm can easily be adjusted to the discounted case.
Since the satisfiability problem for the existential fragment of the first order theory of real closed fields is in PSPACE, it is not surprising that our algorithm can only handle small examples as we have shown in Appendix B. As a consequence, the quest for practical algorithms to approximate d 1 is still open.Since the closure ordinal of ∆ is ω, as proved in Appendix A, an iterative algorithm might be feasible.
As future work, we plan to apply our techniques to obtain approximation algorithms for other behavioural pseudometrics such as, for example, the one for systems that combine nondeterminism and probability presented in [15] and the pseudometric for weak probabilistic bisimilarity in [17].In the latter case the pseudometric can be characterized as the fixed point of a functional based on the Kantorovich and Hausdorff metrics.These can easily be encoded in the first-order theory of the reals.However, the need to consider the transitive closure of the silent transition relation suggests that some non-trivial extension of the work presented here is called for.
Hence, for this system we need ω iterations.
In the rest of this appendix, we prove that we need at most ω iterations for any system.This tells us that the closure ordinal of ∆ is ω, that is, ∆(d ω ) = d ω .As a consequence, d ω is the greatest fixed point of ∆ (see, for example, [12,Example 4.13]).As we will see below, the fact that d ω is a fixed point of ∆ follows from the facts that ∆ is order-preserving (Proposition 4.5) and Lipschitz (Proposition A.6).
In [17, Let The ratio ρ(d 1 , d 2 ) of d 1 and d 2 is defined by Note that we never divide by zero since d 1 ⊑ d 2 and, hence, Below, we will use the convention that the minimum of the empty set is one and the maximum of the empty set is zero.
Given pseudometrics d 1 and d 2 such that d 1 ⊑ d 2 and given an f ∈ (S, d 1 ) ------[0, 1], we next show that there exists a . Therefore g f (s 1 ) = g f (s 2 ) and, hence, the property is vacuously true.Let and Next, we bound f − g f from above.
for all s ∈ S. Furthermore, and Now we can prove that ∆ is Lipschitz, that is, max for some constant λ.
Proposition A.6.Let d 1 ⊑ d 2 .For all s 1 , s 2 ∈ S, Finally, we prove that the closure ordinal of ∆ is ω.Obviously, for a state s without outgoing transitions, we have that τ ω (s) = 1.For a state s that cannot reach any state without outgoing transitions, we have that τ ω (s) = 0.For the remaining states, we can compute the probability of termination using standard techniques as described in, for example, [22,Section 11.2].
Since f was chosen arbitrarily, we can conclude that Given a probabilistic bisimulation R, we can quotient the probabilistic transition system S, π as follows.
Definition B.7.Let R be a probabilistic bisimulation.The probabilistic transition system S R , π R consists of Note that the function π R is well-defined since R is a probabilistic bisimulation.We will apply the above quotient construction for probabilistic bisimilarity (which can be computed in polynomial time [1]).
Example B.8. Consider the probabilistic transition system of Example 2.2.The smallest equivalence relation containing { s 3 , s 5 } is a probabilistic bisimulation.The resulting quotient can be depicted as By quotienting, the number of states that need to be considered and, hence, the number of variables in the formula may be reduced.However, we still have to check that the quotiented system gives rise to the same distances.Next we relate the behavioural pseudometric d 1 of the original system S, π with the behavioural pseudometric d R of the quotiented system S R , π R .Proposition B.9.For all s 1 , s Proof.First all, note that As a consequence, we have left to consider the case s 1 → and s 2 →.We prove that for all n ) by induction on n.We distinguish the following three cases.
• If n = 0 then the property is vacuously true.
(s 1 , s 2 ).In the proof of this case, we make use of the following two observations.For each f ∈ for all s ∈ S. Note that if states s and s ′ are probabilistic bisimilar then d 1 (s, s ′ ) = 0 and, hence, d n 1 (s, s ′ ) = 0 and, therefore, g(s) = g(s ′ ), since g is nonexpansive.• Furthermore, = d ω 1 (s 1 , s 2 ).To simplify the formula even further, we exploit the following three observations.• Since d is a pseudometric, d(s i , s i ) = 0 and d(s i , s j ) = d(s j , s i ).Therefore, in pseudo(d) ∧ post-fixed(d) we can replace all d ii 's with zero and all d ij 's where i > j with d ji 's.As a consequence, we only need to consider d ij 's with i < j.This reduces the number of variables in the formula considerably.
• Let C be the set of pairs of states for which the distances have already been computed.Then ∃d pseudo(d) ∧ post-fixed(d) ∧ d i 0 j 0 ≤ m is equivalent to ∃d pseudo(d) ∧ post-fixed(d) ∧ d i 0 j 0 ≤ m ∧ (i,j)∈C d ij = d 1 (s i , s j ) since d 1 is the greatest post-fixed point.As a consequence, we can replace all d ij 's where (i, j) ∈ C with their already computed distances d 1 (s i , s j ).Again, the number of variables may be reduced.
• If π i 0 j = 0, we can infer that µ ij = 0 for all 1 ≤ i ≤ N .As a consequence, we can replace the occurrences of all those µ ij 's with 0. Symmetrically, if π j 0 i = 0 we can simplify the formula similarly.Also this simplification may reduce the number of variables.
We have implemented these simplifications in the form of a Java program that takes as input the probability matrix π and that produces as output the simplified formula in a format that can be fed to Mathematica. 5xample B.10.Consider the probabilistic transition system of Example 2.2.The simplified formula for this system is given below.Line 3 correspond to pseudo(d), line 4-9 correspond to post-fixed 1 (d, 1, 2) and line 10-15 correspond to post-fixed 1 (d, 2, 1).The formula was reduced to true by Mathematica in 8.2 seconds on a 3GHz machine with 1GB RAM.When feeding Mathematica the formula that has not been simplified, it runs out of memory after some time.
We also attempted to solve this example with a solver called QEPCAD B [9] but the performance of Mathematica on this example was better.