How to read the characteristics of tracking systems

The commercial offer says: "accuracy 10 cm." This is not a characteristic of the system. This is a fragment of a phrase that should have a sequel.

The full phrase sounds something like this: the error does not exceed ten centimeters in half of the measurements with a line of sight of at least three anchors at a distance of up to fifteen meters, with a label fixed at a height of one and a half meters and not shielded by the body, with a travel speed of up to one and a half meters per second and a refresh rate of ten hertz. Remove any of these conditions and the number will change, sometimes tenfold.

The difference between the two formulations is not academic. The first one cannot be verified, which means it cannot be written into the contract. The second one is possible and necessary: acceptance is carried out according to it, and it also determines who is to blame if the system shows a guest in the pool on Saturday evening when he is at the bar.

This article is about how to read such numbers. It is more boring than the rest in the series and more useful than the rest: once you read it, you will stop comparing the incomparable and start asking suppliers questions to which a bona fide supplier has answers, but the rest do not.

What is an error and why one digit is not enough

The error of one measurement is the distance between the point that the system issued and the point where the object actually was. There are many measurements, their errors are different, and all meaningful information is not in one number, but in their distribution.

The distribution is described by several metrics, and they give very different answers on the same data set.

Average error. The most popular in presentations and the most useless. One outlier of thirty meters spoils a hundred good measurements, and vice versa: if outliers are rare, the average looks decent with completely unacceptable behavior of the system in the worst cases. The authors of the review of inexpensive positioning systems directly note that the accuracy is most often either not indicated at all, or indicated precisely as average, and that tail percentiles are important for practical applications.

Median error, or CEP50. Circular probable error: the radius of the circle into which half of the measurements fall. This is the figure that vendors call "accuracy". She is honest in exactly one sense: half the time the system works just as well. What happens in the second half — she does not say.

CEP95 is the 95th percentile. The radius that ninety-five dimensions out of a hundred fall into. This is a working characteristic for any task where not average sensations are important, but the worst reasonable case. The difference between CEP50 and CEP95 in one system is fourfold, and it shows how stable the system is.

RMSE. The standard error, sensitive to outliers, is stronger than the average. It is used in scientific works, it is almost not found in commercial offers.

Maximum error. It is needed in security tasks: if the system shows a person in another room once an hour, it is impossible to build an alarm on it.

A simple example of why this is not a quibble. A system with CEP50 = 0.3 m and CEP95 = 4 m is suitable for a heat map and is not suitable for controlling the capacity of the steam room: five percent of the time the guest will be "in the wrong room". A system with CEP50 = 1.2 m and CEP95 = 1.8 m is formally "four times worse" according to the advertising figure, but for a zonal task it is better — it is predictable.

Displacement and dispersion are different diseases

Two different defects that give the same error in the report and are treated completely differently.

Systematic displacement: all measurements are shifted to one side by forty centimeters. The reasons are an error in the anchor coordinates, an unaccounted height, an incorrect delay in the antenna path, a discrepancy between the plan and the actual geometry of the building. It is treated by calibration and there is one engineer's departure.

Scatter: The measurements are scattered around the true point. The reasons are physical — multipath, noise, geometry. Calibration is not treated: it is treated by adding anchors, changing their location, filtering, or changing technology.

When accepting, it should be separated: if the error is equally directed at all control points, the system does not need to be changed, it needs to be configured.

Horizontally, vertically, by floors

CEP is an error in the horizontal plane. Vertical (VEP) and spherical (SEP) are separate metrics, and horizontal is more important in a multi—storey object: an error of two meters vertically means that the guest was on a different floor, that is, in a different zone and in a different line of the report. If you have two levels — a terrace above the pool, a mezzanine with exercise equipment, a second floor with classrooms — the question "what is your vertical accuracy and how does the system determine the floor" should be asked separately. It often turns out that the floor is determined not by geometry, but by which anchors are heard, and the system guesses on the stairs.

What do the standards say on this topic

The topic is normalized, and this is convenient: you can refer not to an opinion, but to a document.

ISO/IEC 18305:2016 — Standard for testing and evaluation of localization and tracking systems. His logic is important to the customer:

• the system is tested as a black box, entirely, without disassembling the internal device — that is, the result is checked, and not the architecture that was drawn to you in the presentation;
• tests are carried out throughout the building, and not on a convenient site;
• the basic accuracy indicators are errors with a probability of 50% and 95% (CEP50 and CEP95) and error distribution functions;
• in addition to accuracy, coverage, availability and latency are normalized;
• For critical applications, additional characteristics are provided — resistance and susceptibility to interference;
• The standard describes fourteen typical test scenarios collected from building types, types of objects and traffic modes.

The standard has a substantial criticism: in the work presented at the IPIN conference, the authors point out the limitations of the methodology and suggest giving not two percentiles, but four — 50, 75, 90 and 95. NIST, for its part, is working to revise the standard. For practice, this means that it is impossible to refer to 18305 as a mandatory regulation, but as a reference methodology for the acceptance protocol, it is quite, and a competent supplier knows it.

The second useful document is the ISO/IEC 24730 family describing RTLS. In it, accuracy is divided into two classes, and this is very close to how tasks are divided in life: a controlled area — a warehouse, a campus, an airport — with an accuracy of up to three meters, and a limited equipped area with an accuracy of tens of centimeters. If your supplier promises tens of centimeters on the entire area of the complex, he promises the second class where the standard assumes the first, and this is a reason to ask for the calculation of anchor density.

What spoils the accuracy on a real object

Polygon numbers are not transferred to the object for reasons that should be known by name: then in a conversation with the supplier, you can see whether he took them into account or not.

Line of sight and its absence

Radio systems measure the signal travel time and convert it to a distance. If there is an obstacle between the label and the receiver, the signal travels slower and along a longer path, and the system considers the distance to be greater than the real one. This is not noise, but systematic overestimation — the error is always in one direction.

Due to the short pulse, UWB distinguishes between the first incoming beam and reflections, therefore it tolerates the lack of line of sight better than other radio technologies. But he also has a wooden door, a glass block or a human body adds tens of centimeters, and a concrete wall breaks the measurement completely.

For Bluetooth, the difference between line of sight and its absence is the most dramatic: in demonstrations of phase distance measurement (Channel Sounding), according to open data, 10-30 cm in line of sight and 0.5–1 m in a cluttered room is obtained.

Multipath, and why the wet zone is the worst case

The signal comes to the receiver not in one way, but in a fan of reflections. The thermal spa complex reflects everything: tiles, glass, mirrors, water, metal handrails, stainless surfaces. This is the same room in which the human voice is buzzing — acoustics and radio suffer for the same reason.

Practical conclusion: the accuracy figures obtained in the office or in the warehouse do not work in the wet zone. The pilot in your room is not a formality, but the only way to find out the real numbers.

The human body

The mark on the guest is a mark, half of the directions around which the body covers. Water absorbs 2.4 GHz, a person consists mainly of water, and the bracelet on the side of the hand farthest from the receiver loses the signal by an order of magnitude.

Two non-obvious effects follow from this. First: the accuracy depends on how the guest turned, that is, it changes for no reason on the part of the system. Second, the error correlates with the position of the body, and averaging over a set of guests does not remove it. For Bluetooth phase measurements, this is so significant that there have been works on error correction based on inertial sensor data — that is, the system is trying to understand how the label is rotated to correct the distance. A silicone bracelet pressed against a wet body in the pool is the worst possible mounting position for a radio tag, and you need to know this before buying, not after.

Placement geometry

The accuracy depends not only on the number of receivers, but also on how they are positioned relative to the object. If the anchors are lined up along one wall, the system determines the position along the line well and poorly across: the error on one axis may be several times greater than on the other. The same thing happens in the narrow corridor and along the long bowl of the pool.

In satellite navigation, there is a concept of a geometric factor for this; in indoor positioning projects, it is usually not considered, but in vain. A sign that the supplier is thinking about this: he asks for a plan with heights, asks about the material of the partitions and offers an arrangement, and not "one sensor per two hundred meters in area."

People in the frame and on the air

The workload of the facility affects all technologies, but in different ways:

• for radio, a crowd is a set of water bags that shield the signal, that is, a direct degradation of accuracy during rush hour;
• for videos, the main enemy is overlap: people go in for each other, the track breaks, the ID jumps (in video analytics, this is measured by a separate metric, see the article about cameras and AI);
• for counters at the entrance, there is a dense flow shoulder to shoulder, when two people are read as one.

Hence the simple requirement for the pilot: a stop at rush hour is mandatory. A system tested in an empty hall has been tested in conditions that do not interest you.

Temperature, humidity, condensation

There is 90 °C and 100% humidity in the steam room, splashes and chlorine outside the bowl, steam in the locker room. This affects electronics, optics (fogging and the appearance of spots on the lens) and durability. IP execution is the first line to look at in the specification of any sensor for a wet zone, and the service life in such conditions is the second.

Speed and filters

The system does not provide measurements directly: she smoothes them with a filter, usually of the Kalman family. The filter removes the jitter and introduces a lag at the cost of this: with a sharp change of direction, the track "catches up" with the person. Therefore, a smooth, beautiful trajectory in a demonstration video is not necessarily a sign of accuracy; sometimes it is a sign of strong smoothing, which will cut corners and lag on a running child.

This is not important for analytics tasks. For a scenario effect that is triggered when a guest enters the zone, it is critically important.

Refresh rate and delay are different things

Two characteristics that are usually merged into one in sentences.

The refresh rate is how many times a new coordinate appears per second. Delay — how long it takes from the moment when the object was at the point to the moment when your system found out about it. The second consists of three parts:

1. radio channel: when the tag has the next slot for transmission (for systems with a schedule, these are tens and hundreds of milliseconds);
2. Processing: coordinate calculation and filtering;
3. transport and integration: server, queue, database entry, external API call, rendering on the operator's screen.

The third part in projects is usually the largest and the only one about which the positioning provider does not respond. If the alarm reaches the rescuer's smartwatch after eight seconds, of which positioning takes two hundred milliseconds, the problem is not in the sensors.

Task guidelines:

Task	Frequency	Acceptable delay
Heat map, reports	Every 10-30 seconds	Minutes and hours
Zone loading, presence	0.5–1 Hz	Seconds
Capacity monitoring, zone alarm	1-2 Hz	1-3 s
Personnel safety, rapprochement with technology	2-10 Hz	Less than a second
Scenario effect in the show	10 Hz and above	Tens of milliseconds

The correct question to the supplier is: what is the 95th percentile delay from physical movement to the event in our system, and how was it measured. The answer "updating ten times per second" does not answer this question.

Capacity: What will happen on Saturday night

A characteristic that is almost never checked on the pilot and which most often breaks the project during commissioning.

The radio channel is a shared resource. The more tags there are in the range of one receiver, the less ether each gets. For systems with a transmission schedule, the refresh rate drops proportionally to the number of labels: one hundred marks of ten hertz and a thousand marks of one hertz are the same channel. RFID has anti-collision procedures, and in a dense stream on the input group, some of the tags are read on the second attempt or not at all. In Bluetooth, with a large number of transmitters, the proportion of collisions on the air is growing.

What follows from this for a water park for two thousand guests or a thermal spa complex for eight hundred: the numbers from the pilot on ten labels have no relation to reality. You need to ask like this:

• how many labels are served by one cell at the same time at the stated refresh rate;
• what happens to the refresh rate when the number of labels triples;
• how does the entrance group behave when fifty people pass through the turnstile in a minute;
• whether you have an object with a comparable number of labels and whether it is possible to contact it.

Battery, service life and who changes

A triangle from which you can not jump out: battery, refresh rate, service life. Any two parameters are set freely, the third is obtained.

Orders according to open data: LoRaWAN trackers with rare reports claim from fifteen months to several years (for example, Tektelic indicates fifteen months for an industrial wearable device, Milesight for its outdoor tracker claims more than fifteen years with rare reporting). The active UWB tag, running several times per second, lives for weeks and months. The passive RFID tag has no battery at all and lives as long as the bracelet lives — but it does not give a continuous track, only marks on the reading points.

A separate question that decides the fate of the project for the third year: who maintains the park of tags. One and a half thousand bracelets with batteries are replacement regulations, a warehouse, a person with a duty and failure statistics. In projects with active guest tags, this usually becomes clear after the first season, when a third of the tags stop airing and the data quietly degrades.

For objects with guest bracelets, a practical conclusion follows from this: try to make what the guest wears passive or as simple as possible, and hang everything complex and powered on the ceiling, where there is PoE.

Identity as a characteristic, not as a side effect

The system has not only spatial resolution, but also personal resolution, and it also needs to be described explicitly:

• impersonal account — the system knows the number and does not distinguish between people;
• anonymous ID per session — distinguishes objects within the session. After exiting, the connection is lost (this is the best mode for route analytics);
• stable ID — bracelet, card, account in the application: the guest is recognizable between visits;
• binding to a person — a bundle with a full name, contract, subscription;
• Biometrics is a separate and most stringent legal regime.

A design error that can be seen right away: take a stable identifier where there is enough anonymity for a visit. This adds nothing to the analytics and puts the project in a different legal regime. For more information, see the article about the law.

And the downside, which suppliers do not talk about: impersonality can be unstable. Randomized MAC addresses of smartphones are restored by clustering algorithms with a high degree of success in laboratory conditions, and radio tags have their own transmitter features by which they can be distinguished from each other regardless of the identifier. If you declare anonymity to guests, you need to make sure that the system does not really restore connectivity, and not only does not record it in the database field.

Coverage, availability and drift

Three characteristics that do not fall into the sentences at all.

Coverage — the proportion of the area where the claimed accuracy is achieved. Not "we have covered the complex", but "in zones A, B, C, CEP95 reaches ≤ 1.5 m, in zone D — ≤ 4 m, there is no coverage in technical rooms." This formulation allows you not to argue later.

Availability — the fraction of time when the system is running. This includes server crashes, network outages, sensor failures, and, most importantly, the question: how do you know that one of the forty sensors has died. The correct answer is that the system itself reports a silent sensor; the wrong one is "we'll see from the data", because according to the data it can be seen in a week, and the week of statistics has already been spoiled.

Drift. Accuracy degrades over time: someone hung an advertising structure, rearranged cabinets, added a metal rack, moved the anchor when painting the ceiling. Systems based on a radio card (fingerprint) degrade the fastest: the radio card becomes obsolete from rearranging furniture. Hence the requirement: repeated control measurement three to six months after acceptance, fixed in the service contract.

Acceptance measurement protocol: how to put it in the contract

Next is what should be included in the terms of reference and in the act. This section can be used as a template.

1. The standard. A grid of control points applied to the object plan and physically marked on the floor (masking tape, stickers). The coordinates of the points are measured by a laser rangefinder from the structural elements. For dynamic routes — a marked route with markers and a video recording of the movement with visible clocks, which are then compared to timecodes. In scientific works, the motion capture system serves as a reference, but accurate marking is sufficient for the object, provided that its error is at least three times less than the one being checked.

2. Control points. At least twenty per zone, with the obligatory inclusion of: the center of the zone, points at the walls and in the corners, the passage between the zones, points behind the obstacle, points under the metal structure, points on the border of the coating. At least two points for each problematic feature of the object.

3. Statics and dynamics. At each point — at least thirty consecutive measurements in statics. Tracks — at least three passes in each direction, step and fast step.

4. Measurement conditions. It is mandatory to record: an empty object and rush hour with an indication of the number of people; a mark on the body in the working position, not in the hand; for wet areas — with equipment running (steam, fountains, wave).

5. What is recorded in the protocol:

Indicator	How is it considered	Example of acceptance criteria
CEP50, CEP95 for each zone	By a set of measurements in statics	CEP95 ≤ 1.5 m in zones A–C
Incorrect zone percentage	Percentage of measurements assigned to the wrong zone	No more than 2 %
Delay	The 95th percentile from moving to an event in our system	No more than 2 seconds
Share of passes	Percentage of intervals without label data	No more than 1 % per hour
False objects	Number of phantom targets per hour	Not more than 1
Rush hour behavior	The same metrics for the actual download	Degradation of CEP95 is not more than 50 %
Capacity	Refresh rate for the full number of labels	Not lower than 1 Hz at 800 marks

6. Procedure. Who carries out the measurement, who is present, in what form the raw data is uploaded (necessarily raw, not the supplier's report), how long it takes to sort out disagreements, what is considered an undelivered acceptance, and how many attempts are given.

7. Repeated measurement. In three to six months, according to the same protocol, within the framework of the service agreement. Permissible degradation is specified.

One paragraph in the contract with reference to such a protocol is cheaper than any dispute about what was meant by "10 cm accuracy".

Checklist of questions to the supplier

1. What is the precision in CEP50 and CEP95 separately, and not in one digit?
2. Under what conditions were these figures obtained: line of sight, distance to the receivers, position of the label, room occupancy?
3. What is the vertical accuracy and how is the floor determined?
4. How many receivers will be required for our plan and why exactly so many? Show the placement.
5. How will the accuracy change if you remove one receiver from the calculation?
6. What is the 95th percentile delay from moving to the event in our system?
7. How many labels does one cell serve and what happens to the refresh rate when the object is fully loaded?
8. How does the system behave when the label is closed by the body and wet?
9. What is the IP version of the equipment for wet areas and what is the service life in them?
10. How does the system report a non-working sensor?
11. What happens when the network is broken between the receivers and the server: is the data lost or buffered?
12. Where is the data stored, in what form do we receive it, is there any unloading of raw events?
13. What identifier is used and is it possible to work in impersonal mode?
14. Is there an object of the same profile and scale where you can come to see?
15. Are you ready to conduct an acceptance measurement according to our protocol and record its results in the report?

The fifteenth question is a test. The supplier, confident in his decision, agrees without discussion.

Sources

Sources

• ISO/IEC 18305:2016 — Testing and evaluation of localization and tracking systems
• Criticism of the ISO/IEC 18305 methodology, IPIN Conference
• NIST, Location-Based Services — revision of the standard
• ISO/IEC 24730-1:2014 — RTLS, accuracy classes
• Amsters et al., Evaluation of low—cost positioning systems - about metrics and reference
• Correction of Bluetooth phase measurement error by inertial sensors
• Deanonymization of UWB tags by radio frequency fingerprint
• Restoring the connectivity of randomized MAC addresses using machine learning methods
• Tektelic, Milesight — the declared service life of wearable and outdoor trackers (manufacturer data)