PEMP: pathogen environment monitoring in mixed reality

PEMP: pathogen environment monitoring in mixed reality

Pathogen environment monitoring is one of the least glamorous and most consequential routines in food manufacturing. Swabs are taken at defined points across a plant on a defined schedule, sent for testing, and the results are used to find out where organisms are surviving and travelling.

Project at a glance

  • Client: PepsiCo
  • Sector: Food manufacturing, hygiene and food safety
  • Platform: Microsoft HoloLens generation 1, with a companion Windows desktop application and a synchronised server
  • Services: Mixed reality application development, desktop companion application, plant model preparation from Revit, server synchronisation

The brief

Pathogen environment monitoring is one of the least glamorous and most consequential routines in food manufacturing. Swabs are taken at defined points across a plant on a defined schedule, sent for testing, and the results are used to find out where organisms are surviving and travelling.

PepsiCo asked us to build a mixed reality version of that routine. The result, PEMP, is a HoloLens application for the person taking the swabs, paired with a desktop application for the people interpreting the results.

 

The challenge

The weakness in conventional swab monitoring is not the laboratory. It is the geography.

A swab result is only as useful as the location it is tied to, and locations in a food plant are recorded as text: a point number, a written description, a zone reference. Two operatives reading the same description will not always swab the same square of floor, and a point number cannot easily be compared with the drain three metres away.

That makes the most important question hard to answer. Pathogen investigation is fundamentally spatial. You are looking for a source and a route, and a route is a line drawn across a building. A spreadsheet of point numbers does not draw it.

There is also a discipline problem particular to hygiene work. Hygienic zoning divides a plant into areas with different rules about what may cross between them, and the boundaries are not always obvious mid-task. Getting zoning wrong during swabbing risks the very contamination the programme exists to detect.

Finally, the swabber’s hands are occupied and frequently gloved. Whatever the tool is, it cannot be a tablet.

What we built

Two connected applications, plus a synchronised server between them.

On the headset, the session opens with a fit box to seat the device correctly and a safety warning, then syncs data from the server. A swab tutorial takes the user through correct placement step by step, then guided navigation directs them to the swab area they are due to work on.

At the point of sampling, the user places a swab marker in world space. The application generates the bag number automatically, and the user captures a photograph and records a voice note from the headset without putting anything down. Activity is logged throughout.

The visualisation layer is what makes it a spatial tool rather than a hands-free form. Distance rings are drawn by pathogen type, showing the radius that matters for the organism in question. A heat map is spawned in the plant and can be toggled by pathogen, so the operative can see, standing in the room, where results have been concentrating. Hygienic zone visualisation is triggered automatically on entering a zone, so the boundary announces itself rather than waiting to be remembered. World-anchored tags are loaded back from the server, meaning previous swab points remain exactly where they were placed on earlier rounds.

The desktop application carries a 3D model of the plant, with toggles for hygienic zoning and for the flow of people, drains, forklifts and bins. Tags can be inspected with their photographs and voice notes attached, and results assigned against them. Those four flow layers are the investigative heart of it. Pathogens travel on feet, water, wheels and waste, and switching each route on over a heat map is how a hypothesis about a source gets tested.

How it works

The application runs on HoloLens generation 1 using air-tap interaction, the pinch gesture that was that device’s primary input.

Positions are held using world anchors, which is the mechanism that lets a swab marker placed on one round be found in the same physical spot on the next. The value of the record is entirely in that repeatability.

The plant model was sourced from Revit and optimised in 3ds Max before use, so a construction model would run at frame rate on a headset and drive the desktop view from the same geometry.

Server synchronisation ties the two applications together. What the swabber places, photographs and dictates appears on the desktop for assignment of results, and what the desktop holds loads back onto the headset next round.

The outcome

A working proof of concept covering the full monitoring loop: guided navigation to the swab point, hands-free capture of photograph and voice note at the point of sampling, automatic bag numbering, world-anchored markers that persist between rounds, distance rings and a togglable heat map by pathogen in the plant itself, automatic hygienic zone warnings on entry, and a desktop model with people, drain, forklift and bin flows for investigation.

 

Back to Case Studies