Observations

InSkill can extract data from a picture taken during a step. Instead of asking the Technician to enter information, they can photograph a nameplate or operator screen and have InSkill read the information from that picture automatically. You do this by creating a photo observation processor.

📘

Note

This page covers photo observation processors. For observation steps and the feedback variable types in general, see Choosing the Step Type.

How a Processor Reads a Picture

The variables on a processor tell InSkill where to save the data and how to find it. The example below processes a machine nameplate and puts data into the variables model, serial_number, type, and machine_wt. The processor finds each value by looking for target text such as MODEL and reading the text next to it — here, the text below it.

Each variable has these settings:

The Data table beside the template image shows the value read for each variable, so you can check the processor as you build it.

Targets and matching

The target is the text to look for, and it must match the picture exactly — MODEL matches where Model will not.

• Use the ^ character to match whether or not there is a space. In a nameplate reading V~, it is hard to tell whether a space is present, so V^~ matches either way.
• Use the wildcard * to match any single character. A target of MX* matches MX followed by any one character (MX1, MX8, MXp). This differs from ^, which matches zero or one space.

Filter Regex

Many nameplates and operator screens pair a title with its data, so targets and directions capture the information easily. Sometimes, though, the text next to a target holds more than you want, or there is no title to find first. In these cases you can use a Regex — a text-searching standard used in programming that is powerful but also difficult.

For example, on a pump nameplate the target KW might find 0.37 HP 0.5 ln, because that is the text to its right. A filter Regex takes just the 0.37 part. That Regex is /[.0-9]+ /, which must begin and end with a forward slash. Inside the [] brackets, it matches a period and the numbers 0–9; on its own that would match a single character, so the + means as many matching characters as there are. The result captures the numbers including the period, but not the space or the HP that follows. Use a filter Regex to remove any unwanted text.

Useful Regex resources:

• Tutorials at regexlearn.com
• Interactive builders at regex101.com and regexr.com

Creating a Photo Observation Processor

Follow these steps to create a photo observation processor:

1. In your copilot's Task Editor, select the Observations icon on the left edge.
2. Open the -- Add New Observation Processor -- list and select Photo.
3. Give the new processor a Name that describes what it does.
4. Upload the photo template you want to read — for example, a clear picture of the nameplate.
5. Click + Add Variable for each value you want to extract.
6. Enter each variable's name, target, direction, type, and filter Regex.
7. Check the Data table to verify the results.
8. Click Save.

To remove a variable, click its Remove button. To delete the processor, click the red trash can at the top right.

Using a Processor in a Step

In an observation step, choose Take a Photo to prompt the Technician to take a picture with the camera or upload one. On that observation, choose the photo processor you want to run when the picture is uploaded from the Observation Process list, and click Save.