crash probe: retry processing when missing symbols

Missing symbols in backtraces might indicate missing debuginfo from the
crashed program, but another possibility is that on-the-fly debuginfo
has not finished downloading. The Clear Linux OS uses an on-the-fly
debuginfo setup, so this condition is likely for the first instance of a
program crashing on the system.

To decrease likelihood of missing symbols appearing, scan the backtrace
for a frame with incomplete data after first processing, and if found,
reprocess the core file after a 10 second pause.

Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>

src/probes/crash_probe.c

@@ -568,6 +568,23 @@ int main(int argc, char **argv)
                 goto fail;
         }
 
+        /* On Clear Linux OS, missing symbols may appear if automatic debuginfo
+         * downloads are still in flight. So if any missing symbols appear on
+         * the first run (indicated by the presence of "??? - ["), wait 10
+         * seconds and try again.
+         */
+        if (strstr(backtrace->str, "??? - [")) {
+                sleep(10);
+
+                if (prepare_corefile(&e_core, core_fd) < 0) {
+                        goto fail;
+                }
+
+                if (process_corefile(&backtrace) < 0) {
+                        goto fail;
+                }
+        }
+
         g_string_prepend(backtrace, header->str);
 
         if (!send_data(&backtrace, default_severity, clr_class)) {