kunai: failed to retrieve bpf_prog instructions for event=79e45499-680a-7737-b18e-2fb65e8fcca4: Operation not permitted (os error 1) #194

juju4 · 2025-04-06T23:19:14Z

juju4
Apr 6, 2025

how to fix those or related to kernel options?

qjerome · 2025-04-07T06:29:35Z

qjerome
Apr 7, 2025
Maintainer

Wow ! First time I see this error message !
It would worth identifying a way to reproduce this error and open an issue with it so that I can tell you more.

4 replies

juju4 Apr 13, 2025
Author

While I get many matching logs SELECT regexp_replace(body_message, '.* failed to retrieve bpf_prog instructions for event=(.*): .*', '\\1', 'g') as event_uuid FROM "kunai" where body_message like '%failed to retrieve bpf_prog instructions for event%
I get error/no matching events SELECT * from "kunai" WHERE info_event_uuid in (SELECT regexp_replace(body_message, '.* failed to retrieve bpf_prog instructions for event=(.*): .*', '\\1', 'g') as event_uuid FROM "kunai" where body_message like '%failed to retrieve bpf_prog instructions for event%')
(json is flattened by my log system/openobserve)
but got it in journald

$ sudo journalctl -u 00-kunai -l --no-pager -g 44b165cb-7ba0-4a7e-6661-8862272c974a
Apr 08 15:09:46 myhost.internal kunai[9280]: [2025-04-08T15:09:46Z ERROR kunai] failed to retrieve bpf_prog instructions for event=44b165cb-7ba0-4a7e-6661-8862272c974a: Operation not permitted (os error 1)
Apr 08 15:09:46 myhost.internal kunai[9280]: {"data":{"ancestors":"","command_line":"/lib/systemd/systemd --system --deserialize 124","exe":{"path":"/usr/lib/systemd/systemd"},"id":117121,"prog_type":{"id":8,"name":"cgroup_skb"},"tag":"df070673fd0e5827","attached_func":"","name":"","ksym":"bpf_prog_df070673fd0e5827","bpf_prog":{"md5":"?","sha1":"?","sha256":"?","sha512":"?","size":0},"verified_insns":null,"loaded":true},"info":{"host":{"uuid":"e4984793-693b-5414-a5a0-e21817caa551","name":"myhost.internal","container":{"name":"myhost.internal","type":null}},"event":{"source":"kunai","id":21,"name":"bpf_prog_load","uuid":"44b165cb-7ba0-4a7e-6661-8862272c974a","batch":5949448},"task":{"name":"systemd","pid":1,"tgid":1,"guuid":"92700008-0000-0000-de87-435701000000","uid":0,"user":"?","gid":0,"group":"?","namespaces":{"mnt":4026531841},"flags":"0x400100","zombie":false},"parent_task":{"name":"swapper/0","pid":0,"tgid":0,"guuid":"00000000-0000-0000-de87-435700000000","uid":0,"user":"?","gid":0,"group":"?","namespaces":{"mnt":4026531841},"flags":"0x4200002","zombie":false},"utc_time":"2025-04-08T15:09:46.790968722Z"}}
$ sudo journalctl -u 00-kunai -g "failed to retrieve bpf_prog instructions" -l --no-pager | tail -2
Apr 13 20:13:52 myhost.internal kunai[40227]: [2025-04-13T20:13:52Z ERROR kunai] failed to retrieve bpf_prog instructions for event=d77a5698-ddcd-d23e-37f2-2733f8a699f3: Operation not permitted (os error 1)
Apr 13 20:13:52 myhost.internal kunai[40227]: [2025-04-13T20:13:52Z ERROR kunai] failed to retrieve bpf_prog instructions for event=a3ffabe4-ee38-f8d9-80c0-a756777c2454: Operation not permitted (os error 1)
$ sudo journalctl -u 00-kunai -g d77a5698-ddcd-d23e-37f2-2733f8a699f3 -g a3ffabe4-ee38-f8d9-80c0-a756777c2454 -l --no-pager
Apr 13 20:13:52 myhost.internal kunai[40227]: {"data":{"ancestors":"","command_line":"/lib/systemd/systemd --system --deserialize 124","exe":{"path":"/usr/lib/systemd/systemd"},"id":174869,"prog_type":{"id":8,"name":"cgroup_skb"},"tag":"485cc2f9dbf141cd","attached_func":"","name":"","ksym":"bpf_prog_485cc2f9dbf141cd","bpf_prog":{"md5":"?","sha1":"?","sha256":"?","sha512":"?","size":0},"verified_insns":null,"loaded":true},"info":{"host":{"uuid":"e4984793-693b-5414-a5a0-e21817caa551","name":"myhost.internal","container":{"name":"myhost.internal","type":null}},"event":{"source":"kunai","id":21,"name":"bpf_prog_load","uuid":"a3ffabe4-ee38-f8d9-80c0-a756777c2454","batch":16044791},"task":{"name":"systemd","pid":1,"tgid":1,"guuid":"92700008-0000-0000-b731-b35901000000","uid":0,"user":"?","gid":0,"group":"?","namespaces":{"mnt":4026531841},"flags":"0x400100","zombie":false},"parent_task":{"name":"swapper/0","pid":0,"tgid":0,"guuid":"00000000-0000-0000-b731-b35900000000","uid":0,"user":"?","gid":0,"group":"?","namespaces":{"mnt":4026531841},"flags":"0x4200002","zombie":false},"utc_time":"2025-04-13T20:13:52.468761930Z"}}

host is Ubuntu 22.04 LTS (systemd 249.11-0ubuntu3.15)
I don't get those on the Debian12 host (systemd 252.36-1~deb12u1)
Related to https://www.baeldung.com/linux/systemd-switched-root-deserialize but how it changes between distributions and version is undocumented it seems

qjerome Jun 1, 2025
Maintainer

Wow, this is weird indeed ... The only thing that could explain (purely based on assumptions) would be that some eBPF programs gets loaded before the root switch but we try to access it after the root switch ! I believe that if this error stays marginal (i.e. only happening at that very specific moment where we are in-between two FS) it can be considered as acceptable. What do you think ?

juju4 Jun 8, 2025
Author

qjerome Jun 10, 2025
Maintainer

Even though you see this error, you must see a bpf_prog_load event related to it. If you search back in your logs for the event uuid displayed in the error message you should find one. The only thing is that this event will be not contain all the information, in this case the hashing information of the bpf program will be missing, all the other fields will be correct. The error is just there to attract the attention and also give a details about the failure.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kunai: failed to retrieve bpf_prog instructions for event=79e45499-680a-7737-b18e-2fb65e8fcca4: Operation not permitted (os error 1) #194

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

kunai: failed to retrieve bpf_prog instructions for event=79e45499-680a-7737-b18e-2fb65e8fcca4: Operation not permitted (os error 1) #194

Uh oh!

juju4 Apr 6, 2025

Replies: 1 comment · 4 replies

Uh oh!

qjerome Apr 7, 2025 Maintainer

Uh oh!

juju4 Apr 13, 2025 Author

Uh oh!

qjerome Jun 1, 2025 Maintainer

Uh oh!

juju4 Jun 8, 2025 Author

Uh oh!

qjerome Jun 10, 2025 Maintainer

juju4
Apr 6, 2025

Replies: 1 comment 4 replies

qjerome
Apr 7, 2025
Maintainer

juju4 Apr 13, 2025
Author

qjerome Jun 1, 2025
Maintainer

juju4 Jun 8, 2025
Author

qjerome Jun 10, 2025
Maintainer