Last week, a support ticket came in that the newest version of the application wasn't working on MacOS. That was strange because I had spent hours personally testing it on a MacOS laptop to make sure that the new build process was flawless.
The customer's symptom: our application wasn't running.
The actual symptom: whenever you ran our application, it was instantly killed and this was the only output:
Killed: 9
Let the troubleshooting begin
The "Killed: 9" thing was suspicious because that's the kind of error that we would get when we didn't sign our MacOS binaries correctly (which is why I was worried that it was related to switching our build process). However, I confirmed that we were signing them correctly, and the local MacOS tools agreed:
codesign -vvv /Applications/my-app.app/Contents/MacOS/my-app
/Applications/my-app.app/Contents/MacOS/my-app: valid on disk
/Applications/my-app.app/Contents/MacOS/my-app: satisfies its Designated Requirement
The only difference between the last version that worked and the one that didn't was a single dependency upgrade in our go.mod file: github.com/projectdiscovery/nuclei/v3. Nuclei is used for pentesting, which our application can do as part of its suite of tools.
Our application doesn't even use the nuclei code unless it's specifically requested, and we were getting "Killed: 9" even when we ran it with "--help".
This seemed nuts; how could upgrading a package that we use for a specific scheduled action cause MacOS to instant-kill the application with "Killed: 9"?
What I knew so far:
- The application wasn't doing anything interesting (it never printed a single one of our logs).
- MacOS was killing it on purpose.
What's the kernel think?
I asked the kernel for its logs while I ran our application with "--help":
sudo log stream --predicate 'sender = "kernel"'
When it ran, the only interesting thing appeared to be this line:
kernel: CODE SIGNING: process 27627[my-app]: rejecting invalid page at address 0x10232000 from offset 0x0 in file "<nil>" (cs_mtime:0.0 == mtime:0.0) (signed:0 validated:0 tainted:0 nx:0 wpmapped:1 dirty:0 depth:0)
So MacOS was doing something related to the signing of our binaries, but I didn't know what. Remember, asking it about the binary itself resulted in no errors, but I was seeing one now when it ran.
The error did reference a file ("<nil>"), so maybe if I built it with debug symbols it would help.
I also stumbed onto some diagnostic information in /Library/Logs/DiagnosticReports with a series of files called "my-app-XXXX-XX-XX-XXXXXX.ips", where the X's represented a date-time. Those were JSON files with some information, and one of them appeared to be a stack trace:
[...]
"faultingThread": 0,
"threads": [
{
"triggered": true,
"id": 734760,
"threadState": {
[...]
"trap": {
"value": 14,
"description": "(invalid protections for user instruction read)"
},
Other than that error ("invalid protections for user interaction read") in thread 0, there wasn't much useful information.
I rebuilt the application with debug symbols this time and tried again.
Same deal, but this time, in the stack trace in the ".ips" file, register 8 had the symbol:
"r8": {
"value": 133683616,
"symbolLocation": 0,
"symbol": "*/jitdec.Decode"
},
Aha! A function! It was calling "jitdec.Decode".
Source code archaeology
"jitdec.Decode" wasn't anything that I had heard of before (it certainly wasn't one of our functions), so I googled.
"jitdec.Decode" is part of the "bytedance/sonic" package, which is a high-performance JSON package for Go. Nuclei uses sonic instead of the standard JSON package for some reason; I guess someone complained at some point that its JSON operations were too slow or something.
I now knew where it was breaking, but I didn't know (1) why, or (2) why that function was even being called.
Since it was breaking during "--help", I suspected that it had to be an "init" function somewhere. In Go, an imported package's "init" function is called before anything else, so nuclei had to be doing something stupid up front. After a bunch of digging through source code, it turned out that nuclei did have an "init" function where it loaded the local configuration from disk. The format of that local configuration? JSON.
Okay, so I knew that on startup, the application would try to load the nuclei config files, and that doing that called the "jitdec.Decode" function, which caused MacOS to kill the application.
Why kill it now all of the sudden? Nuclei had been using the sonic package for ages.
I diffed the nuclei versions involved (we upgraded from v3.4.5 to v3.4.7); nothing looked all that interesting in terms of actual code that changed, but they did upgrade sonic from v1.12.8 to v1.13.3.
I diffed the sonic versions involved (they upgraded from v1.12.8 to v1.13.3) and found some changes in the "jitdec" package. In particular, there was a "//go:build" comment that specifically excluded Go 1.24 (which we use), and after the upgrade, it excluded Go 1.25. So whatever was in those files, it used to be skipped for our Go version, and now it wasn't.
What was this whole "jitdec" thing, anyway? Apparently it does some just-in-time (JIT) stuff to somehow decode JSON faster? Again, this seems like overkill for a project (nuclei) that is not, to the best of my knowledge, bottlenecked by slow JSON encoding and decoding.
But that's the difference: previously, our application didn't include any JIT, and now it did. And because it read the nuclei config files before anything else, the application was compiling code on the fly to decode all 200 bytes as fast as possible to crush the benchmarks. MacOS noticed that the application was running code that wasn't signed, and it killed the application.
Now what?
Entitlements
My first instinct was to see if I could just switch the package to the standard JSON package, but I couldn't figure out how to do that. Some packages let you do a specific import up front to turn on or off their weird, high-performance overrides, but not nuclei. So we were stuck with JIT.
MacOS has a series of "entitlements" for an application: at signing time, you also bundle in a list of special things that the application is allowed to do. One of them is relates to JIT specifically, and one relates to unsigned executable memory.
rcodesign has a "--entitlements-xml-file" option to specify an entitlements file. I plugged that in, rebuilt, reinstalled, and tested everything, and the application ran normally.
In this particular case, I had to grant these two entitlements to my application to make MacOS happy about whatever sonic was doing:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.cs.allow-jit</key>
<true/>
<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
<true/>
</dict>
</plist>
Summary
MacOS sucks. It's horrible to work with and the documentation is bad.
However, these things might help you in the future:
- You need to sign your MacOS applications.
- The MacOS kernel will kill your application if it doesn't like something about it ("Killed: 9").
- Use the "codesign" tool to see what MacOS thinks about your installed application.
- Use the "log stream" command to see what the kernel is doing when it kills your application.
- Turn on debug symbols during your build process.
- Hunt down the ".ips" file for your killed application and see what the stack trace looks like.
- There are a whole bunch of entitlements that your application may need to have; if it's a runtime error, there's a chance that it needs one that it doesn't have.