Experimenting with Jev for Offensive Security

2026年9月24日9月24日

文章目录

A few days ago I published a quick demo of using the new Jev model to measure OPSEC strength of a command being typed into Mythic:

嵌入推文（联网查看）

Since posting the video I have had a few questions about just how this works, and a lot more from people who didn't know that this model existed, so I wanted to show the code and explain the approach so others can explore this for themselves.

A Quick Primer on Jev

First, what is Jev? Well Jev is a newly released model from TypeSafe AI, which shares some similarities with LLMs. Trained with "Reinforcement Learning for Calibrated Decisions (RLCD)", Jev is designed to return typed, structured decisions which is what sets this apart from LLMs.

Unfortunately, TypeSafe AI have not released any papers into the Jev architecture. Although looking at the number of open source projects which are quickly popping up (plus this post from a user claiming to have built a similar Jev model a year ago), it won't be long until we have local versions to play around with.

And because of this, it seems obvious that we're about to see a wave of these small classification models drop, so its time to put on our hacker hats (although I'd love to see Jev models hack their way out of a sandbox).

Jev Hijacking its Sandbox!

Jev takes in a "state" value (similar to initial context in LLMs) and a list of questions framed within one of the following types:

Noul - Yes/No decision
Choice - Choose one of several presented options
Score - Assign a rating against a provided scale

One of the reasons Jev got people so excited is because it is fast... VERY FAST. Like, sub 500ms for responding to 200 questions simultaneously fast.

Oh, and Jev is extremely cheap too, costing $0.042 per million tokens and with free responses.

Let's dig into the model a bit more and see what it has to offer. Let's say we have a short story:

As the Brit walked into the room, he quickly shouted to the American, "Tea should only be made using a kettle, and never in the microwave!". Of course the Brit was correct.

This story would first go into the model's state as our context for Jev to work with.

Then we might want to propose a question, "What shouldn't be made in a microwave?". If we were using the Choice option, we would need to give a few options for Jev to select from:

Tea
Eggs
A Kettle

Next, say we also wanted another question answered: "Did the Brit state that Tea should be made in the microwave?". As this is a boolean question, we would use the Noul type:

True - The Brit did state this in the story
False - The Brit did not state this in the story

And finally we may also want our short story graded, so we could propose a question of "On a scale of 0-4, how accurate was the short story to real life?" using the Score type:

0 - The story was completely inaccurate and has no resemblance to real life
1 - The story was somewhat accurate to real life
2 - The story was moderately accurate to real life
3 - The story was very accurate to real life
4 - The story was extremely accurate to real life

Now the cool thing is, we can ask all of these questions at once, and receive a single response containing all of the answers along with how confident Jev is with its responses:

Turns out my story isn't going to win any prizes :(

Using Jev for OPSEC

While considering what this model could be useful for, the idea that popped into my mind was OPSEC feedback (I don't get out much). And with a 70-500 ms response time from Jev, I wanted to see if I could get realtime feedback as I typed commands into Mythic.

To play around with the model, you just uv add the TypeSafe SDK, which allows us to use the client with an API key:

from typesafe_sdk import TypeSafeClient

client = TypeSafeClient(api_key="api-key-here")

Then it is just a case of framing any OPSEC questions that we may want answering in a similar format to our above short story:

    candidates = {}
    candidates["grade_command"] = Score(
        instructions="""
            What is the likelihood that the user selecting the Mythic `command` will be detected by EDR `edr_type` on the target system running `os` based on the information of the command and the target system configuration?
        """,
        criteria=[
          "none; The command being used is not a valid Mythic command, or has an internal execution_method which indicates that the command would not be executed by the implant directly.",
          "low; The command being used is generally considered to be benign or safe.",
          "medium; The command being used has a chance of being detected by an extremely verbose EDR configuration.",
          "high; The command being used has a chance of being detected by a well deployed EDR configuration.",
          "critical; The command being used has an extremely high chance of being detected by even a passive EDR configuration."
        ]
    )
    candidates["grade_arguments"] = Score(
        instructions ="""
            What is the likelihood that the users typed `arguments`, when executed by the Mythic `command`, will be detected by EDR `edr_type` on the target system running `os` when executed by the implant?
        """,
        criteria=[
            "none; No arguments were provided",
            "low; The arguments, when executed by the Mythic `command`, have a low chance of being detected by the identified EDR as malicious.",
            "medium; The arguments, when executed by the Mythic `command`, have a medium chance of being detected by the identified EDR as malicious.",
            "high; The arguments, when executed by the Mythic `command`, have a high chance of being detected by the identified EDR as malicious.",
            "critical; The arguments, when executed by the Mythic `command`, have an extremely high chance of being detected by the identified EDR as malicious."
        ],
    )
    candidates["has_ldap"] = Noul(
        instructions="""
            Has the user provided a ldap query within the `arguments` field?
        """,
        criteria=NoulCriteria(
            true="The user provided a ldap query within the `arguments` field.",
            false="No ldap query was provided within the `arguments` field.",
        ),
    )
    candidates["grade_ldap"] = Score(
        instructions ="""
            What is the likelihood that the users typed ldap query within the user typed `arguments` will be detected by EDR `edr_type` on the target system running `os` when executed by the implant?
        """,
        criteria=[
            "none; No ldap query was identified",
            "low; The ldap query looks mostly benign or does not have a known malicious intent.",
            "medium; The ldap query looks somewhat suspicious and is indicative of a potential malicious reconnaissance activity or malicious intent.",
            "high; The ldap query looks very suspicious and is indicative of a potential malicious reconnaissance activity or malicious intent.",
            "critical; The ldap query looks extremely suspicious and is indicative of a potential malicious reconnaissance activity or malicious intent."
        ],
    )
    candidates["has_cmd"] = Noul(
        instructions="""
            Has the user provided a cmd.exe command/script to execute within the `arguments` field?
        """,
        criteria=NoulCriteria(
            true="The user provided a cmd.exe command/script to execute within the `arguments` field.",
            false="No cmd.exe command/script was provided within the `arguments` field.",
        ),
    )
    candidates["grade_cmd"] = Score(
        instructions="""
            What is the likelihood that the users typed command, when executed by cmd.exe, will be detected by EDR `edr_type` on the target system running `os`?
        """,
        criteria=[
            "none; No inline command was identified",
            "low; The inline command looks mostly benign or does not have a known malicious intent.",
            "medium; The inline command looks somewhat suspicious and is indicative of a potential malicious reconnaissance activity or malicious intent.",
            "high; The inline command looks very suspicious and is indicative of a potential malicious reconnaissance activity or malicious intent.",
            "critical; The inline command looks extremely suspicious and is indicative of a potential malicious reconnaissance activity or malicious intent."
        ]
    )
    candidates["has_powershell"] = Noul(
        instructions="""
            Has the user provided a powershell command/script to execute within the `arguments` field?
        """,
        criteria=NoulCriteria(
            true="The user provided a powershell command/script to execute within the `arguments` field.",
            false="No powershell command/script was provided within the `arguments` field.",
        )
    )
    candidates["grade_powershell"] = Score(
        instructions="""
            What is the likelihood that the users typed powershell command/script, when executed by PowerShell, will be detected by EDR `edr_type` on the target system running `os`?
        """,
        criteria=[
            "none; No inline powershell command/script was identified",
            "low; The inline powershell command/script looks mostly benign or does not have a known malicious intent.",
            "medium; The inline powershell command/script looks somewhat suspicious and is indicative of a potential malicious reconnaissance activity or malicious intent.",
            "high; The inline powershell command/script looks very suspicious and is indicative of a potential malicious reconnaissance activity or malicious intent.",
            "critical; The inline powershell command/script looks extremely suspicious and is indicative of a potential malicious reconnaissance activity or malicious intent."
        ]
    )

Now, this is where Jev's weakness for offsec tasks comes into play. In my testing, the model just wasn't strong at domain-specific tasks such as offensive security. I don't think this is anything too unexpected, as TypeSafe explicitly state this Jev is designed to be a "System One" model. But this did mean I needed to do a lot of handholding in the state to give the model a fighting chance at identifying issues.

Sidenote - I'm conscious that this may be a misrepresentation of the models purpose and may also be intended behaviour. Rather than storing a dense corpus of facts in its weights, it could be that Jev is just focused on being efficient at taking general purpose questions to be answered from its state information. But only experimentation will give us just how much general knowledge is stored in the model, until some information on the architecture is released.

I won't post the full state here due to how long it is (you can see the complete POC on GitHub here if you are interested), but to give you an example of the type of information held:

state = {
    "schema_info": {
        "mythic_commands": [
            {
                "name": "Name of the command",
                "usage": "Usage of the command when entered into the Mythic input field",
                "description": "Description of the command and its actions",
                "execution_method": {
                    "win32": "Implant will only use win32 API calls to execute its tasking.",
                    "internal": "Command will be executed internally by the Mythic framework and will never be executed by the implant directly.",
                    "cmd.exe": "Command will be executed on via the implant using cmd.exe /c <command>",
                    "powershell.exe": "Command will be executed on via the implant using powershell.exe -c <command>"
                }
            }
        ],
        "target_info": {
            "edr_enabled": "Whether EDR is actually enabled on the target system.",
            "edr_type": "The name of the EDR agent running on the target system.",
            "edr_ldap_query_inspection": "Whether the EDR agent is configured to inspect LDAP traffic and alert based on suspicious queries"
        }
    },
    "target_info": {
        "edr_enabled": True,
        "edr_type": "Crowdstrike",
        "edr_ldap_query_inspection": True,
        "os": "Windows"
    },
    "command": "",
    "arguments": "",
    "mythic_commands": [
        {
          "name": "assembly_inject",
          "usage": "assembly_inject [pid] [assembly] [args]",
          "description": "Inject the unmanaged assembly loader into a remote process. The loader will then execute the .NET binary in the context of the injected process.",
          "execution_method": "win32",
          "notes": "Validates the PID, then uses the selected injector. Default sinks: VirtualAllocEx, WriteProcessMemory, VirtualProtectEx and CreateRemoteThread; payload I/O uses a named pipe."
        },
        {
          "name": "blockdlls",
          "usage": "blockdlls -block true|false",
          "description": "Block non-Microsoft DLLs from loading into sacrificial processes.",
          "execution_method": "internal",
          "notes": "No immediate OS call. Future process creation uses UpdateProcThreadAttribute with PROC_THREAD_ATTRIBUTE_MITIGATION_POLICY, then CreateProcessW."
        },

Finally I assigned weights to each scoring criteria to calculate an overall score out of 10.0 which could be tweaked based on testing:

weights = {
    "command": 0.5,
    "arguments": 1.0,
    "powershell": 2.0,
    "cmd": 2.0,
    "ldap": 2.0,
}

Then the rest is just Flask wrapper code to allow the Mythic JavaScript to invoke this code as an API and draw the gauge.

Initial Observations

Jev's standout feature is absolutely its speed, followed quickly by its very low cost. Given that this POC was using a JavaScript function to monitor keypresses in an input field, it is only because of Jev's quick response time combined with answering several questions in parallel that this demo was possible without having to create my own classification model.

And the ability for Jev to identify context within commands being typed was an unexpected highlight of the POC. For example, given that the provided state shows information on each Mythic command, if we type "shell dir", we see that Jev correctly picks this up as a cmd.exe argument:

If we extend this to include some Powershell, Jev now deselects the cmd option and correctly picks the Powershell category:

And if we throw both in there, Jev picks both tags correctly:

On the surface, much like with early LLMs, you can see the huge potential for this technology, which is getting me excited about what future versions and design patterns are going to look like (I told you, I don't get out much).

But if you're thinking of firing over a "Will this malware sample be detected by Crowdstrike?" prompt you might have to wait a while longer before that becomes a possibility.

That being said, there are a lot of areas of offensive security that can benefit from this model without having to rely on deep technical knowledge.

Hunting for Sensitive Data

One example of this is the "hello world" of offensive security ML, hunting for sensitive data on file shares.

I (or rather an LLM) created a file structure consisting of 1000 "corporate sounding files" nested in "corporate sounding directories":

Man, PTSD from working in corporate!

Next, 5 sensitive files were hidden in this file structure by an LLM:

/01 Executive/Restricted - M&A/CEO Passwords and Private Accounts - CONFIDENTIAL.txt
/05 Information Technology/Information Security/Privileged Access/PRODUCTION - Break Glass Admin Credentials.txt
/05 Information Technology/Cloud Platform/AWS_Production_Root_Access_Keys.csv
/02 Finance/2026/Treasury/URGENT - Treasury Portal Login and MFA Backup Codes.txt
/01 Executive/Restricted - M&A/CEO Passwords and Private Accounts - CONFIDENTIAL.txt

Then I wrote a quick Python script to enumerate the file structure, collecting all filenames and paths, batching them into buckets of 250 each. These were then all sent off to Jev with a question of:

Of the file paths presented, which are most likely to contain sensitive data?

No state was used (although you would probably use state to tune what would be considered "sensitive data").

Using the Choice type, I had Jev score each of the buckets, keeping the top 5 highest "most sensitive" paths, and recursing on each until I was left with the top 5 filenames of the 1000 starting paths.

And the result?

The 2.323 seconds is my terrible coding, that can be made a lot faster

The correct selection of files, taking ~2 seconds in total for all files!

With tuning you can certainly get this number down, but from directory tree to a ranked list of candidates in 2 seconds is pretty awesome.

Finally, I also wanted to see if Jev could pick up on sensitive strings within each file it identified. Using the same batching system, this was the result: