Episode 700: A New Co-Host, TypeSafe’s Jev, and Qwen3.8 at Home

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

We've been doing this for so long the CTO advisor podcast for so long that I stopped at around 300 or 400 or so episodes. I stopped numbering them, but We are adding a bit of rigor. So I have to we have to invent a number I think we should we're going to call this episode 700 and for episode 700. It's kind of like Batman 500 Calvin. Yeah We were rebooting the program I want to welcome my new co-host Calvin Hendryx-Parker Calvin welcome to the program.

I'm excited to be here Keith I I had approached you a while back and said I think there's something we should do to collaborate together And this seemed like a really good way to to help Get going again and had that discussion around what CTOs need to hear how we Give them direction and the advice that they're looking for and I couldn't think of a better person to host it with because You're kind of like my yin and my yang. You're my infrastructure to my development and I'm excited to have both perspectives on on the pod yeah, it's really interesting because Those lines are beginning to blur if someone asked me today.

Hey I'm looking to get into Technology, where should I start on the infrastructure side or the development side? The answer is kind of Yes, it's yes. It's right in the middle there and then even from a CTO perspective and I know you've had this challenge yourself What do I need my people to learn and the answer is again? 58. What's the Qwen? 338 the Qwen the Qwen 3-8 model from Alibaba. Yeah, so both big topics Let's start out with you just giving an overview of Both.

Well, let's start with Jev. What is Jev? Let's start with Jev If you've not been living under a rock for the last two weeks This has probably been one of the biggest news items has come across in a while as far as kind of super Accelerant growth and what's interesting is it's not a model you can talk to so I think that's one important parts to first clarify with With the the folks out there who are having questions about this is what is Jev?

Jev is a classifier model is going to look at a Crate set of criteria a set of like judgment statements and answer things like yes or no or answer a Confidence score about something or it can answer a multiple choice Given these five options what the user just typed into the prompt is does it satisfy one of these options? 5 that just released today You can't fill Jev's contact when context window like that. It's going to take those options of those Data you pass to it put it in parallel and ask the question simultaneously across say a million rows and give you back an answer in milliseconds and And that I think is probably the thing that got me to go ah So one of the things that I've been doing in the layer 2c lab has been a lot of fine-tuning and One of the most difficult things about fine-tuning has been question pairs and Finding the right set of question pairs and you know You don't need that big of a model to be a decent classifier No, he you know a decent amount of compute and I have a decent amount of compute locally with my NVIDIA Sparks, but it still takes time and effort to go through let's say 250,000 segments, which is what I have on my data set and then create or or Wait answer pairs and say is this you know, hot dog, not hot dog, whatever that whatever I'm yeah And that's and that's where you're getting that and and I think what's interesting is like people have taken For example the other topic here Qwen 3a they have fine-tuned the six billion parameter version of that model to be a classifier to basically be an open weights version of Jev because the other side of the coin for Jev is it is a third-party Model that is proprietary closed weight and closed source.

You don't know who's behind it. There's the what's the name of the company? It's TypeSafe. Yeah type TypeSafe AI. I don't know who TypeSafe AI is but if you are handing them 200,000 rows of your data to do a classifier against they now have your 200,000 rows of your data And in some people may be thinking yeah, okay, that's not a big deal my you know I'm running through this stuff through open AI or whatever. So that's not a huge deal But what is a huge deal is the nuance both of us have done enough AI and I think this is the executive level concern is That can a general-purpose Model even a classifier know the nuance of my business Enough to give me Effective results.

What would Keith Townsend say or what would Calvin say when it comes to X Y or Z? We may have enough data out there that there's enough data in the model that that answer will come Relatively strong, but what happens if I don't mean Keith L Townsend and I mean Keith, right? Or that do I so you still fall into the Challenges that we have with with general-purpose models that are not tuned to our specific use case Yeah, and you Jev again being closed weights.

I don't know if I mean this is so new the game Right now you go in Luckily invites are back open again. You can go sign up you put down a credit card and you're on your way But like I don't think you can do fine-tuning yet on top of that model. You can't A bunch of questions like, you know can't can there be a vector store behind this and a search and You know, basically a rag pipeline for adding my own data What because it's not a model that I've talked to But obviously it works like any other model in the sense that there has to be some type of vector search capability in addition to the models base training itself and Plus base training as we know base training.

You can't as much as I love All the secondary stuff whether it's fine-tuning or rag It's very very difficult to defeat the preferences of a base training on a large language model Just try to do voice or or some type of writing with one and see how many times That model will default to his training versus your guard yeah, well and I think but the Model like Jev and the concepts that it presents to the end users Will allow them to produce better for example eval frameworks in your CI pipeline You may want to incorporate this as part of your eval framework to be like how close to the mark is The current large language model I'm talking to getting to what I was expecting from that prompt Yeah, so I'm not dismissing the technology at all.

It is an absolute brilliant use of a large language model I've been thinking about classifiers for a while now Whether we're talking about Google big data or big query and you know kind of using big query to do some with some similar Activities, you know what? It's nice to have something kind of squishy and a little bit probabilistic Yeah doing this what was deterministic work. It blurs the line. It gives us. Yeah Yeah, another tool no quiver I think it's just one you got to be a little careful with because we don't know on The nuances yet.

Is there a business account? Is there a team account? Is there an enterprise agreement? Can you get a BAA like? 58 and why should we care? Well, the the 3 3 8 is the latest release from the Alamada folks What's interesting is it they're keying us up for what's going to be the the Qwen for release with the flash next Release of that model. So the 26 billion parameter version of 3 8 which released in rapid succession Which is why you find a lot of people saying oh, it's 3 4 3 5 Nope, it's 3 8 got released like what week and a half ago two weeks ago But I put it to use right away in my house on my homeless home automation System with my internal voice usage.

So none of my data leaves the house. 5 for some reason. 5 this stuff is moving exceptionally fast, but Gemma For 26 be 26 or 27 be 26 B is a 4 billion parameter moe model that does this so if you're familiar with a Mixture of expert models from the Gemma for sense. 8 larger model more parameters, but smaller In memory footprint which allows us to run this one lighter Infrastructure to give Hannah a sense of scale the Gemma models There's a Gemma for 30 B.

I basically get about 10 tokens a second running that on my NVIDIA Spark the 26 be with the 4 billion active parameters. 8 was the Closest thing that I got to a frontier model. Yeah without running it on dual sparks there's there's a couple of Larger models that compete but run it on a single spark I was able to complete like 21 or 21 of 22 really complex problems of Which only the closest thing that came locally that was not running on two sparks was like 17 of 22 so yeah, I do like that kind of just playing around with it locally, and I took the Was it Scott Pocock's research skill That's kind of a deep research tool usage and I put it against the Qwen 3 8 26 B 27 26 27 B model on my I have a framework desktop with the AMD plus chip on it and Boy, did that thing work hard and actually complete the tasks and produce what I wanted Which was a lot of web research a lot of creating spreadsheets a lot of creating markdown research and you know It's been a good, you know our against the research The puppy has opinions.

Yeah, he did. He had to get some real opinions about the grass fan mode outside right now. Yeah, so Where I want to kind of wrap this up and bring these two Components together is that one open wave models are getting yeah, very good. Not only are they getting good They're getting more and more efficient. I think more so than we could ever Imagine Jev is something like a 42-per-million Otis or something to that effect but I Think most CTOs I talked to are not necessarily in a rush to go out and try Jev Yes, it's fun.

It's cool. It is fun. 8 classifier specifically to emulate What Jev does? Yep, that's the ones on my list the Jev K5. There's also sim f sim f which used to be open Jev What's interesting is but the first one there the Jev K5. 0 licensed open source open weights model and This is you know where we're and we'll tease this. Maybe we'll talk about this next week is this idea of Cloud Versus local Inference and where should the investment go?

Both of us have made not small investments in local AI Yeah we have some sense of what can practically be done with local AI and what I would love to talk about is the cost benefit of AI Locally versus just simply consuming API's from the cloud, which is exceptionally cheap because the math I don't know if you've done the math I haven't run into much math that has told me that local AI is cheaper than cloud AI so there has to be advantages and what considerations are we seeing in the field?

Yeah I think it depends on what's in the field and I think you also see that you get a consistent price over time If you buy your own hardware or rent GPUs in the cloud as opposed to the API endpoints Yeah, this is a bit of a spoiler alert. This is This is cloud math like the Application may have changed but the math has not a lot of their drivers for why you would use local versus cloud or why you use cloud versus Local has not changed if you have strong opinions about this share it in the comment sections on the video session of This podcast or I don't know if the CTO advisor has a comment section I don't think it does people don't comment on blog post anymore, but not anymore I'm sure we'll get in the video.

We'll get some comments. I'm sure people will have opinion There's always some type of opinion about local versus people on the internet with an opinion You know what? The best way to get engagement is to be wrong on You'll get all the Everyone everyone in the world will correct you. Yeah, Calvin. This is your first time on where can people find you? One close and plus plug. What do you do for them? Like we didn't say what do you do?

So I am CTO and co-founder of Six Feet Up We are a Python and software agency that love solving hard problems. You can find me on LinkedIn It's probably the most active spot I'm at so just calvinhp and all the other socials whether you're on X Mastodon Bluesky Come find me on those spots, and I'd love to chat All right, and if you want to find me, of course If this is Calvin's audience for the first time you're trying to find me. I'm Keith Townsend founder CEO of the CTO advisor the the the Name of the podcast you can find me on the webs at CTO advisor on all major platforms.

I I'm still pretty Hard X Twitter user that is probably the quickest way to engage with me Otherwise, I do spend an awful lot of time on LinkedIn if you have suggestions About what we should be talking about if you have questions. My DMS are open on X Calvin, I don't know if you're saying that's the case you Calvin's DMS are open on X at me on LinkedIn More than happy to take on requests or answer questions in the case. We're pretty active on socials until then We'll talk to you episode 701 sounds good CEO advisor podcast Calvin go watch some baseball, man.

Sounds good Keith you too