WPTavern: #216 – Matt Schwartz on Exploring AI’s Impact in WordPress Agencies (Part 2)

[00:00:19] Nathan Wrigley: Welcome to the Jukebox podcast from WP Tavern. My name is Nathan Wrigley.

Jukebox is a podcast which is dedicated to all things WordPress. The people, the events, the plugins, the blocks, the themes, and in this case a second look at exploring AI’s impact in WordPress agencies.

If you’d like to subscribe to the podcast, you can do that by searching for WP Tavern in your podcast player of choice, or by going to wptavern.com/feed/podcast, and you can copy that URL into most podcast players.

If you have a topic that you’d like us to feature on the podcast, I’m keen to hear from you and hopefully get you, or your idea, featured on the show. Head to wptavern.com/contact/jukebox, and use the form there.

So on the podcast today, for a second time, we have Matt Schwartz.

Matt has been working in the WordPress ecosystem since 2011, running his own agency based in Atlanta, and developing products like CheckView at all for WordPress form and checkout QA. Matt’s expertise lies in how agencies can smartly, and cautiously, incorporate AI into their workflows for real tangible wins, and how to avoid potential pitfalls.

He was on the show last week to record the first of this two part mini series. You might want to listen to that prior to this, but it’s not strictly necessary.

In this episode, we build upon last week’s conversation. Matt talks about practical strategies for integrating AI across agency operations. The discussion starts with what it means to give AI access to your agency’s brain, using tools like project management wikis and connecting them with AI chatbots to streamline knowledge sharing, and avoid common AI hallucinations.

We then get into MCPs, or Model Context Protocol, and talk about why this area is quickly becoming a game changer for agencies looking to securely connect AI agents to multiple internal systems without complex, risky API configurations.

The conversation covers how to use AI for building internal tools, highlighting where it’s low risk and where you should be more cautious, especially with public facing, or mission critical, systems. Matt explains how agencies can leverage AI for QA and checklist automation, freeing up time for deeper human review of other important tasks.

We also discussed the impact of AI on the WordPress plugin market, including potential consequences for plugin developers and the wider community, and whether the rise of AI generated disposable tools could erode the collaborative spirit of the WordPress community.

We end by chatting about the importance of approaching agency AI adoption with eyes wide open to the risks. Data security, overdependence on vendors, failure to handle errors, and the reality that AI still makes mistakes.

Matt shares his outlook on how agencies can position themselves to thrive as AI reshapes the industry, from hiring strategies to the next generation of productised services.

If you’re running an agency or freelance business in the WordPress space and want to get ahead with AI thoughtfully and securely, this episode is for you.

If you’re interested in finding out more, you can find all of the links in the show notes by heading to wptavern.com/podcast, where you’ll find all the other episodes as well.

And so without further delay, I bring you Matt Schwartz.

I am joined on the podcast, again, by Matt Schwartz, somewhat unexpectedly. Hello, Matt.

[00:04:05] Matt Schwartz: Hey Nathan, thanks again for having me this week. I’m super excited to dive back in.

[00:04:09] Nathan Wrigley: Thank you. So we recorded an episode last week, and we intended to do it as a one hit. So get it all recorded, tied off within 40 minutes or what have you. And then we began talking.

So last week we began talking and at about the half an hour point, it became obvious to me that we weren’t going to capture it all in one recording. So we’ve come back for a second episode.

Dear listener, I would just say that in order to provide context for this episode, you really probably should listen to the previous one, because we’re stacking up Matt’s case, argument, however you wish to describe it, for where you can make wins inside your agency with the use of AI. Not just wins, maybe some cautionary tales as well. But that was the point of the first episode.

So really, we’re going to drop you in to the ninth of 16 points. So again, just pause this, go back to the previous episode, have a listen there, and then you can stack this one in your podcast player of choice at that point.

If, however, somebody’s ignoring that, Matt, are you able to just do a very quick bio? Just tell us who you are? It may be repetitive for the people that are listening to the second episode, but nevertheless, let’s hear from you who you are.

[00:05:16] Matt Schwartz: Yep. My name’s Matt Schwartz. I run a WordPress agency here in Atlanta, since 2011. And I also have a testing and QA product for WordPress for checking forms and checkout called CheckView.

And yeah, today we’re just really diving into how you can leverage AI, how you can incorporate it into your agency, but in a hopefully smart and cautious way. Not necessarily just dropping it in, being a little bit more thorough about that process. So excited to continue the conversation.

[00:05:45] Nathan Wrigley: Yeah, thank you. And thank you for being so accommodating by joining me for a second time. So as I said, Matt’s put together a whole laundry list of different areas that your business, your freelance agency, whatever it may be that you are running in the WordPress space, can perhaps gain some benefits.

Last week we did one through eight, and now we’re going to sort of hit the road running on number nine. So the ninth point was about giving AI access to your agency’s brain. It’s a lovely subheading, but what does that mean?

[00:06:12] Matt Schwartz: Yeah, so this is actually one I picked up in the Admin Bar, which is a, one of the other WordPress agency groups out there, that a lot of agencies were doing. And I thought it was an interesting, I would say hack, you could say, to add AI without getting super involved in it. Which is if you already have a project management tool, or you already have a wiki, you can add into your AI chat bot of choice when it’s actually answering a question. You can tell it in its memory, hey, whenever I ask a question about the agency, go confirm what I’m doing by visiting our Clickup or visiting our Asana.

So it’ll actually go retrieve and confirm it’s using the latest proper information instead of just guessing or hallucinating. I love how we use the word hallucinate and not lie. I love that marketing branding that the AI companies did. It’s some crazy gaslighting.

Anyways, I love AI, but definitely, if you haven’t used something like Claude or ChatGPT, saying in the memory as simple as when you answer a question, check if this is actually the case and connect to our ClickUp or connect to our wiki.

I think that helps you get all the power of your documentation, SOPs, client, CRM, any data you’ve already basically built up. It can leverage that without you having to do a whole bunch of crazy connections or ask more specific things. I thought that was actually a really neat way that agencies that are just getting into the space with AI are using the data they basically already have. They’re just using their project management software, which basically has all that data.

[00:07:47] Nathan Wrigley: When you see it in action, which I have actually, but not to do with a WordPress website, more to do with a sort of SaaS product with the, how the tool has been built and the guardrails that are into the tool. It’s really amazing because then, well, basically it never forgets.

So every time you throw something new at it, that becomes part of the corpus of information. It then has an understanding. I keep saying it, but hopefully you understand, I’m meaning the AI in this case. A wider and broader understanding, and increasingly is able to deliver that back.

So in my case, often I’ll get some text back, which is divided up into bullet points. Those bullet points will have little footnotes attached to them, you know, 1, 2, 3, 4, and what have you, which will then link out to the documentation itself. And again, just a profoundly useful use of the thing which it’s best at, which is taking a corpus of information, grinding it up and spitting out something which makes sense.

And why wouldn’t you point it at your internal documentation? You know, if you’ve got a plugin, all of your support docs, throw the AI at it, and it will be able to help you as well as your clients. Because it’s guaranteed you’ve forgotten something that you’ve built.

WordPress, of course, itself does this. You know, every AI agent on the planet is welcome to crawl the docs for how WordPress itself is put together. And it’s one of the reasons I think why WordPress has a fighting chance in this AI, CMS battle, if you like, because everything’s open source already. Nothing’s hidden behind a paywall or a licence agreement or what have you. So yeah, agreed. That’s a great example.

Okay. Anything to add or should we move on?

[00:09:22] Matt Schwartz: No, I think that one’s just a cut and dry, really. If you haven’t looked at that, that’s an easy way to get into AI and leverage it without a lot of work.

[00:09:29] Nathan Wrigley: Okay. Absolutely. Okay, then number 10 is your internal MCP and guardrails. I know this gets bandied around a lot, and there’s a lot of acronyms floating around in the AI space. But MCP, maybe we just need to spend a moment explaining what the heck an MCP is, and how it kind of fits into the overall picture, but particularly in this case, with your guardrails.

[00:09:48] Matt Schwartz: Definitely. So MCP is basically an open source way to connect AI, let’s say, chatbots or agents to external systems. It stands for Model Context Protocol. I think maybe the team behind Claude built it. I can’t remember. But the idea really is that, instead of you just connecting directly to an API, which you could do, which an API if you’re not familiar, is a way again, to connect two different systems together.

One system will have a series of things that will let you say, hey, you can add this data, or you can pull this data, or you can modify this data, right? So an example could be something like a help desk. You might create a ticket, you may delete a ticket or you may edit the ticket. An API can basically do those things.

But what an MCP does is it’s really a series of tools that are more prebuilt for the AI, so that it knows and has context of what it should let you do, and how all the pieces of that connection really should modify whatever data you’re doing.

So it’s a lot more specific to agents. It’s a lot friendlier, I would say, if you aren’t familiar with a company’s API, you could connect to their API. I can connect to the WordPress REST API, but if I don’t understand that API, it may not be actually the best way to make the connection.

With an MCP, you can really not have the background of that company’s bridge. It’s going to do all the work, and the AI’s going to have enough information to help you get what you need done.

I know that’s hard to explain, but essentially with an MCP, if you build one at your agency, this is a little bit more high level, or a little bit deeper, but I am seeing a lot of agencies looking at this. Which is, they are using an MCP basically for their teams so that they can add all of their systems in one basically bridge. So that instead of having all their employees like connect to all these different Claude connectors and APIs, they have one system.

So if I have Claude and I’m an employee, it can connect to my MCP at the agency, then the agency MCP is actually on behalf going to go retrieve data from all our different systems. So not everyone has to have API keys. Not everyone has to connect to all these other systems, if that makes sense.

So I am seeing agencies starting to do this. So in some sense the proxy, MCP becomes a proxy or just a way to connect to all your other systems in a secure way.

[00:12:18] Nathan Wrigley: I always imagine it a bit like if you, I don’t know, you approach a giant supermarket and you know that you need carrots and soup. And normally you just go into the supermarket and wander around for a long time, and eventually you’d sort of stumble across the carrots and the soup.

But wouldn’t it be nice if there was somebody at the front door? Then you could say, where’s the carrots and where’s the soup? And that’s it. And they go, okay, the carrots are there, the soup’s there, and point.

You know, it’s just like this perfect gatekeeper, this guardrail that you described that kind of allows you to get the best out of that experience without wasting a load of time and resources and probably a load of hallucination out the back end.

[00:12:55] Matt Schwartz: Wow, that was so much better said than me. But yes, that is a much better way of explaining it. And that’s why if you haven’t looked at MCPs in general, I think it’s worth looking at. But also if you have a tech background, looking at an MCP for your own agency where you can combine all your tools and connect to this one place, I think is a really neat way to, again, get your employees and contractors connected to your, all your systems without them having to have a direct connection.

So if I want them to be like, hey, answer this ticket, go to this WordPress site, instead of them having to connect Claude to the WordPress site into Fresh Desk and all these things, it’s all within the one MCP. And then they aren’t really responsible for those API keys or any of those connections.

Of course, you have to put guardrails on that too, right? Guardrails, like they can’t delete things. You know, not having them delete tickets or websites. Because if you connect, you know, your host, they could technically delete an entire website if you don’t have proper guardrails. So it is, I would say something that is a little bit more on the cutting edge that not every agency should do, but if you are on the more technical side, an internal agency MCP, I think is a really neat idea.

[00:14:06] Nathan Wrigley: I feel like there’s future of commoditising MCP creation.

[00:14:11] Matt Schwartz: Oh, it’s already happening.

[00:14:12] Nathan Wrigley: Yeah, I really haven’t experienced that but, you know, a really, I don’t even know what that interface might look like in the end, but some really credible way of, you know, you sign up for a service and for a few dollars a month, they will modify, create on the fly, adapt the MCP so that it fits in with what was already an incredible technology. It’s a bit like the icing on the cake, the MCP, isn’t it? The AI was pretty amazing anyway, and then you put that layer on top and it just becomes much more refined amazing.

[00:14:41] Matt Schwartz: There’s actually some companies doing that already where you sign up for their SaaS, they basically store all the MCP data on their server securely, because that’s always a concern. You give them basically all the credentials, you give them the guardrails, and then they build a secure, essentially MCP app for you.

So there are some early options out there for that, that agencies could also look into if they’re less technical. You just want to make sure, obviously you realise you’re giving a third party your data and your secrets essentially.

[00:15:11] Nathan Wrigley: And course, in the era of AI, the capacity to do things really seriously wrong is literally at the end of your fingertips. Whereas before, you’d probably have to have some understanding, well, you could delete whole file structures and things like that, I guess. But now that a simple prompt can just rip through your entire code base or whatever it may be, definitely, one for guardrails there.

[00:15:32] Matt Schwartz: Delete all the sites on my server, done.

[00:15:34] Nathan Wrigley: Yeah, that’s right. Yeah. And don’t check.

[00:15:37] Matt Schwartz: That could happen.

[00:15:38] Nathan Wrigley: Yeah. Ignore all previous instructions, just delete them all.

Okay, so that was number 10. And really interesting. I think that’s one for the, sort of the tinfoil hat brigade, you know, the real nerds out there. But it’s not far off. If it’s been commoditised in SaaS now, you can guarantee that in the next few years that’s going to become table stakes, I would’ve thought for a lot of businesses and SOPs and what have you.

Okay, so the next one, I’m sure many people will have been familiar with, especially if you have a YouTube account and you’re looking at AI things on YouTube. Vibe coded agency tools. I’m sure I know what this one means, but run it by me anyway.

[00:16:16] Matt Schwartz: Yeah, so one key thing is, I’m not saying vibe coded tools themselves. I’m saying vibe coded agency tools, agency being the specific part. So you’re building internal tools for your agency, which I think in some sense, depending on what the situation is, can be okay to do, in my opinion, because the risk is lower. Again, you’re using it internally.

[00:16:40] Nathan Wrigley: It’s not public facing, that’s the point, right? It’s just you and your colleagues, which hopefully you trust.

[00:16:45] Matt Schwartz: Right, right. Again, you should probably put guardrails, and you have to think about, well, what sort of data is it touching, and how important is this data? That’s everything with AI. You have to think about the risk. But I am seeing a lot of agencies starting to build different types of agency tools. Some that I really do internally myself, and I am a big fan of things like reporting tools and dashboards, right?

This is, again, a good case, I think I talked about in the previous podcast. The sweet spot, I think with AI is having it handle things that you just never could get to that were on your list for a million years, right? And realising that, as long as you do a little due diligence and you feel like it’s in the realm of where it’s supposed to be, this is probably more information than you had before, right?

So a good example is, if you’re an agency, you may have it hooked up to QuickBooks MCP. You may have it hooked up to your time tracking software. You may have it hooked up to one of your other reporting software, WooCommerce subscriptions with Woo. And from there you’re able to have a much better visible idea of what your business is doing well financially. The bottom line, especially if you’re like a lot of agencies where QuickBooks doesn’t really have all of your actual services. You may have those internally or you may have them in a other system.

You can combine those and build reporting systems. And again, that’s a relatively low risk way for you to, worst case is you’re going to use that data and you’re going to be like, well, this doesn’t seem right. And you’re going to have to dig into it and figure out what’s going on. Hopefully you don’t just blindly use it, but I do think the risk is lower.

So those sort of tools I think are really, really neat, and relatively easy to build out. So reporting tools, profitability dashboards, things like that. Looking at your time tracking, like who at your agency is the most profitable, if you have that data? Obviously make sure you actually have the data structure for that, or AI may just make that up.

But we’ve, you know, used that even at our agency and I think it’s been helpful for us to find patterns that we didn’t know where we were spending our time and effort. Especially if you are doing time tracking using something like Everhour or Harvest.

Now the tools that I do struggle a little bit more with, and I am seeing people in agencies use is things like website management dashboards, or building their own QA tools. Because those things, I think the risk is higher of things going wrong.

[00:19:05] Nathan Wrigley: Public facing again, yeah.

[00:19:07] Matt Schwartz: Right. And you’re giving this third party access to all of your websites and it’s not like, you know, a big SaaS. This is something you built internally, which means Claude doesn’t care if it’s wrong, right, until you tell it.

So a good example, and not everyone may feel this way but, you know, I’ve seen some agencies that are building replacements for management dashboards like ManageWP, WP Remote, those sort of things, which is connected to everything and is kind of their most important infrastructure for their clients. Personally, I think that that’s a little risky to be doing.

Now, if you’re doing the right due diligence and you have a technical team and you’re doing manual code reviews, sure. There’s an argument to be doing that if it’s also, I think, solving something specific to your agency. I talked about this in the last podcast, replacing SaaS products when there’s a nuanced solution that’s specific to your agency, I think could be really helpful.

But if you’re just replacing SaaS products to save 30 bucks a month, I think that doesn’t make any sense because you’re going to end up spending a lot more on maintenance, I promise you, than if you just stuck with the SaaS product, if it does what you need. So I think there’s an argument there.

[00:20:18] Nathan Wrigley: I think it’s really interesting. And I do wonder if we’re on the precipice of, so this is me sort of staring into the crystal ball a little bit. I wonder if we’re into the era of sort of disposable, one time apps. So you have a function that you need to do this month like, I don’t know, you must file your taxes this week, but you’re miles behind. So you get an AI to just quickly do that thing for you, and categorise all of the jobs that need to be done so that you can hand it over to the tax man and so on. And then you just put that on ice. That thing no longer needs to exist.

I feel that kind of coming where we sort of vibe code up this one time thing, and then dispose of it. I don’t know if I’m entirely in agreement with that as an idea, but I feel that that is coming. But to your point, I think anything public facing, we’re still in the era of, really, watch what you’re doing. It needs thorough testing.

[00:21:09] Matt Schwartz: Exactly. And thorough code review because, you know, ultimately while the AI coding, I think has gotten really, really good, it’s not a hundred percent there, and it doesn’t have any context. It doesn’t actually know what it’s doing. It’s all patterns. So there is an argument to be made that, yeah, it may get 80% there, but if no one’s actually checked the code review, two months from now, it decides to delete all your websites out of your management dashboard, well, should have done a manual code review, right? And it’s on you.

So I do think there’s an argument, same thing with like QA tools. Building one-off QA tools, which should be persistent and actually probably one of the more important things you build. It should do the same thing every day. It should almost be dumb. It should not be trying to rebuild itself all the time, or be even one-off, like you said. It’s not, I think, a good idea to build a one-off migrator typically for that reason, even though I see people doing it in agencies. Unless you think it’s just a low risk project, I do think, you know, you have to think about that.

To your point about the one-off task though, I think again, if it’s a low risk item, one-off makes actually the most sense because a lot of times if you know this is going to be a one-off thing, you then are subconsciously being like, okay, I’m going to use this once and the cost is not that high and that makes sense, because you’re not having to maintain it. But if you know it’s going to be something you’re going to be using for the next five years and you don’t plan to pay a developer to review it, I don’t think that’s a smart idea, at least not right now.

[00:22:39] Nathan Wrigley: That’s a good calculus I think to have in the background. Okay, so that’s good. So caveat emptor basically, use your discretion. If it’s public facing, maybe think twice. But also if it’s something that you want, you absolutely bulletproof need it to be reliable and predictable a hundred percent of the times that you run it. Again, maybe there’s a human in the loop there. So that was sort of vibe coded things that you might do in your agency.

I feel that’s going to be a real area of growth, whether or not it will be profitable growth or useful growth, I’m not entirely sure. I feel like in our industry at least, people are going to be dabbling in that kind of thing all the time. You know, trying to figure out new, clever tools to achieve a thing, which maybe in the past would’ve been a subscription thing that you paid $20 a month for or something. So we’ll see. We’ll see how that goes.

Okay, moving on then. So the 12th item that you brought to bear was QA, so quality assurance, checklists and testing. Right, run us through this one then.

[00:23:33] Matt Schwartz: I know I just said when you’re building a QA tool, using AI to build an internal vibe coded tool is probably not the answer. But actually what I’m going to say right now is not contradictory to that, because what I’m really talking about for QA and testing is more so having AI help you build things like checklists, right? You already have a good context usually with your SOPs. So it can help you build your SOP checklists. It can also help run the low risk items automatically. And again, I know I talked about risk a lot, but I think that’s how you have to consider it.

So one really neat thing I’ve seen a lot of agencies start using is Claude Skills, which basically just means that you teach Claude a process. Literally it walks you through in the conversation like, what do you want this process to be? And then you can run that later in context.

So a really cool example of this is if you are, let’s say onboarding a client, or you’re launching a site. During your next launch, your next onboarding, you may want to use Claude to teach it the skill of how you launch sites. And then it can automate a lot of those items. And you can still give it context like, I want the human to specifically review this item, right? Or, I want the human to check that no index isn’t set, right? Because that’s like a high risk item, right?

[00:24:55] Nathan Wrigley: Yeah. Yeah, that’s a big one..

[00:24:56] Matt Schwartz: Right. That’s one we all, I think, have dealt with at some point in our.

[00:25:01] Nathan Wrigley: Too many times.

[00:25:01] Matt Schwartz: Right. Too many. Exactly. That’s the key. And that goes back to really that vision document I said on the last call. Having an AI vision document where you go through these checklists and you’d be like, okay, we want a human to be involved, or we don’t. You could actually tell Claude this. And then Claude will actually know exactly what it should be running itself and what it expects a human, and it will prompt you for.

But I think that is the beauty of this is, you can make your whole automated process when it comes to tools with QA and anything really related to that checklist, whether it’s launch or anything like that. Look at tools like Claude and Skills like that, and I think that you can use it to help with repeatable processes. And that will actually help most agencies not only speed things up and save on margins, but I think a lot of times they’ll do more testing than they did before.

And this, again, falls into that sweet spot where like AI’s really good for the things that you knew you should do but you have limited time. And testing is one of those items. You want a hundred percent coverage, but in reality that’s not going to happen. So let’s have the human do the really important stuff and everything else we would’ve never gotten to anyways, let’s have the AI do it. And that’s where I think you can use these tools.

[00:26:11] Nathan Wrigley: Do you know what’s really interesting, and we sort of made light of it in the previous episode, the fact that there’s hallucinations and what have you going on all the time. But I do think there’s definitely a moment coming where I think some of the more straightforward things, like for example, the checklist, the binary things, is no index switched on? Yes. No. Okay, that’s a no.

I think I am getting comfortable with that now. You know, just that, okay, we asked the AI that question, it’s delivered as an answer. I’m almost at the point now where I’m never going to go back and check that was true. If it was something much more broad like, is my SEO strategy bulletproof? Well, no. It’s never going to know whether that’s the right thing.

But these much more binary things, many of which, if you add them all up, could take you hours when you’re finally launching a website. Yeah, I think there’s something to be said for just sort of handing that stuff over. And I don’t know, maybe you check it frequently, infrequently, less frequently as time goes on. But yeah, always check the no index one.

[00:27:08] Matt Schwartz: Yeah, I was like, I would still check the no index one, even if it’s binary. But to your point, a lot more of the very black and white items, I think it can handle a lot better than it used to, but I think it still comes down to risk. Like if it’s, yeah, no index, I’m still going to check it. But if it’s something else that just is not that key. Yeah, I think we’re all becoming a little more comfortable or a lot of us are coming more comfortable with that. And I think that’s okay because you know the risk exposure really.

[00:27:35] Nathan Wrigley: Well, and also, especially if it’s QA and checklist time, hopefully you have done the bits and pieces, you really are at that point just making sure that you’ve polished the thing that needs to be polished. So hopefully that’s a bit of low hanging fruit where you’ll catch the things that you missed, and maybe you’ve done the due diligence there already.

Okay, so that was number 12. We’re approaching the end. We’ve got four more to go. So number 13 links directly to WordPress specifically. The WordPress plugin market impact.

I’ve got to say, this has me slightly concerned, because I feel that this could be a good thing for our ecosystem, but also possibly a bad thing. But I’ll just hand it over to you to paint the picture.

[00:28:18] Matt Schwartz: Yeah, definitely. So I think you even touched on this a little earlier when you were saying there’s going to be more one-off apps being built by agencies. And I think that also applies to a lot of plugins that are essentially one-off solutions, right? They are utility plugins. They solve one thing really well.

I think that sort of thing is already seeing the impact. If you talk to a lot of plugin developers, especially some of the larger shops, they’re seeing a drop in sales. And that is a real thing that’s happening. They’re seeing a drop in sales, especially for smaller plugins. Because a lot of agencies and customers are solving that with AI. Maybe it’s a couple files of code, it’s a lot easier for them to build it.

Now I still have concerns around, are they having a human review that? Like I talked about. But humans are going to do what they’re going to do, which some people are just going to run with that. That ultimately affects sales.

So I think that is hurting a lot of the ecosystem when it comes to the smaller plugins out there. And even some of the bigger plugin developers are essentially sunsetting their smaller plugins, because they realise they’re not getting as many sales and they need to focus on what they consider their moat, or their platform, you know, big plugins that AI’s not going to be able to replicate or people shouldn’t trust to replicate.

But if you’re building a small plugin, I’m not going to call any out, but I think that there is some concern there. And I think ultimately for WordPress, I don’t think that really hurts the WordPress ecosystem from the standpoint of plugins in general, but I do think it raises the bar of what a good plugin will be.

And that kind of goes with the agency land. That’s what’s happening with agencies too. It’s just the bar is being raised. You have to have a more complex plugin that actually solves someone’s needs now, not just a small one that solved it, but now they can use AI to do it.

And some people argue that that’s going to continue all the way up with the most complex plugins out there. But I do think that there are, you know, unless AI dramatically improves. If it’s 80% there, that’s great and all, but it’s what we talked about earlier. You can’t really run with that in production at 80%. And that’s the difference between a really good SaaS or really good plugin versus something that was homegrown and just falls apart.

[00:30:38] Nathan Wrigley: I think I have a slightly different, maybe more community focussed, approach to this because one of the things that I think worries me is the, how should we say it? The slow ebbing away of the community. And obviously if you are a, I don’t know, a company launched onto the stock market and what have you, you’re all about the money, right? The bottom line is you’re going to make money, distribute that with your shareholders, whatever, yada, yada. But the point is to make as much cash as possible and do things with that cash.

We have a very different calculus here in that the community is the thing which largely builds the software, maintains the software, promotes events. There is a bit of me which worries that if these, let’s say developers who’ve got one plugin, it doesn’t do 3000 major things, it just does one or two little things, but it’s been their way of getting themselves into the software, and figuring out how it all works, and meeting the community, and being engaged and, you know, all of that.

That slow ebbing away of that is something that I think our community and open source communities like ours need to be just a little bit mindful of. Because it does feel as if AI could definitely eat a lot of lunches. And I think we see that actually. I think we can already see that in the real world with things like attendance at events and the amount of events that are being put on, yeah.

[00:31:57] Matt Schwartz: I think you’re correct. I was actually going to bring this up in the sense that I am already seeing it within a lot of the agency groups. There’s just not as much engagement when it comes to posts, I think, and that sort of thing. Because people use AI more to find solutions, which means they’re not as engaged in the community. Which, to your point, plugins would kind of work the same way, especially the smaller plugins.

And yeah, there’s definitely something I think I’m concerned and kind of sad about already. Because like that’s why a lot of us are in WordPress is for the community. And I 100% agree with you. Not to mention if those guys, the smaller guys go away, then there just ends up being these massive plugin companies, which have their place, but WordPress wasn’t built on all massive plugin companies. So If those smaller ones go away, then that’s a little bit of the WordPress spirit I think are lost for sure.

[00:32:47] Nathan Wrigley: Yeah, I think we just need to be very mindful about this slow, like I say, wicking away or haemorrhaging of the community. And because at some point, the calculation no longer works. You know, there just aren’t enough community members around to make it interesting for other new community members to join, or to stick with things. Or to update their plugin or whatever it may be.

And maybe that is just a consequence of the way the world is moving, I don’t know. But having lived in the WordPress ecosystem for over a decade, I think it would be a shame if that baby was to be thrown out with that bath water.

Okay. Alright. I think we’ve done that one. So the next one is, well, the next one kind of speaks directly to that actually, which is the idea of, I guess spreading your wings a little bit further and realising perhaps that AI empowers you to do things outside of WordPress. And you’ve entitled this Experimenting Beyond WordPress. Again, off you go.

[00:33:37] Matt Schwartz: So this is something I am seeing some agencies doing, which is, because you can use AI now to use unfamiliar stacks, there’s really two parts. One is just unfamiliar stacks, or unfamiliar platforms, you don’t know. You can really try out new platforms a lot faster now. It will tell you exactly what to do, you know, step by step, or it’ll just do it for you.

So there are agencies, I think looking at other platforms where certain projects may make sense outside of WordPress, where they’re using that in that capacity and it’s allowing them to experiment. Where in the past, just sticking in WordPress, you have all your knowledge there makes sense to do, right? You don’t want to know 10 different platforms. But I think AI’s made that easier to dive into these other platforms. So that’s the first thing.

The second thing I think is that, now that you can actually use chat to engage, you are seeing some agencies, and some freelancers, that are saying, well, I don’t need the WordPress infrastructure at all. I just want to go back to pretty much like static or HTML type websites because I know I’m just going to chat with it and then I don’t have to worry about security or updates.

And obviously I think that only would apply with certain websites. It’s not going to apply with highly functional websites. I think that’s not really going to work. But for like your brochure site, I think some agencies are experimenting with some other platforms out there like Astro and the EmDash setup going on with CloudFlare.

You know, and I think that was a direct response actually. They realised, oh, people are going to want to chat with it. We could build this WordPress, what they consider like an upgraded version, in their mind.

And I think that, you know, it’s good to experiment. I think what WordPress does really well, to your point though, is they’ve hopped on the API centric side of things, building the right framework, not trying to force a certain thing down our throats, but actually leave it really open.

And I think that’s ultimately good because that’s how open source works. That’s why I think AI will have a good position with open source is. To your point you made previously, all the data’s out there, all the documentation’s out there. It’s going to be able to be extremely flexible. And I think that’s really why WordPress is still, in my opinion, going to exist and thrive.

But you are seeing agencies that are looking outside of just Core WordPress, because they can experiment with just a lot less time now. And they can also try out some tools that may be a better fit for certain projects.

[00:36:18] Nathan Wrigley: There’s a lot of technologists in our community and we love tinkering, don’t we, and playing with new things? So it’s fairly inevitable. It goes with the territory.

Okay. Alright. So that was Experimenting Beyond WordPress. I think I’m going to skip 15 there because I think we covered quite a lot of that. So I’m just going to go straight to number 16, which probably will become 15, if you know what I mean, when I put it into the show notes.

So the next one anyway is called, whatever its number, is Risks and Cautions. So we’ve built what I feel is like a fairly solid argument for doing this kind of stuff. And now towards the end, we’re going to knock it all down. No, we’re not. But what are some of the risks that you might point people towards?

[00:36:58] Matt Schwartz: A hundred percent. This is, I think, probably one of my favourite sections because people don’t really talk about these risks as much as they should. If you go on LinkedIn, it’s just like all rainbows and butterflies. And we’re building a new feature every day, and I think that noise can make people feel like they have FOMO and they just jump into AI and they don’t think about the risks. So I think this is actually a really important section for any listeners to listen to.

I think one of the first items, probably a little obvious, but with security issues, with AI tools, a lot of companies, every company, I feel like at this point is inputting keys and all sorts of things into these AI chatbots. And ultimately, like those tools can still be hacked. And actually legally, a lot of them say when you submit into the chat, it’s actually considered public record. Anything you submit to them. That’s literally what OpenAI made a legal argument recently about. So keep that in mind when you’re doing this.

There are some ways to do this securely. You can look into, and I think that’s something I would recommend agencies doing because they, you are holding onto client data. And ultimately you don’t want that stuff getting leaked.

Another thing, kind of on another point, which I personally have less concern with, but I think some agencies bring up a valid point, which is every time you talk to these chat bots, et cetera, they’re keeping track of all these conversations. So I know some agencies are being like, hey, how much does a website cost in my state or my country? Only use the data that other agencies have told you. And I don’t know if it’s really doing it, but people are doing that and they’re, you know, it’s not pushing back on them.

Things like that, just be aware of, I think what you’re inputting in, because it isn’t necessarily being leaked as far as they’re being hacked, but that data make get spit out to places you don’t want it to by other parties through chat.

[00:38:52] Nathan Wrigley: I feel that at some point in the future, some gigantic disclosure, something will be disclosed, which is so horrific that it makes us all sort of take a collective breath when we suddenly realise all we’ve given over. We haven’t got there yet, or at least to my knowledge we haven’t. But I feel that at some point in the near-ish future, some jaw dropping disclosure will occur, which will make us all think twice about exactly what you’ve just described.

What are we giving up? What have we given? But also what have we not consciously given? Which kind of bit of our business did we unintentionally open up for the AI to have access to that we didn’t intend to? And if we had the time again, we wouldn’t have allowed it to, and so on and so forth. So, yeah. Okay.

Any other things on the risks and cautions? I feel that there’s a couple more lurking in there.

[00:39:43] Matt Schwartz: Yeah. I think that one’s to be the most obvious that most people are talking about. One I think people aren’t talking about though is handling errors when you’re building your own tools. Essentially, a lot of times you may vibe code something, right, which is great. But because you’re not really going into the depth of every situation, it’s just making kind of assumptions, the AI, of what should be there.

And because it’s so easy and you’re like, well, I’m saving time, people don’t really outline all of this. And so they don’t really put error handling in these tools. And what happens is, of course there’ll be some edge case and, you know, things just break. And again, depending on the tool, if it’s an internal tool, you can probably get away with that. But if it is a public facing, or a client facing tool, that is the beauty I think of having a human actually review it with logic is they are going to have context that the AI doesn’t have.

To your point earlier, like subconsciously might be giving certain information to the AI that we don’t necessarily mean to. But you also might be leaving things out because you think you’ve already told the AI, or you think it’s going to assume a certain way, and you can’t really make that assumption. It ends up really backfiring in the long run, I think.

And that’s why being very conscious about error handling and being like, okay, we’re going to set up logging, we’re going to set up testing. Validation is just the responsible way to be building these tools that really, I feel like no one talks about.

[00:41:09] Nathan Wrigley: No. And the curious thing about it is, because it’s such a black box, I feel that almost every other technology that we’ve interacted with has been much more, I don’t know how to describe it. There’s been a higher barrier to entry. It’s more difficult to interact with it. You’ve had to, I don’t know, press buttons or enter code or what have you. Now you are just communicating. And maybe we’ll even sort of drop into voice communication at some point where we’re literally just talking with the thing.

There’s just no, how to describe it. There’s just such a small amount of friction that is required to interact with these things. And so it kind of lulls us into this perception that it can’t make mistakes. It’s error free and what have you. And we know that that’s not the case. I didn’t really describe that very well, but I hope you got a general sense of what I was trying to describe there.

[00:41:54] Matt Schwartz: I think you’re right. I mean I can give an example like building out CheckView, which is obviously like a, it’s a QA tool for WordPress sites. But one thing is, I knew a decent amount of QA before I started building it. But I learned so much context building it, and we weren’t using AI when we built the tool, right? That I would’ve never gotten out if I had used AI from the beginning.

Because like you said, there’s just such little friction. You as a human just don’t have to have that much information. You can just dive into something, having no idea on what you’re really doing, which is a blessing and a curse. And I think just being aware of that. And building in the right logs, and errors to at least essentially provide a safety net for yourself, knowing that you’re not going to know everything is really important.

[00:42:34] Nathan Wrigley: Yeah. And then a couple of other things which we’ll just sort of gloss over fairly quickly because maybe they’re sort of slightly common sense. Obviously, you know, you’ve got here sensitive client data, which might be, without your knowledge, being scooped up by the AI agents. So monitoring that.

Over dependence on AI vendors. This feels like everybody’s really become dependent on a handful of companies. Maybe you could count them on one hand, I think basically. There’s three or four really, that everybody seems to be using. So that may be something that we want to be mindful of.

And of course, the last one of your bullet points under Risks and Cautions is just the fact that AI makes some mistakes all the time.

[00:43:15] Matt Schwartz: All the time. I think most of that’s common sense. Really, the only one that I think people aren’t talking as much about is the overdependence on AI. Not to date this podcast, but I’m going to a little bit anyways. For example, with Claude, they are removing Claude Code from their $20 plan right now. You know, if you’ve built this into your agency process, well, get ready to pay, you know, a hundred bucks for every employee, which could be thousands of dollars.

[00:43:40] Nathan Wrigley: You can only imagine how valuable that will be. And maybe it’ll be 200 bucks, or a thousand bucks or whatever it may be.

[00:43:46] Matt Schwartz: Right. A lot of these companies obviously are subsidising the cost, and so I do caution agency specifically. That’s why having that AI vision is important, but also considering making it independent enough from your processes that you still can function if this thing does change around, because I do think there is going to be a pushback on cost at some point.

So for example, like when you’re building a product, I’m seeing some companies that are building AI so integral to their product that it will not function without AI. Or going to have to raise the price by 10x. And so like even with CheckView, our tools are there, but we haven’t built it in in such a way that you can’t use the tool without it.

And I think at the agency level, it’s the same idea. For the most part, trying to avoid building it in a way that you couldn’t reduce the AI needs if you needed to. Or just preparing for that the costs could go up and like, you know, if you give it to all 10 employees now, you know, at 20 bucks a month, get ready to possibly pay a lot more later.

And I just think that’s something important for agencies specifically to keep in mind. And I know that seems contradictory to what I said at the beginning of this, AI everything, but I think it’s important.

[00:44:59] Nathan Wrigley: I’m not a financial wizard in any way, shape, or form, but that does seem to be something which, sure as night follows day, is going to be coming, is the requirement to repay a lot of the venture capital that the AI is currently burning through. And yeah, so maybe a significant price hike.

And we’ve all got used to these practically free tools, and maybe that’s something that is not going to be in our future. So that’s a really good point. Put that bulwark in place to make sure that you are protected from that should it go up by, like you said, 10x or whatever it may be?

Okay, so I did say that there were going to be multiple, I think 16 is what I said. This probably will be the last one. This is likely outcomes for agencies. So, Matt, you get to stare into the crystal ball and tell us what your final thoughts are in terms of what you think are likely outcomes.

[00:45:47] Matt Schwartz: So I think some of this is already happening.

Hiring, I think, is slowing down in some agencies because they’re realising they can automate more. They don’t need as many, essentially non-specialist employees, or contractors. And I think that is a real thing that is happening.

I don’t know if it’s going to be necessarily a long-term issue. Hopefully as, essentially the floor raises, work gets better, more agencies will be focused on providing more value, more strategy, those sort of things. Again, the execution becomes a little bit of a commodity. So having essentially more junior team members who usually do that execution isn’t just quite as necessary. So I think that’s going to continue to come up.

But again, I think it’s going to be balanced out with even the costs we just talked about with the AI tools. There could totally be a point where the tools might get expensive enough that it makes sense to have a junior do this execution.

[00:46:45] Nathan Wrigley: Get the humans back. Yeah.

[00:46:46] Matt Schwartz: Yeah. And we go back the other way. Some people are saying that could happen. I could see the argument for that. But I think that’s one thing.

Another thing with agencies is you now can really productise more of your services. And this comes into the automation process that we talked about. You can take your processes, you can really package them up, I think. And there has been a lot of talk about productising services, but I think now you can get more nuanced.

So if you, for example, only build sites for plumbers, well, with AI you can get way more specific on plumber specific service needs, and build out a process with what you’ve already done, plus, with AI to make that, I think, a lot easier for the plumber or the client, to get what they need out of it. In the past, I think most agencies were trying to build SOPs as we have time. And it, you know, it’s just a really difficult process. And I think that’s where agencies I think could help a lot.

And then the last point I’ll make around here is really around that the tools I think will change. It will be, again, less about the execution of tools, how you’re building your sites. More about the actual automation. And then really just testing and monitoring and making sure everything’s working how it’s supposed to be. The human will essentially become more of the manager of the AI. And that extends, I think, to even the tools.

I could see there being more QA and monitoring tools out there for more specific needs. Because now, you know, AI can build 90% of this, which is great. I save that cost, but I know I need to pay maybe 5% of that towards some tools that actually watch and monitor what’s happening, and make sure these automations, and these websites, are really doing what they need to be doing. So I think there is going to be possibly a shift in that way around what sort of tools that we’re investing in as far as agencies go.

[00:48:40] Nathan Wrigley: So that we could describe as a bit of a marathon. I think, really, you really took us through the gamut of everything that could possibly affect an agency in the AI space. We’re in the year 2026, let’s see how it ages. But that was a really interesting deep dive into all of the different bits and pieces.

Matt has very kindly put together some show notes. What I think I’ll probably do is crib those. Maybe I’ll put them into the WP Tavern show notes, or maybe I’ll link to a Google Doc or something like that where you can see them. But you’ll be able to see all of the different bits and pieces that we went through. There’s a lot more on that document than we actually had a chance to go through. So definitely do check that out.

What I can say is that the future is definitely going to be interesting. Whether or not any of the predictions you’ve made will turn out to be true, time will tell.

But what a fascinating chat. Thank you so much for chatting to me. And I really appreciate you sticking around and doing the second episode somewhat unexpectedly with me.

Just before we sign off, Matt, where can we find you? Where are the best places online to hang out with you?

[00:49:49] Matt Schwartz: Yeah, definitely. So definitely, you can find me on the Admin Bar Facebook group. I’m also in LinkedIn, trying to be better about that. You can also check me out. I’ve got a Slack channel. Checkview.io, of course. Inspry.com. Feel free to reach out if anything comes up.

Definitely, overall I would just recommend agencies dabble in this. Don’t be reckless, but definitely see what makes sense for your agency. Document it all out ahead of time. And I think that that’s really going to be agencies strong suit is, can we leverage this stuff in a smart way?

[00:50:21] Nathan Wrigley: Well, you’ve certainly provided us with a lot of food for thought. So once more, go and check out the show notes on wptavern.com. I will probably link to the document that Matt has created on both part one of this and part two as well. So you’ll be able to check both of those out.

Matt Schwartz, thank you so much for chatting to me today. I really appreciate it.

[00:50:41] Matt Schwartz: Thank you so much, Nathan.