Ion: GraphQL on the backend (and everywhere else)
At Nerve, whenever I fetch any data from anywhere, I use a GraphQL-style query to do it.
Hear me out! Hear me out. I can feel your rage through the screen, but I've been using this approach on a substantial project (i.e. not a toy) and it's been working quite well, so I wanted to write it up in case anyone else finds it handy. It's not for everyone, but it may be for you!
To be clear right off the bat: Ion is not GraphQL. It looks like GraphQL and it follows some of the same principles, but the machinery is, in general, much simpler. Sorry if you feel betrayed by the headline; it was the most efficient description I could think of.
Why...?
Conventional wisdom says GraphQL is for handing off data to the frontend in large, mature systems, and conventional wisdom...is right! GraphQL solves a lot of problems that only pop up for specific companies in specific situations, and many of us have discovered the hard way that, carelessly applied, it tends to result in overwrought spaghetti.
But it does offer some compelling concepts. What always appealed to me was how GraphQL queries are loosely coupled and self-describing - as a client, you have no opinion on where your data comes from or how it's stitched together, but you do know exactly what it will look like when it gets to you, and it's obvious at a glance what fields you're actually depending on. For OLTP use cases, at least, this seems about the simplest and most ergonomic way to do it.
The main idea
What if there were a lightweight, schema-based, in-process data access layer that allowed for declarative queries like GraphQL? Emphasis on lightweight - if you're having scary flashbacks to, say, J2EE Entity Beans, I don't mean anything like that; I mean having a global namespace where you describe what you want, like:
const query = ```{
bash: {
currentDirectory
currentUser
}
}```;and then doing something like await getData(query) and getting back just what you asked for.
Well, that's Ion. More specifically, ion is a package I wrote that makes it easy to implement this pattern in Typescript projects.
How to use it
First, let's look at the query format. Ion is a single library that runs entirely within a Typescript process. This means we can get away with making queries objects instead of strings. This saves us a bunch of string manipulation and provides some extra type safety.
A typical Ion query looks like this:
const gitQuery = {
localGit: {
currentCommit,
editedFiles
}
};Except...you can't have a key without a value in Typescript objects. We need a 'default value' that is really just there so we can include the keys in the object.
I thought about using some simple sentinel value like zero or the empty string, but those have pretty unclear semantics and might cause collisions if I ever end up extending the query language. I settled on using a simple class with a simple member named something that other people probably wouldn't use:
export class I {
static ON = new I();
public __i_reserved_type__ = 1;
}There's a single static instance of our class so we don't have to keep writing new ().
The legal version of our query looks like this:
import { I } from "ion-types";
const gitQuery = {
localGit: {
currentCommit: I.ON,
editedFiles: I.ON
}
};(The I stands for indicator. The static instance is called ON to communicate that you're "flipping on" the indicator. Put 'em together and you get Ion!)
Ion's core export is a "query engine" object that runs your queries. For example:
import { getEngine } from "ion";
const qe = getEngine(gitResolvers);
const localGitData = (await qe.run(gitQuery)).localGit;
// outputs "2e4d3e8e485f380ad66443edff3a8eb1004d5313"
console.log(localGitData.currentCommit);
// outputs "['README.md']"
console.log(localGitData.editedFiles);What's that gitResolvers argument we're passing in to getEngine? Well, like GraphQL, Ion needs resolvers to tell it how to fetch different parts of a query. We provide our resolvers to Ion via a simple nested object where each leaf is either a function or a constant expression. Here's an example:
export const gitClient = require("simple-git/promise")();
export const gitResolvers = {
localGit: {
fileRoot: process.cwd(),
editedFiles: async () => {
const status = await gitClient.status();
return [...status.created, ...status.modified, ...status.not_added];
},
currentBranch: async () => {
const summary = await gitClient.branch();
return summary.current;
},
currentCommit: async () => {
const log = await gitClient.log();
return log.latest.hash.slice(0,7);
}
},
remoteGit: ...
};(Note that this resolvers object closes over a persistent client handle. I use this pattern a lot when writing Ion resolvers.)
Like GraphQL, Ion allows resolvers to return objects. If a resolver returns an object for a certain part of the query, Ion will use the rest of the query to select nested keys on that object. For example:
export const resolvers = {
a: async () => {
return {
b: {
d: 40
e: 50
},
c: {
f: 20,
g: 10
}
};
}
};
const qe = await getEngine(resolvers);
const result = qe.run({
a: {
b: {
d: I.ON
},
c: {
g: I.ON
}
}
});
// prints '40'
console.log(result.a.b.d);
// prints '10'
console.log(result.a.c.g);
// these are undefined!
console.log(result.a.b.e);
console.log(result.a.c.f);You can also return an array of objects from a resolver and Ion will recurse down into each of them:
export const resolvers = {
one: async () => {
return {
two: [{
three: 40
four: 50
}, {
three: 60,
four: 70
}, {
three: 80,
four: 90
}]
};
}
};
const qe = getEngine(resolvers);
const result = await qe.run({
one: {
two: {
three: I.ON
}
}
});
/*
result contains:
{
one: {
two: [{
three: 40
}, {
three: 60
}, {
three: 80
}]
}
}
*/Resolvers are globally namespaced, so it's simple to merge two resolver objects together. The way I organize my resolvers at Nerve is to group them into specific files; when my backend boots I take all the resolvers, merge them into one big object, and pass that object to getEngine. Then I use the resulting engine across the codebase.
Resolver arguments
Fields in GraphQL queries can have arguments which are passed to the corresponding resolver. Looks like this:
{
human(id: "1000") {
name
height
}
}Ion wouldn't be very useful if we didn't have a similar way to provide arguments, but the syntax above is not legal in Typescript. It's pretty nice to have queries be objects and we'd like to avoid going back to strings, but there's no easy way to attach extra information to object keys. The best compromise I could come up with was to keep the arguments under a nested key called _resolverArgs. So in Ion, the query above would be:
const query = {
human: {
_resolverArgs: {
id: "1000"
},
name: I.ON,
height: I.ON
}
};This does mean that if you want to provide arguments to a leaf node resolver you'll need to add an extra node under it. I usually name this something generic like val:
const resolvers = {
human: {
name: (_resolverArgs) => {
const computedName = _resolverArgs.salutation + " Matt Prast";
return {
val: computedName
};
},
height: // ...
}
}
// query it like so...
const qe = getEngine(resolvers);
qe.run({
human: {
name: {
_resolverArgs: {
salutation: "Mr."
},
val: I.ON
},
height: I.ON
}
});
A little awkward, but I haven't often needed to parameterize leaf resolvers, and all things considered I think this approach is still cleaner than the alternatives.
You can also use resolver arguments to construct nested resolvers at runtime, like this:
const resolvers = {
one: async (_resolverArgs: { prefix1: string }) => {
const prefix1 = _resolverArgs.prefix1;
return {
two: async (_resolverArgs: { prefix2: string }) => {
const prefix2 = _resolverArgs.prefix2;
return {
three: [prefix1, prefix2, "innermostString!"].join("::")
}
}
}
}
}
const qe = getEngine(resolvers);
const message = qe.run({
one: {
_resolverArgs: { prefix1: "theFirstPrefix" },
two: {
_resolverArgs: { prefix2: "theSecondPrefix" },
three: I.ON
}
}
}).one.two.three;
// logs "theFirstPrefix::theSecondPrefix::innermostString!"
console.log(message);
(This trick opens up a lot of possibilities; we'll dig into some of these a little further down!)
Type safety
Ion is made for Typescript, and what is Typescript without its types? (I know, I know, it's Javascript.) Since Ion is completely in-process, it's a lot easier to type it than it is to type GraphQL proper. In particular, we can do everything we need using only plain-vanilla Typescript types. No codegen, no extra build steps!
Even better is that by getting a little clever with mapped types and conditional types, we can auto-derive our query types, result types, and resolver types from one "mother" type, which cuts down on boilerplate a lot.
Let's look at an example. Those Git resolvers from earlier are real and I actually use them in some internal tools I built. Here's what the typing looks like:
import { ResolverMap } from "ion-types";
export interface GitSchema {
localGit: {
editedFiles: Array<string>;
/**
* The root of the current git repo. Here 'current' means
* "the repo where the current node process is running"
*/
fileRoot: string;
currentBranch: string;
currentCommit: string;
};
remoteGit: {
_resolverArgs: { remoteName: string }
branches: Array<string>;
};
}
export const gitResolvers: ResolverMap<GitSchema> = {
...
};GitSchema is the "mother" type here - it represents the shape of the underlying data that the resolvers should return.
ResolverMap<GitSchema> crawls GitSchema and turns it into the appropriate type for the resolver object. The logic is this: for each field type \(T_f\) of the object type GitSchema:
1) If \(T_f\) is a scalar type (number, string, etc.), the resolver can either be a literal of type \(T_f\) or a function that returns \(T_f\). So the resolver type should be Tf | () => Promise<Tf>
2) If \(T_f\) is itself an object type (in the way that GitSchema["localGit"] is an object type, for example), then the resolver should either be an object literal that is itself a resolver map, or a function that returns such an object. The correct type would be something like ResolverMap<Tf> | () => Promise<ResolverMap<Tf>> - but we also have to add logic for resolver arguments. The type of the resolver arguments should be the type of the _resolverArgs key on \(T_f\) - if that key exists at all! We can set this up using a conditional type. The full type looks like:
ResolverMap<Tf> |
(
"_resolverArgs" extends keyof Tf ?
() => Promise<ResolverMap<Tf>> :
(_resolverArgs: Tf["_resolverArgs"]) => Promise<ResolverMap<Tf>>
)Ion also offers a QueryMap<T> type that basically does the same thing as ResolverMap, except that it turns \(T\) into a query type instead of a resolver type. QueryMap returns a type where all of the values must either be the literal I.ON or a subquery object.
When you create your query engine, Ion infers the mother type from the type of the resolvers you pass into the constructor, like so:
export class QueryEngine<A> {
private _resolverMap: ResolverMap<A>;
constructor(resolverMap: ResolverMap<A>) {
this._resolverMap = resolverMap;
}
...Later, when you run a query, Ion uses this mother type to ensure that your query has a valid shape:
async run<B extends QueryMap<A>>(query: B) {
...Technical note: the type extraction & comparison happen at compile time and not at runtime, of course, but the effect is the same.
As I mentioned before, Ion is actually able to infer result types as well as query types. This is a little more involved than the rest of the typing stuff so I won't get into the details, but I can give you a quick sketch of how it works:
Typescript types object literals as narrowly as possible. That means that if you have
const myObj = {
a: 10,
b: "hello!",
c: {
d: "it's me"
}
};typeof myObj is not {}, nor is it Record<string, string | number | {}> . It's
{
a: number,
b: string,
c: {
d: string
}
}The general idea is to take the type of the query object (which is an object literal) and use it to select corresponding fields from the mother type. This gets us the type behavior we want: the result of the query is typed correctly, and we didn't have to type it ourselves. So, for example:
const qe = getEngine(gitResolvers);
const gitResults = qe.run({
localGit: {
editedFiles: I.ON,
fileRoot: I.ON
}
});
// "typeof gitResults" is {
// localGit: {
// editedFiles: Array<string>;
// fileRoot: string;
// }
// }
// this won't typecheck!! "currentBranch"
// does not exist on gitResults
console.log(gitResults.currentBranch);At Nerve I use Ion on both my backend and my frontend, as well as in various internal tools. Here are a few of the things I use it for:
- Pulling from a Kubernetes API server
- Pulling from Redis, Neo4j, and Elasticsearch
- Pulling user data via Better Auth
- Pulling from local browser storage
- Pulling information on spawned subprocesses
- and more...
Pros and cons
Pro: there's one way to do everything
This is the big one. Ion is at heart an organizational tool; it doesn't actually fetch anything for you, it just gives you a clean and systematic way to do it yourself. When I wrote Ion I considered it a fun experiment, but I wasn't sure if it'd have much of a payoff. Now, after a few years of using it on a big project, I'm surprised at how much and how often I appreciate having a consistent interface to my data.
I try to be very strict about always using Ion when I need to grab something that isn't in-memory. The big advantage of this is that it makes it easy to confidently tell which parts of my code depend on the environment, and which parts are internal business logic. If a call is qe.run, it's pulling data from the outside; if it isn't, it's not. Calls to Ion also look pretty distinctive, and it's easy for me spot them even if I'm just skimming through a file (and if I can't find 'em I can always just search for qe.run or I.ON.) It's also really nice knowing all the fetching logic has to be stuffed in resolvers; if I need to check what sources I can currently pull from, or if I'm looking for a particular source, I can reference my resolver files.
This is one of those things where I'm saving a couple seconds on something I do hundreds of times a day. Ion just makes working on the codebase feel a little smoother and more natural, and I find myself missing it when I don't have it around.
There are a few architectural benefits here, too; mocking, for example, is super simple. Since all of my data access happens through Ion query engines, I just need to mock out the engine, instead of trying to mock a hundred different source APIs. In particular I can just monkey-patch the static getEngine method:
const merge = require("deepmerge");
import { isPlainObject } from "is-plain-object";
import * as Ion from "./index.js";
import { ResolverMap } from "ion-types";
let originalGetEngine: typeof Ion.getEngine;
export function unmockEngine() {
if (originalGetEngine) {
(Ion as any).getEngine = originalGetEngine;
}
}
export function mockEngine<T>(mockResolvers: ResolverMap<T>) {
unmockEngine();
if (!originalGetEngine) {
originalGetEngine = Ion.getEngine;
}
(Ion as any).getEngine = (resolvers: any) =>
originalGetEngine(merge(resolvers, mockResolvers,
{isMergeableObject: isPlainObject}));
}This allows me to inject mock resolvers into Ion before every unit test. You'll notice that the merge call at the bottom lets me do partial mocks - if I don't supply a mock resolver for a particular key, Ion will fall back to using the originally provided resolver.
This lets me mock all of my external data sources without having to use any mocking libraries at all - and it only took ~15 lines of code!
Con: a little bit of boilerplate
There's some scaffolding involved in using Ion - mainly having to do with writing the resolvers, merging them, and passing them to getEngine. This is strictly more code than you'd have to write if you were calling the relevant APIs directly (you still make the same API calls when using Ion, you just call them inside of a resolver.) All this said, I don't think it's an unreasonable amount of extra boilerplate; I'd feel much worse about it if I weren't able to auto-derive so many of the necessary types.
Pro: it's data-driven
I'm a really big proponent of using code to control behavior only when absolutely necessary, and using data-driven approaches otherwise. Yeah, using code is more flexible upfront, but sticking to data can pay off down the line in unexpected ways.
Case in point: using Ion across the board means that we can do all sorts of tricky stuff with Ion queries - which, after all, are just data. For example, I do, in fact, use Ion to hand things off from my backend to my frontend. To do this I have two sets of resolvers - one on the frontend and one on the backend - that share the same "mother type." This type is stored in its own file, called nerve-frontend-api.ts, and it basically defines what kinds of data the frontend can pull from the backend. Looks like this:
export interface NerveFrontendAPI {
frontendApi: {
v1: {
nodeSemaphore: string;
allAgents: Array<AgentView>;
services: {
_resolverArgs: Array<ServiceView["id"]>;
val: Array<ServiceView>;
},
fieldsOrPointers: {
_resolverArgs: {
fieldIds?: Array<FullyQualifiedFieldId>;
pointerIds?: Array<FullyQualifiedPointerId>;
entityIds?: Array<FullyQualifiedEntityId>;
serviceIds?: Array<string>;
};
results?: Array<ServiceView>;
},
...
}
}
};I use NerveFrontendAPI to type resolvers for both the frontend and the backend. The frontend resolvers relay requests to the backend for certain types of data, and the backend resolvers actually fetch the data and return it to the frontend.
(On the backend, the resolvers look something like this:)
export const nerveFrontendAPIResolvers:
ResolverMap<LaceFrontendAPI> = {
frontendApi: {
services: async (_subQuery: {}, resolverArgs: Array<string>) => {
const qe = getEngine(backendModelResolvers);
// this pulls from the
// database
const backendServices = (await qe.run({
backendModel: {
services: {
_resolverArgs:
[ServicesFilterType.ServiceIdsEqual, resolverArgs],
id: I.ON,
orgId: I.ON,
displayName: I.ON,
authConfig: I.ON,
thumbnail: I.ON
}
}
})).backendModel.services;
return { val: backendServices };
},
...
}
};The upshot is that when I'm working in the frontend I can fetch data from the backend using regular ol' Ion queries:
const services = (await qe.run({
frontendApi: {
v1: {
services: {
// payload is defined earlier in the file
_resolverArgs: payload.serviceIds,
val: {
id: I.ON,
displayName: I.ON,
thumbnail: I.ON
}
}
}
}
})).frontendApi.v1.services.val;Since my backend and frontend are both written in Typescript, doing things this way gives me a convenient, tRPC-esque experience. I get typed queries with no extra build steps and no heavyweight dependencies, and the backend will only give me the keys I actually ask for, just like in GraphQL.
And it was very easy to build. The first step was to extend the resolver logic so that Ion passed each resolver a subQuery parameter which contained the rest of the query (i.e. the parts of the query below the resolver's key.) Once I had subQuery to work with I could just have my top-level resolver on the frontend send the entire query to the backend:
export const nerveFrontendAPIClientResolvers: ResolverMap<NerveFrontendAPI> = {
frontendApi: async (subQuery: any) => {
// backendURL is defined earlier in the file
const response = await axios.post(backendURL + "/ion", { data: subQuery }, {
withCredentials: true,
transformResponse: [(data: string) => {
try {
return JSON.parse(data);
} catch (SyntaxError) {
return data;
}
}]
});
return response.data.frontendApi;
}
};On the backend I literally just nest the query under the frontendApi key (to make sure client requests can only trigger resolvers under frontendApi, and not arbitrary resolvers), run the query, and return the result to the client.
// query comes from the request
const body = (await qe.run({
frontendApi: query
})) as any;
// put body in a json response and return to the clientI think the whole thing took me 15 minutes.
Con: dynamic queries are awkward to type
You'll recall that to get result types to work out we needed to take advantage of the fact that Typescript types object literals very narrowly. You may have asked yourself "Well, what if the query isn't an object literal? What if we construct it on the fly instead?"
That's a really good question!..and, uh, I don't have a great answer. Typescript will do its best to maintain the type information on an object as you manipulate it, but in general it won't be able to figure out its exact shape after a complicated piece of code unless you tell it what that shape is (or unless it literally ran the code itself, which typecheckers usually don't do.) And if we don't know the shape of the query, we don't know the shape of the results, which means we lose that result type inference which was so nice to have.
The bottom line is that Ion works best when your queries are object literals. To be honest, I thought this was going to be a bigger problem than it ended up being, at least for my use case. I'm sure there's some dark magic I could do to make Ion a little more pliable in this regard but literal queries work just fine the vast majority of the time - and if I really need to I can always just switch between two separate calls to qe.run.
Pro: it's minimal and flexible
It may be more accurate to say that it's flexible because it's minimal. It's two files and ~500 LOC, and it really just does one thing, which is to crawl a resolver object, executing any functions it finds along the way.
I mentioned earlier that GraphQL doesn't have an opinion on how you satisfy a query - and neither does Ion. Although the resolver framework is a handy convention, nothing stops me from injecting any degree of dynamism I want, and combined with Ion's data-driven nature this gives me a lot of latitude. Here's an example from the backend: even though the Express request object is technically in-memory I prefer to treat it as an "external data source" because it's not something I really control. I have a set of resolvers that let me pull from an Express object that I pass in via a resolver argument:
import { ValidationMap } from "validator-types";
import { ResolverMap } from "ion-types";
import express from "express";
export interface ExpressSchema {
request: {
_resolverArgs: express.Request<any>;
JSONPayload: Record<string, any>;
url: string;
urlParams: Record<string, any>;
...
};
}
export const expressResolvers: ResolverMap<ExpressSchema> = {
request: async (_subQuery: {}, req: express.Request<any>) => {
return {
JSONPayload: async () => {
if (!req.body) {
await (new Promise<void>((resolve, _reject) => {
express.json()(req, (undefined as any), () => resolve());
}));
}
return req.body;
},
url: req.path,
urlParams: req.query
...
};
}
};It's convenient in a number of ways to have custom Ion resolvers for Express (for example, mocking a web request is super easy - you just replace the request resolver with a mock resolver that throws away its argument and returns a bunch of fixtures.) If Ion didn't let you generate resolvers on the fly I wouldn't be able to do this.
Another backend example: I have an Ion resolver that allows me to query my database in terms of my logical backend models (which are plain 'ol Typescript types.)
First, I have a BackendModelSchema type that looks like this:
// Field, Pointer, and Service are all backend model types.
// FieldsFilter, PointersFilter, and ServicesFilter
// define what you can filter each model by.
export interface BackendModelSchema {
backendModel: {
fields: Array<Field & {_resolverArgs: FieldsFilter}>;
pointers: Array<Pointer & {_resolverArgs: PointersFilter}>;
services: Array<Service & {_resolverArgs: ServicesFilter}>;
...There's one corresponding resolver, and it's very simple:
const backendResolvers: ResolverMap<BackendModelSchema = {
backendModel: async (subQuery: {}) => {
// turns the subQuery into something we can run against the database
const query = buildQuery(subQuery);
// runs the query against the database and converts the raw
// results back into the logical data model
return convertResult(
await runQuery(
query,
// we're closing over connectionPoolCache, which is
// defined elsewhere in the file
connectionPoolCache.connectionPool
));
};
}What's nice about this is that the query translation and query execution functions are pluggable. For example, you could translate your queries to something Prisma understands, or just turn them into raw SQL using Kysely. We could even get bold and move to a different engine like NoSQL. Ion decouples our physical storage layer from the logical model (and everything above it!)
Ion only does one thing, but I like it because (in my completely unbiased opinion) it does its one thing very well. Plus, keeping it small makes it easy to pick up, easy to customize, easy to understand, and easy to rip out.
Con (sort of): less expressive than GraphQL
The flip side of the last item. GraphQL does a lot of things Ion doesn't (no GraphiQL, no fragments, aliases, variables, or directives, no subscriptions or mutations.) Some of these don't really fit Ion's usecase. Other are obviated by Ion's Typescript-native approach (for example, Ion doesn't need a built-in type system.) Still others are things I maybe could have included, but decided weren't worth the extra bulk. If you're looking for something that exactly replicates GraphQL, you should keep looking (but if I'm being honest...I kind of consider that more Pro than Con.)
What's next?
Probably nothing! For what I'm using it for, this library works great. If I ever need more features I'll add 'em, but for now I'd like to keep Ion as simple and minimal as possible.
I'm happy to keep using Ion on my own, but if there's interest it'd be pretty easy to open-source too. So, drop me a line if you'd to use Ion in your own project!